Growth Designers Monthly
Posts
One A/B Test, Two Opposite Results

One A/B Test, Two Opposite Results

Before you copy that “win”... re-test it. This is a case study of the same experiment ran at two similar fintech companies that got different results.

January 12, 2026

In partnership with

Dear Readers,

This issue is about one idea, two fintech products, and wildly different outcomes.

A GrowthDesigners.co community member, Andy Hsu at Mercury, ran a call-to-action button A/B test with blockbuster results that inspired me to re-run it as my first test at a brand new job.

Most growth content is over-hyped or flat-out wrong, especially A/B test “wins.” I was confident enough to try this button test because it was shared in a community by the designer himself and then others chimed in to say similar tests had earned strong results for them.

If you’re curious about how messy second-hand stories can get, I once fact checked an influencer-created Pinterest case study with the practitioner behind it—and the published “lesson” didn’t match what actually happened.

A/B testing is applied science. At GrowthDesigners.co, we act like a scientific community of peers no matter what your job title is. We amplify each other, but also question and re-test each others’ hypotheses. (Join us or sign-in to the community.)

Experimentally yours,

Molly Norris Walker, Community Co-Leader, Lead Growth PM

The Mercury Test

Mercury is a US-based startup bank where Andy ran a lightweight test that changed a deactivated button to an active state that displays error handling instead. He hypothesized that by making the form active by default, the UI would give better direction to users creating virtual credit cards, thereby increasing conversion to card creation.

This was a BIG win! 20.2% increased click-through rate.

It huge result surprised the entire Mercury team. The interaction pattern of active buttons with error states became Mercury’s design system default; they deprecated all deactivated buttons across the entire product experience.

The Prosper Test

Prosper is a US-based personal finance company with loans and a credit card. I was so inspired by Andy’s Mercury test that I decided to try it as my first experiment as a new growth product manager at Prosper. I ran the experiment on the flagship personal loan application to improve page-over-page click-through rate. I made the button active by default like Mercury’s virtual card creation page. An error state would appear on click when required fields like email address were unfilled (instead of deactivated button until all required fields were filled.)

The experiment result? 🤏 Flat - No or insignificant change.

At Prosper, we did NOT roll this active-by-default button state out because we saw now results. We kept the existing pattern that buttons are inactive until all required fields are filled and then they become active and the user can progress to the next page in the loan application.

In the community, most people who had tried a similar test saw positive results like the winning experiment. Members across industries, including teams at Klaviyo and Spring Health, reported strong results. And yet, it didn’t move the needle for Prosper. What if I had adopted this UI pattern without testing it myself? It would have been a lot of wasted effort and now I can provide a counter opinion in the community. In an AI era, checking A/B test ideas like this with diverse practitioners who actually ran the experiments themselves is more important than ever.

Concluding Thoughts

Even though Mercury and Prosper ended up with opposite results, both experiments were still valuable. The takeaway is that the power of scientific peers to challenge your ideas and re-test hypotheses. When communities like GrowthDesigners.co share what actually worked and what fell flat, we all get better at designing the right tests for our own users instead of chasing someone else’s outlier.

About the Author

Molly Norris Walker is a product growth person with a design lens. She’s a co-leader of the GrowthDesigners.co community, who has focused her career on the intersection of data, growth, and customer experience. For the last five years, she has worked on products with applied machine learning at their core and a product-led growth market motion as a serial venture builder. Formerly at Prosper, she now leads predictive maintenance for The Boeing Company.