"Failed" Experiments (Growth Design Monthly)

August 2021

Dear Readers,Last month, Fonz Morris from Netflix shared his best experiment ever. It was a multi-year effort that led to an additional $30 million in annual revenue for Netflix. These case studies inspire us, but the reality of design experimentation is that the default result is either inconclusive or a failure. You lose, you lose, you lose, and then—if you’re patient and lucky—you win! So I wanted to dredge up some “failed” experiments. Ultimately, these experiments taught the team more than any win could have. They were wake-up calls to deeper user experience challenges. It’s a cliche but these stories show that it’s not about winning and losing, but winning and learning.Your favorite loser,MollyGrowthDesigners.Co Newsletter Editor

Why It Failed

The real work begins when an important experiment fails. Investigate why it failed. Replay user sessions, book user interviews, DIG IN. Understanding why an experiment failed can generate a crystallizing moment of massive learning and momentum to address root causes for failure.

Experiment 1 - Review Your Cart

GrowthDesigners.co Founder Lex Roman said she would jump off a cliff if the experiment failed. That’s how certain she was that adding an order review step to the checkout at The Black Tux would decrease drop-off. It took a month of engineering effort to build and only hours to fail. The variant led to a -35% worse checkout conversion rate and needed to be rolled back immediately. 

Hypothesis: If a cart review step is added then more people will checkout.

Shock and then work. Lex researched why it failed and learned that the new checkout review step revealed order inaccuracies, especially in sizing, that had been previously hidden. Customer support was dealing with these issues post-purchase; now the customer was seeing these mistakes and going back to fix orders, and sometimes giving up. As a result of this “failed” experiment, The Black Tux invested in long-term fixes to their shopping cart and size selection experience. Luckily for us, Lex didn’t jump off that cliff. 

Experiment 2 - The Video Intro

I was trying to drive adoption of a major big bet feature at InfluxData. The team noticed a macro trend of data workers moving from dashboards to notebooks as the preferred user interface paradigm. And yet, the new notebook feature InfluxData released wasn’t seeing the adoption that the team hoped for. So they came up with the idea to add an educational video about “Notebook Concepts” to the first notebook a user created. 

Was an ability gap to blame for low adoption and would a video help teach users this new feature? After running the experiment, the intro video only had a 14% probability to best over no video. But worst of all, it had a decreasing impact on feature engagement over time after an initial spike. Video is simply too expensive to produce and update without a substantial, proven evergreen impact. The quest to drive feature adoption continues with a greater focus on core functionality needed to unlock user engagement. Since then, the team has replaced groundbreaking design for coding experiences in notebooks.

Concluding Thoughts

Losing important experiments can sting, but the pain can also be clarifying if you take the time to research why they failed. We’ve seen again and again some of the most impactful experiments are the surprise losses that offer a turnaround moment if the team leans in.

About the Author

Molly Norris Walker is the newsletter editor and facilitator for GrowthDesigners.co. She is a serial head of design at high-growth startups.

Happy graduation to the Growth Design Fellows! Over the last five months, we shared in the struggle of planning the right experiment and in the joy of finding a new design that really hits the mark. Together, we explored with growth design leaders from GitLab, OpenTable, Vidyard, Google and Netflix.

Join the Growth Designers SlackConnect with growth designers in our Slack channel. Share ideas, discuss challenges and learn from each other. 

Looking for Work? Hiring?

Well have we got some resources for you! Check out our Jobs page and our Growth Design Directory.