Hey Friends,
Amy here, an economist-turned-growth-designer at Amplitude.
Most B2B SaaS teams are trying to become more AI-native right now by launching or integrating AI features into their products.
You've probably felt some version of the tension between AI features existing, but adoption staying stubbornly low. The users who would benefit the most from AI are the ones least likely to see it out on their own. Moreover, any drastic change in existing workflows could lead to confusion or backlash on the existing user base.
Over the past 3 months, my team ran two experiments on the Amplitude home page with the most successful experiment increasing adoption of AI features by +10%. I’ll share the specific real-world A/B tests in Part 1. In Part 2 tomorrow, I will discuss how one sentence in the UI converted another +30%.
~ Amy Lin, Product Designer, Growth at Amplitude
Initial AI Chat Launch
The Inspiration
A viral post on X about AI-first homepages that inspired Amplitude’s experiment
At Amplitude, we initially launched AI chat on our homepage as a bolt-on module to our existing dashboard homepage where users land when they sign into our popular analytics platform. After launch, the follow-up experiment I proposed was to completely replace the existing homepage with a chat input box that was inspired by a viral post on X with Linear, PostHog and a few SaaS products with a chat-first home. Given that Amplitude homepage has historically been a high traffic but low intent surface of the product, it is a safe playing field for a more drastic change in design.
The Before Design:
The Before: Amplitude’s Dashboard-First Homepage with an AI chat feature pinned to the top.
The change was simple but important. We got rid of the data cards that were used to surface recent charts and dashboards on the existing home and made the AI chat entry point the only element on the home page.
Meanwhile, I was cautious that in B2B SaaS, a big UI change usually produces a vocal group who wants the old version back immediately. Therefore, we launched the chat-first home as the default with the option to switch back to classic dashboard-first home.
The First Experiment:
V1 of the experiment removed dashboard data cards to focus entirely on the AI chat
We tested it against the classic dashboard-first home as an A/B test for 2 weeks under the hypothesis that if we changed what users saw when they opened Amplitude, we could change what they reached for first.
What we found:
Users in the treatment group (chat-first home) were more likely to send their first Global Chat message from the home page and more likely to send a second one, too. Better entry point, better continued engagement.
AI usage ran at roughly 40% in treatment vs. ~30% at baseline. (The baseline included many users who never saw the homepage at all, so the real gap was even larger.)
Fewer than 3% of users switched back to classic dashboard-first home.
Key takeaways:
The core bet was right; changing the surface changed the behavior.
Measuring second action matters. It suggests the new homepage wasn’t just generating curiosity clicks, but helping users build an early habit.
For any drastic change, test boldly but build guardrails to avoid breaking existing workflows and pissing the existing user base off.
The Second Experiment:
The second experiment introduced personalization with user activity feed and past queries
In our next iteration, we introduced 2 new elements:
Contextual prompt cards: Combination of top ten converting prompts with attached context from the user's own activity. For example, this could be a chart they recently viewed or a dashboard they edited. These prompt cards were NOT generic analysis of any data available, but rather situated starting points that meet users where they already are.
User activity feed: A feed of recent AI Chat conversations happening across the user's organization. The goal is to help Amplitude users learn from their colleagues who were already using Amplitude AI Chat on prompting techniques, and create a sense that AI usage was already happening around them.
The overall AI engagement improved 10% in relative percentage and the time it takes for users to try out AI Chat shortened by 40%. Better yet, we were able to drive AI engagement improvement without hurting broader product engagement with non-AI features.
Tomorrow in Part Two, I’ll share how we were able to design the user activity feed with privacy-first approach despite complicated push back from internal testing. I’ll share how a privacy feature on user activity drove adoption by an additional +30-50% depending on user segment.
Until then, see you in the GrowthDesigners.co free community Slack workspace (Sign In or Join.)
Stop typing what you could say in 10 seconds.
Wispr Flow turns your voice into clean, professional text inside any app. Emails, Slack, client updates — speak once, send without editing. 4x faster than typing.
Growth Design School is Now On Demand
Growth Design School is our community-created program that provides fully-remote learning, including 5 hours of video content, case studies, customized assignments, and more!
Learn more - https://www.growthdesigners.co/school
