Overview
We rebuilt the onboarding click funnel for a regulated telehealth product: the path from creating an account, through a health questionnaire, choosing a treatment, and verifying identity, to checking out. Then we ran the new version side by side with the old one.
The first read said the new flow turned about 20% more new accounts into paying customers than the old one. A closer read says something more useful. The new flow did change behaviour where it was built to: more people finished the questionnaire, chose a treatment and reached checkout. But a smaller share of the people reaching checkout paid, and purchases per new account showed no clear change either way.
This page is the closer read, including three traps that each made the new version look better than it was. Updated 29 October 2026: an earlier version of this page led with the first number. The company is kept unnamed on purpose; the method is the story.
How we tested it
The new onboarding shipped behind a feature flag and ran alongside the old one on the same days, so seasonality, pricing and marketing pushes hit both versions. Each account was assigned a version when it signed up, and that version travels with its server-side records. Other onboarding versions for specific products and channels ran at the same time; this comparison covers only the main two.
What did not hold still was the split. Over six weeks the new version's share of new accounts was changed several times: about a fifth in the first week, then about half, close to zero for several days, about half again, and about three quarters in the last week. Conversion moved too, on the old version as much as the new: the first week was the weakest and the last the strongest. The new version had a small share of the weak week and a large share of the strong one, so adding up all six weeks flatters it.
So every number on this page compares the two versions within the same week, then combines the weeks. Because the split also changed partway through some weeks, we checked the main results by matching day by day as well; the figures move slightly, the conclusions do not. The outcome is a new account completing a first payment within seven days, counted only for accounts old enough to have had their full seven days.
Where the redesign changed behaviour
Each row is the share of people at the previous step who reached this one; the first row starts from new accounts. Browser events, same weeks compared.
View as table
| Step | Old | New | Change |
|---|---|---|---|
| Questionnaire completed | 78.5% | 84.0% | +5.5 pts |
| Treatment chosen | 62.8% | 74.0% | +11.2 pts |
| Identity verification started | 86.5% | 88.4% | +1.9 pts |
| Checkout started | 37.1% | 37.7% | +0.6 pts |
| Checkout completed | 44.5% | 39.5% | −5.0 pts |
The browser's events put the biggest gains at the questionnaire and treatment steps, the parts we rebuilt. Read those as direction, not size. These step rates come from the browser's six-step funnel, the same kind of funnel the third trap below warns about, and the old version lost more of its mid-funnel events, so part of the pattern is measurement.
The server's records cover the steps that matter most and settle the size. Comparing the same weeks, 20.7% of new accounts reached checkout on the new flow against 18.0% on the old one: about 15% more. That gain is real, but it is about half of what the browser's step-by-step chain implies.
What happened at checkout
Then checkout. On the server's records, about 40% of new-flow checkout starters paid against about 46% on the old flow, a drop of about five points (95% range roughly two to nine points), comparing the same weeks.
That does not by itself mean checkout got worse. The new flow sent more people to checkout, and the extra people may have been less ready to buy, which lowers the completion rate even if checkout itself is unchanged. Whether it is the people or the page is the next question.
Put the two together and the gain on the way in is cancelled out at the door. On the server's own record of new accounts and payments, which ad blockers and closed tabs do not affect, purchases per new account changed by about zero: +1.6% matching week by week and −2.2% matching day by day, with a 95% range from roughly 12% worse to 13% better. The browser's copy says about +8 to +10%, but that is within noise, and the browser is the copy that both loses events and mislabels versions. This test cannot rule out a modest gain or a modest loss. What it does rule out is the 20% first read.
Three traps that flattered the new version
- Adding up weeks while the split moved. Picture two weeks with the same number of signups. Both versions convert 10% in the first week and 14% in the second, so the redesign changes nothing. But the new version gets a fifth of the first week and three quarters of the second. Add the weeks together and the new version averages about 13.2% against about 11.0% for the old one: a 20% lift produced entirely by timing. Comparing within each week gives the right answer, no difference. Our test had the same shape.
- Counting the browser's copy. Accounts and payments are tracked twice, once in the browser and once on the server. The browser loses some events to ad blockers, and in this test it also filed more new accounts under the old version and fewer under the new one, which made the old version look worse. The server copy is the one to trust for money and accounts, and it showed a smaller gap even before the weekly comparison: about 11% instead of about 20%.
- Trusting the longest funnel. A six-step funnel showed the biggest gap of all, because the old version lost more of its intermediate events, and an ordered funnel drops anyone with a missing step. More on that in building click-funnel events you can trust.
One thing we got right from the start: only accounts that had their full seven days to complete a first payment are counted, so recent signups do not drag either version down.
None of these traps is exotic. Each one can quietly favour one version, and here all three favoured the new one. Each produced a chart that looked perfectly reasonable.
What it points to next
On its main goal, more paying customers, the redesign has not shown a result either way. It did change behaviour where it was meant to: more people finished the questionnaire, chose a treatment and reached checkout. That has not turned into a measurable gain in customers, and the open question is at checkout.
Two things come next. First, find out why the share paying at checkout fell: whether the new flow brings less-ready people to the payment step, or whether checkout does not fit what the new flow sets up. Second, run the next comparison on a fixed 50/50 split, assigned at random per account and held steady for the whole test, with purchases per new account on the server as the primary measure, sized in advance to detect a change of about 5%, and refunds and clinician declines as guardrails.
The first number was the one we hoped for. The useful one took three more passes to find.