Is my Shopify A/B test result significant?
Short answer: Enter the sessions and orders for each test group into the free experiment calculator. It tells you the chance that B converts better than A, and whether that's enough to call the test. A result is called at a 95% chance or more, and only after each group has at least 100 sessions and the test has at least 10 orders in total.
What “significant” means here
A test result is “significant” when the gap between A and B is too big to be explained by luck. With small numbers, luck is large. Three extra orders can turn a loser into a winner overnight.
The calculator answers one plain question: given the traffic so far, how likely is it that B really converts better?
- Below 95%: keep running, or accept that there's no clear difference.
- 95% or more: B is likely better (or worse). The difference probably isn't chance.
Example
Example data — the calculator's pre-filled numbers, not a real store.
Conversion rate Likely range A · Control 2.00% 1.6% – 2.4% B · Treatment 2.60% 2.2% – 3.1% Chance B converts better: 98% → Verdict: B is likely converting better (98% chance).
The two ranges barely overlap, so the calculator is confident. If the ranges overlapped a lot, the chance would sit closer to 50% and the verdict would be “Keep running.”
The five verdicts
| Verdict | What it means | What to do |
|---|---|---|
| Too early to call | Not enough sessions or orders yet | Keep the test running. Add “Days running” to see about how many more days. |
| Keep running | B may be ahead or behind, but not by enough | Don't end it yet. Ending now could lock in luck. |
| B is likely converting better | The gap is unlikely to be chance | Before applying, check revenue and profit per session. |
| B is likely converting worse | B is probably hurting conversion | Keep the original, unless B earns clearly more per order. |
| No clear difference after enough traffic | Enough traffic to see a meaningful change, and none showed up | End the test and try a bolder idea. |
Where to find your numbers
In your Shopify admin, go to Markets › Rollouts and open the experiment. You need the sessions and orders for the control and the treatment.
Keep in mind how Shopify counts rollout results:
- Results update a few minutes after each visit, not in real time.
- Bot traffic isn't counted.
- You can't filter the results by device or country.
- For a rollout with a discount change, discount-specific revenue and orders aren't available.
Three mistakes that fake a “significant” result
- Peeking and stopping on a good day. Check as often as you like, but only stop when the verdict says so.
- Running less than a week. Weekday and weekend shoppers behave differently. Cover at least one full week.
- Editing a variant mid-test. Shopify warns that changes during an experiment affect the results. Restart instead.
Significant isn't the same as profitable
A significant lift in conversion can still lose money. A discount, for example, can win on conversion and still lose on profit. Read Why did my conversion rate go up but my profit go down? before you apply a winner. Applying a rollout can't be undone.
Run experiments in Rollouts? Upshot shows each test group's conversion, revenue and profit side by side, with the same plain-language verdict as the calculator. Free to install. Revenue and profit by test group come with a 14-day free trial.