Help

Is my Shopify A/B test result significant?

Short answer: Enter the sessions and orders for each test group into the free experiment calculator. It tells you the chance that B converts better than A, and whether that's enough to call the test. A result is called at a 95% chance or more, and only after each group has at least 100 sessions and the test has at least 10 orders in total.

Open the calculator →

What “significant” means here

A test result is “significant” when the gap between A and B is too big to be explained by luck. With small numbers, luck is large. Three extra orders can turn a loser into a winner overnight.

The calculator answers one plain question: given the traffic so far, how likely is it that B really converts better?

  • Below 95%: keep running, or accept that there's no clear difference.
  • 95% or more: B is likely better (or worse). The difference probably isn't chance.

Example

Example data — the calculator's pre-filled numbers, not a real store.

Conversion rateLikely range
A · Control2.00%1.6% – 2.4%
B · Treatment2.60%2.2% – 3.1%

Chance B converts better: 98% → Verdict: B is likely converting better (98% chance).

The two ranges barely overlap, so the calculator is confident. If the ranges overlapped a lot, the chance would sit closer to 50% and the verdict would be “Keep running.”

The five verdicts

VerdictWhat it meansWhat to do
Too early to callNot enough sessions or orders yetKeep the test running. Add “Days running” to see about how many more days.
Keep runningB may be ahead or behind, but not by enoughDon't end it yet. Ending now could lock in luck.
B is likely converting betterThe gap is unlikely to be chanceBefore applying, check revenue and profit per session.
B is likely converting worseB is probably hurting conversionKeep the original, unless B earns clearly more per order.
No clear difference after enough trafficEnough traffic to see a meaningful change, and none showed upEnd the test and try a bolder idea.

Where to find your numbers

In your Shopify admin, go to Markets › Rollouts and open the experiment. You need the sessions and orders for the control and the treatment.

Keep in mind how Shopify counts rollout results:

  • Results update a few minutes after each visit, not in real time.
  • Bot traffic isn't counted.
  • You can't filter the results by device or country.
  • For a rollout with a discount change, discount-specific revenue and orders aren't available.

Three mistakes that fake a “significant” result

  1. Peeking and stopping on a good day. Check as often as you like, but only stop when the verdict says so.
  2. Running less than a week. Weekday and weekend shoppers behave differently. Cover at least one full week.
  3. Editing a variant mid-test. Shopify warns that changes during an experiment affect the results. Restart instead.

Significant isn't the same as profitable

A significant lift in conversion can still lose money. A discount, for example, can win on conversion and still lose on profit. Read Why did my conversion rate go up but my profit go down? before you apply a winner. Applying a rollout can't be undone.

Run experiments in Rollouts? Upshot shows each test group's conversion, revenue and profit side by side, with the same plain-language verdict as the calculator. Free to install. Revenue and profit by test group come with a 14-day free trial.

Install Upshot

Related questions