When should I end a Shopify Rollouts experiment?
Short answer: End it when you have enough visitors and orders in each variant for the difference to be real, the test has run through at least one full weekly cycle, and nothing unusual (a sale, a mid-test edit) has skewed it. Be careful: once you apply a rollout, Shopify can't revert the change.
A checklist before you end
- Enough data in every variant? A handful of orders can swing either way. Look for a stable gap, not a single good day.
- At least one full week? Weekday and weekend shoppers behave differently.
- Did a sale or promotion overlap? Results from a sale week may not hold in a normal week.
- Was a variant edited mid-test? Shopify warns that editing the treatment or control during an experiment affects the results.
- Did you look at money, not just conversion rate? Check revenue or profit per session (see: Why did my conversion rate go up but my profit go down?).
- Are you sure about applying? “After you apply a rollout, you can't revert the changes.” Archiving is also permanent.
Ending early: roll back or apply
You can end an experiment early and choose to roll back or apply. If the result isn't clear yet, rolling back keeps your options open.
With Upshot
On the Pro plan, Upshot gives every experiment a plain-language verdict — “Too early to call. About N more days needed.”, “B is likely converting better”, “No clear difference after enough traffic. Consider ending the test.” — with the probability that a variant converts better once there's enough data. On every plan, including Free, it walks you through a short checklist before you apply. A notebook keeps your hypothesis, result and lessons for next time.
Check your own numbers with the free experiment calculator.