Help

How do I read my Shopify price test results?

Short answer: Don't pick the winner by conversion rate. A higher price almost always converts a little worse. Compare profit per session for each group, then check that the conversion gap isn't just luck with the free experiment calculator.

The three numbers to compare

For each test group, write down three numbers:

  1. Conversion rate = orders ÷ sessions. Expect the higher price to be lower.
  2. Revenue per session = revenue ÷ sessions. This shows whether the higher price made up for fewer orders.
  3. Profit per session = (revenue − product cost − discounts) ÷ sessions. This is the one to decide on.

Worked example

Example data — a made-up store. Product cost is $20 per unit, and each order has one unit.

PriceSessionsOrdersConversionRevenue / sessionProfit / session
A · Control$404,0001002.50%$1.00$0.50
B · Higher price$464,000842.10%$0.97$0.55

Reading it step by step:

  • Conversion fell from 2.50% to 2.10%. On conversion alone, A looks like the winner.
  • Revenue per session is nearly flat: $1.00 against $0.97.
  • Profit per session rose about 9%: $0.50 to $0.55. Each B order keeps $26 instead of $20.
  • Is the conversion drop real? Put 4,000 / 100 and 4,000 / 84 into the calculator. The chance that B converts worse is about 88%. That's below the 95% line, so the verdict is “Keep running.” The drop might be real, but it isn't proven yet.

So in this example, the higher price is ahead where it counts, and the test needs more traffic before you call it.

Before you call a price test

  • Cover at least one full week, and avoid ending during a sale or holiday. Price sensitivity changes on sale days.
  • Check that nothing else changed. A theme edit or a new discount during the test mixes two effects.
  • Watch returns and repeat orders. A higher price can bring fewer but better customers. That shows up after the test ends.
  • Use the same costs for both groups. If shipping or fees change with the price, include them in profit.

Running a price test with Shopify Rollouts

A rollout can include catalog changes. Setting a catalog to Active publishes its prices, so an Experiment can show different prices to its control and treatment groups. Your store must be on the Grow plan or higher to run experiments.

Shopify's rollout results show rate metrics such as conversion. You can't customize the metrics, and for rollouts with a discount change, revenue and orders attributed to the discount aren't available. To get revenue and profit per session for each group, combine the rollout's sessions with your order data, or use an app that does it.

Upshot puts net sales, discounts, profit and revenue per session for each Rollouts test group side by side. Revenue is estimated for discount tests. Free to install. Revenue and profit by test group come with a 14-day free trial.

Install Upshot

Related questions