How Crovio decides a winner
The three stages every test goes through before Crovio calls it, and the rules you can adjust.
Crovio never calls a test on a fixed number of orders. A big store can need thousands of orders per group to tell a 3% lift from noise; a small store needs far fewer to spot a 30% lift. So every test goes through three stages, checked on the cumulative result at the end of each day.
1. Collect enough data
- At least a shortest run (7 days by default) so every weekday is counted.
- Enough data in every group to measure a lift the size of the one showing, at 80% power and 95% confidence. The smaller the lift, the longer this takes. Crovio shows the target, for example "A lift this size needs about 1,200 orders per group".
- Optionally, a fixed minimum per group that you set (off by default).
2. Noise settles
The same group leads, with a strong chance to win (90% by default), and the lift barely moves for several days in a row (5 by default). That's the point where early swings have died down. The test shows Settled, confirming.
3. Confirm it holds
The test keeps running for a few more days (3 by default) until every group has more data than when it settled (20% more by default), or a maximum number of days for slow tests (7 by default). If the result moves, the test goes back to collecting and shows Results moved, still settling.
The call
Only a result that held through confirmation is called:
- [Variant] is likely better: chance to win at or above the call bar (95% by default).
- Control is likely better: control beats every variant at the same bar.
- No meaningful difference: the 95% range rules out lifts bigger than 5% either way (checked after twice the shortest run), or the test reached its time limit (42 days by default) with no leader. Safe to end and keep whichever version you prefer.
Presets
Change the rules in Settings → A/B testing. They apply to every test's results.
| Rule | Balanced (default) | Cautious | Fast |
|---|---|---|---|
| Shortest run | 7 days | 14 days | 7 days |
| Leader holds steady for | 5 days | 7 days | 3 days |
| Counts as leading at | 90% | 90% | 85% |
| Keep running at least | 3 days | 7 days | 2 days |
| Until every group has | 20% more data | 30% more data | 10% more data |
| But no longer than | 7 days | 14 days | 5 days |
| Call a winner at | 95% | 97% | 90% |
| No meaningful difference within | 5% | 3% | 8% |
| Time limit | 42 days | 56 days | 28 days |
- Balanced: recommended for most stores.
- Cautious: fewer false winners, slower calls. For big stores and big decisions.
- Fast: quicker calls with more risk of a fluke. For low-traffic stores.
Pick Custom and use Fine-tune the rules to set each one yourself, including Require a minimum per group. Reset to Balanced undoes your changes. A/B testing settings.
The maths, briefly
Win chance is a Bayesian probability (flat prior) that a group beats control. Revenue per visitor and average order value use a normal approximation; the interval is the 95% range for the lift. Crovio also checks that visitors split the way you set them (sample ratio mismatch) and warns you if not.
Was this page helpful?