A/B testing rules
Choose how careful Crovio is before it calls a test, with presets or your own rules.
On this page
Settings → A/B testing decides when Crovio calls a test. Crovio waits for the noise to settle, keeps collecting to confirm the result holds, then calls it. The rules apply to every test's results, including tests already running.
Pick a preset
- Balanced: recommended for most stores.
- Cautious: fewer false winners, slower calls. For big stores and big decisions.
- Fast: quicker calls with more risk of a fluke. For low-traffic stores.
- Custom: your own rules, set below.
Click Save rules. Reset to Balanced undoes your changes.
Fine-tune the rules
| Stage | Rule | What it does |
|---|---|---|
| 1. Collect enough data | Shortest run | Never call a test sooner. Keep it to whole weeks so every weekday is counted |
| Require a minimum per group | Optional. A fixed number of orders (or goal completions) every group must reach, on top of the target Crovio sets from the size of the lift | |
| 2. Noise settles | Leader holds steady for | Days in a row the same group must lead with a steady lift |
| Counts as leading at | Chance to win a group needs before it counts as the leader | |
| 3. Confirm it holds | Keep running at least | Extra days after the result settles |
| Until every group has | More data than it had when the result settled | |
| But no longer than | For slow tests that would take weeks to gather the extra data | |
| 4. Decision | Call a winner at | Chance to win needed once confirmation is over |
| No meaningful difference within | Called flat once the range rules out lifts bigger than this either way | |
| Time limit | No leader by then means any difference is too small to measure at your traffic |
The values for each preset are in How Crovio decides a winner.
Was this page helpful?