Concurrent testing means running multiple experiments at the same time. While this approach can speed up insights, it also increases the chance that tests might affect one another, potentially influencing your results.
Managing concurrent 2-cell experiments
When running more than one 2-cell experiment in the same country, it’s important to balance faster insights with the risk of test overlap. Here’s our view on how risk increases as you add more experiments in a market:Two 2-cell experiments
Generally low risk, though it’s wise to check for any overlapping areas between test groups.
Three 2-cell experiments
Higher risk. We recommend reviewing the overlap between each pair of tests to help ensure one experiment does not skew the results of another.
More than three experiments
Considerable risk. Only proceed if you understand that increased interference may blur the true effects, and you’re prepared to accept that risk for business-critical decisions.
Guidelines for markets with few regions
For countries or regions with fewer than 50 unique areas, we strongly recommend running a single 2-cell experiment. This helps prevent overlap and keeps your insights clear. Larger markets like the US, AU, IN, UK, DE, IT, and ES typically offer enough regions to support multiple concurrent tests.
Risks of too many concurrent tests
Running many tests at once can lead to:- Understated results: The true impact of your marketing might appear smaller than it is because the tests interfere with one another.
- Confusing outcomes: Overlapping experiments can produce results that are hard to interpret or may mix effects from different tests.
- Attribution challenges: It becomes more difficult to determine which experiment is responsible for specific changes in performance.