Concurrent tests
Learn how to run multiple 2-cell experiments concurrently in Dema and understand the risks and best practices.
Concurrent testing means running multiple experiments at the same time. While this approach can speed up insights, it also increases the chance that tests might affect one another, potentially influencing your results.
Managing concurrent 2-cell experiments
When running more than one 2-cell experiment in the same country, it’s important to balance faster insights with the risk of test overlap. Here’s our view on how risk increases as you add more experiments in a market:
Two 2-cell experiments
Generally low risk, though it’s wise to check for any overlapping areas between test groups.
Three 2-cell experiments
Higher risk. We recommend reviewing the overlap between each pair of tests to help ensure one experiment does not skew the results of another.
More than three experiments
Considerable risk. Only proceed if you understand that increased interference may blur the true effects, and you’re prepared to accept that risk for business-critical decisions.
Note: These guidelines apply to a single country. For example, you might run two 2-cell tests in the United States and another test in the United Kingdom.
Guidelines for markets with few regions
For countries or regions with fewer than 50 unique areas, we strongly recommend running a single 2-cell experiment. This helps prevent overlap and keeps your insights clear. Larger markets like the US, AU, IN, UK, DE, IT, and ES typically offer enough regions to support multiple concurrent tests.
Risks of too many concurrent tests
Running many tests at once can lead to:
- Understated results: The true impact of your marketing might appear smaller than it is because the tests interfere with one another.
- Confusing outcomes: Overlapping experiments can produce results that are hard to interpret or may mix effects from different tests.
- Attribution challenges: It becomes more difficult to determine which experiment is responsible for specific changes in performance.
Strategies to reduce risk
Smaller control group size
Reducing the size of your control group can help lower the risk of overlap, as long as it remains large enough to provide a solid baseline for comparison.
Overlap analysis
Dema offers tools to review the extent to which your experiments overlap. Work with your Customer Success Manager to use these insights and adjust tests as needed to minimise interference.
Balanced budget allocation
Keeping budgets similar across experiments not only creates a fair comparison but also helps reduce interference between tests. Note that if spend is too low, it may be harder to detect a significant impact.
Running platform tests alongside geo tests
If you run a conversion lift study on a platform (like Meta or Google) at the same time as a geo-based test, keep in mind that it counts as one of your concurrent tests. Evaluate the overlap between these tests carefully to avoid unintended interactions.
By understanding and managing these factors, you can run multiple 2-cell experiments concurrently in Dema while maintaining clear and actionable insights. For more detailed guidance or if you have specific concerns, please reach out to your Dema specialist.