Learn how to run multiple 2-cell experiments concurrently in Dema and understand the risks and best practices.
Concurrent testing means running multiple experiments at the same time. While this approach can speed up insights, it also increases the chance that tests might affect one another, potentially influencing your results.
When running more than one 2-cell experiment in the same country, it’s important to balance faster insights with the risk of test overlap. Here’s our view on how risk increases as you add more experiments in a market:
Two 2-cell experiments
Generally low risk, though it’s wise to check for any overlapping areas between test groups.
Three 2-cell experiments
Higher risk. We recommend reviewing the overlap between each pair of tests to help ensure one experiment does not skew the results of another.
More than three experiments
Considerable risk. Only proceed if you understand that increased interference may blur the true effects, and you’re prepared to accept that risk for business-critical decisions.
Note: These guidelines apply to a single country. For example, you might run two 2-cell tests in the United States and another test in the United Kingdom.
For countries or regions with fewer than 50 unique areas, we strongly recommend running a single 2-cell experiment. This helps prevent overlap and keeps your insights clear. Larger markets like the US, AU, IN, UK, DE, IT, and ES typically offer enough regions to support multiple concurrent tests.
Reducing the size of your control group can help lower the risk of overlap, as long as it remains large enough to provide a solid baseline for comparison.
Dema offers tools to review the extent to which your experiments overlap. Work with your Customer Success Manager to use these insights and adjust tests as needed to minimise interference.
Keeping budgets similar across experiments not only creates a fair comparison but also helps reduce interference between tests. Note that if spend is too low, it may be harder to detect a significant impact.
If you run a conversion lift study on a platform (like Meta or Google) at the same time as a geo-based test, keep in mind that it counts as one of your concurrent tests. Evaluate the overlap between these tests carefully to avoid unintended interactions.
By understanding and managing these factors, you can run multiple 2-cell experiments concurrently in Dema while maintaining clear and actionable insights. For more detailed guidance or if you have specific concerns, please reach out to your Dema specialist.