Concurrent testing means running multiple experiments at the same time. While this approach can speed up insights, it also increases the chance that tests might affect one another, potentially influencing your results.

​
Managing concurrent 2-cell experiments

When running more than one 2-cell experiment in the same country, it’s important to balance faster insights with the risk of test overlap. Here’s our view on how risk increases as you add more experiments in a market:

Two 2-cell experiments

Generally low risk, though it’s wise to check for any overlapping areas between test groups.

Three 2-cell experiments

Higher risk. We recommend reviewing the overlap between each pair of tests to help ensure one experiment does not skew the results of another.

More than three experiments

Considerable risk. Only proceed if you understand that increased interference may blur the true effects, and you’re prepared to accept that risk for business-critical decisions.

Note: These guidelines apply to a single country. For example, you might run two 2-cell tests in the United States and another test in the United Kingdom.

​
Guidelines for markets with few regions

For countries or regions with fewer than 50 unique areas, we strongly recommend running a single 2-cell experiment. This helps prevent overlap and keeps your insights clear. Larger markets like the US, AU, IN, UK, DE, IT, and ES typically offer enough regions to support multiple concurrent tests.

​
Risks of too many concurrent tests

Running many tests at once can lead to:

  • Understated results: The true impact of your marketing might appear smaller than it is because the tests interfere with one another.
  • Confusing outcomes: Overlapping experiments can produce results that are hard to interpret or may mix effects from different tests.
  • Attribution challenges: It becomes more difficult to determine which experiment is responsible for specific changes in performance.

​
Strategies to reduce risk

​
Smaller control group size

Reducing the size of your control group can help lower the risk of overlap, as long as it remains large enough to provide a solid baseline for comparison.

​
Overlap analysis

Dema offers tools to review the extent to which your experiments overlap. Work with your Customer Success Manager to use these insights and adjust tests as needed to minimise interference.

​
Balanced budget allocation

Keeping budgets similar across experiments not only creates a fair comparison but also helps reduce interference between tests. Note that if spend is too low, it may be harder to detect a significant impact.

​
Running platform tests alongside geo tests

If you run a conversion lift study on a platform (like Meta or Google) at the same time as a geo-based test, keep in mind that it counts as one of your concurrent tests. Evaluate the overlap between these tests carefully to avoid unintended interactions.

By understanding and managing these factors, you can run multiple 2-cell experiments concurrently in Dema while maintaining clear and actionable insights. For more detailed guidance or if you have specific concerns, please reach out to your Dema specialist.