Analyze a suggested experiment

​

Understanding the suggested experiment table

When Dema generates a suggested experiment, it provides a detailed table to help you understand how the test is structured and what to expect. This table breaks down the test’s estimated performance metrics and key characteristics of the treatment and control groups.

Here’s what each column represents:

​

Experiment statistics

1. Estimated spend

   • The amount of advertising spend that will be removed (in a decrease spend test) or added (in an increase spend test) in the treatment group.
   • This helps you understand the magnitude of the change being tested.

2. Spend previous period

   • The amount of advertising spend during the same time period as the planned test, but before the test begins.
   • This serves as a baseline to compare current spending and measure changes.

3. Estimated total gross sales

   • The total gross sales expected across all regions in the test, including both treatment and control groups.
   • This provides an overview of total expected revenue during the test period.

4. Estimated incremental gross sales

   • The amount of sales expected to be lost (in a decrease spend test) or gained (in an increase spend test) in the treatment group due to the test.
   • This is calculated based on the ROAS or epROAS applied to the estimated spend changes.

5. Treatment group proportion

   • The percentage of regions assigned to the treatment group versus the control group.
   • A lower proportion indicates that fewer regions will experience the test intervention, helping to reduce potential disruptions.

6. Estimated lift

   • The percentage change in performance (e.g., sales, profit) expected as a result of the test.
   • This metric indicates the potential impact of the experiment.

7. Minimum detectable effect (MDE)

   • The smallest measurable change that can be reliably detected by the test.
   • This reflects the sensitivity of the test design; smaller MDE values indicate higher sensitivity.

8. Treatment group correlation

   • The correlation between treatment and control groups in terms of historical performance.
   • Higher correlation values (e.g., above 90%) indicate that the groups are well-matched, which improves the reliability of the test.

​

Geographical targeting

​

How to use this information

​

Interpreting estimated metrics

• Understand the impact on spend and sales: Compare the estimated spend and incremental gross sales to gauge the scale and potential outcomes of the test.
• Plan for changes in overall performance: Use the estimated total gross sales to anticipate any shifts in overall revenue during the test period.
• Assess test sensitivity*: Review the MDE and treatment group correlation to ensure the test design is robust and likely to yield actionable insights.*

​

Preparing for the test

By understanding the suggested experiment table, you can effectively plan and execute tests that drive meaningful insights into the incremental impact of your marketing efforts.