Understand test results
Learn how to interpret key metrics, graphs, and insights from Dema’s incrementality test results.
Dema’s test results provide a clear breakdown of performance between your treatment group (regions where marketing spend is modified) and control group (regions where spend continues as usual). This page helps you evaluate whether your marketing efforts drove true incremental lift in key metrics such as sales and profit.
Test types and group definitions
Lift Test
The treatment group consists of regions where ads are paused or reduced (holdout treatment), while the control group continues business as usual.
New Channel Test
The treatment group consists of regions receiving ads in the new channel, while the control group continues with existing channels only.
Key metrics explained
Core metrics
-
Spend
- The total advertising spend during the test period, shown separately for treatment and control groups
- This helps quantify the investment used to drive the observed lift
-
Net sales
- The total sales observed in the treatment group compared to the control group
- The difference between these values is a core indicator of incrementality
-
Incremental value (net sales)
- The additional sales generated by the treatment group beyond what the control group achieved
- This measures the true incremental impact of your ads
-
Incremental factor (incremental ROAS)
- The ratio of incremental value to spend. Depending on the target variable, this represents:
- ROAS: If the target variable is sales
- epROAS: If the target variable is GP2 or GP3 (net profit). This ensures you measure true efficiency based on the selected outcome metric
Interpreting the graphs
Net sales graph
This chart shows how net sales evolved over time for both the treatment and control groups. Key insights include:
- Parallel trends in pre-test periods: This confirms that the groups were well-matched before the experiment began
- Divergence during the test period: A clear gap between the treatment and control groups indicates incremental lift from marketing activities
- Confidence intervals on the graph: The shaded areas around the lines represent the confidence intervals, which show the range of possible values for the true impact. Narrower intervals suggest higher precision
Causal effect graph
This chart shows the cumulative causal effect of your marketing efforts over time:
- Cumulative impact: The total incremental sales or profit generated by the treatment group compared to the control group
- Stability during post-treatment: If a post-treatment period is included, this helps capture delayed effects of your marketing efforts
- Confidence intervals in cumulative results: The shaded areas provide a range of possible outcomes for the causal effect, helping to interpret the certainty of the observed lift
Key terms and FAQs
What is a confidence interval?
A confidence interval (C.I.) represents the range of values within which the true incremental impact is likely to fall. Narrower intervals indicate more precise results, while wider intervals suggest more variability or uncertainty in the data.
Example: If the C.I. for incremental sales is 100-500 units, the true impact is likely within this range, though the point estimate (e.g., 300 units) is considered the best approximation.
How do confidence intervals relate to treatment group size?
The width of the confidence interval depends on the size of the treatment group:
- Larger treatment group (e.g., 50%): More precise results but affects more regions
- Smaller treatment group (e.g., 10%): Less precision but minimizes business impact
What is the incremental factor?
The incremental factor represents the incremental ROAS, calculated as the incremental value divided by spend:
- Incremental ROAS: If the target variable is sales
- Incremental epROAS: If the target variable is net profit (e.g., GP2 or GP3)
Example: If the incremental value is $10,000 and the spend is $5,000, the incremental ROAS is 2x, meaning you generated $2 in value for every $1 spent.
Drawing actionable insights
If results show significant lift:
- Validate platform metrics: Compare test results with platform-reported performance to ensure consistency.
- Optimize for profit: Focus on campaigns or regions with the highest incremental ROAS to maximize profitability.
If confidence intervals overlap zero:
- Reassess power and duration: Consider increasing the treatment group size or extending the test to improve precision.
- Check for external factors: Ensure that seasonal events or regional differences did not impact results.
If lift is minimal or negative:
- Evaluate creative or targeting: Test new ad formats, messaging, or audience segments.
- Test alternative channels: Explore whether shifting spend to other platforms could improve results.
By understanding and interpreting Dema’s test results, you can make data-driven decisions that optimize your marketing spend and drive true business impact.