Structuring tests
Learn how to design and structure incrementality experiments in Dema to measure true business impact.
Experiment design
Dema supports two core types of tests to help you measure and optimize marketing performance:
1. Lift test
A test where the treatment group has ads paused or reduced (holdout treatment) to measure the true incremental impact of your marketing efforts.
Example: A brand runs ads on a platform like Meta and wants to see if those ads drive incremental sales. Dema selects treatment regions where ads will be paused, while control regions continue as usual.
2. New channel test
A test where the treatment group receives ads in a new channel to measure its incremental contribution to sales or profit.
Example: A company experiments with TikTok ads to determine if they deliver incremental value beyond existing marketing efforts.
How to set up a test
Click 'New experiment'
Navigate to the experiment dashboard and select “New experiment” to start setting up your test.
Fill in the experiment details
- Type of test: Choose between “Lift test” or “New channel test” based on your testing goal
- Experiment name: Provide a descriptive name to identify your experiment
- Storefront: Select the relevant storefront for the test
- Country: Choose the geographic location where the test will run
- Channel and funnel: Select the marketing channel (e.g., Google, Meta) and the relevant funnel. You can select a specific funnel (e.g., Upper Funnel) or “All funnels” if running campaigns like Meta ASC
- Target KPIs: Choose between “Gross Sales” or “Net Gross Profit 2 (GP2)”. If you select GP2, you’ll set an effective ROAS (epROAS) target
- Test period: Set the start date and duration (typically 6-8 weeks)
Review and create
After filling in the details, review your selections and click “Create experiment.” Dema will use this information to suggest statistically optimized experiment designs.
How test length and ROAS/epROAS affect test structure
- Shorter tests (4-6 weeks): Requires larger treatment group to achieve statistical significance
- Longer tests (8+ weeks): Allows smaller treatment group while maintaining statistical power
Recommendation: Choose the longest feasible test period to maximize precision while minimizing the required treatment group size.
Balancing treatment group size and precision
The treatment group size impacts both accuracy and opportunity cost:
- Larger treatment groups (30-50%): Provide greater precision but affect more regions during the test
- Smaller treatment groups (10-20%): Minimize disruption but may result in wider confidence intervals
Dema dynamically calculates the optimal treatment group size based on your test length, expected ROAS/epROAS, and the variability in your historical data.
Best practices for all tests
Active campaign management
To minimize variables and maintain a business-as-usual strategy during testing, ensure that any adjustments—such as deploying or pausing ads—are applied consistently across both treatment and control groups. This ensures that performance differences are driven solely by the test variable, not by operational discrepancies.
Budget consistency
Keep budgets evenly distributed to isolate the effect of the variable being tested. While budget variations can occur, Dema’s analysis adjusts for these differences to provide meaningful insights into incremental lift and efficiency.
Delayed response effects
Some channels may exhibit delayed responses to marketing actions. Dema supports a post-treatment observation window to capture lagging conversions or sales, ensuring the full impact of your marketing efforts is accurately measured.