Terminology

​

Test Structure Terms

​

Treatment and Control Groups

Treatment Group Regions where marketing spend is modified for the experiment. There are two types of treatment:

• Holdout treatment: Regions where ads are paused or reduced
• New channel treatment: Regions where new channel ads are added

Control Group Regions where marketing spend continues as usual (business as usual). Used as the baseline for comparing treatment effects.

​

Geographic Terms

Commute Zones Clusters of zip codes that represent real-world consumer movement patterns, accounting for how people shop across city boundaries.

Regions Geographic areas (typically groups of zip codes) used to segment test and control groups.

​

Statistical Terms

​

Confidence and Precision

Confidence Interval The range of values within which the true incremental impact is likely to fall, indicating the precision of results.

MDE (Minimum Detectable Effect) The smallest change in performance that can be reliably detected by the test given its design.

​

Experiment Metrics

Treatment Group Proportion The percentage of regions assigned to the treatment group (e.g., 13%). A lower proportion means fewer regions are affected by the test intervention.

Estimated Lift The predicted percentage change in performance (e.g., -9.96% indicates an expected decrease in performance when ads are paused in a lift test).

Minimum Detectable Effect (MDE) The smallest percentage change the test can reliably detect (e.g., -9.96% means the test can identify true effects of this size or larger).

Treatment Group Correlation How well the treatment and control groups match historically (e.g., 94.61%). Higher correlation (>90%) indicates well-matched groups for reliable comparison.

​

Analysis Methods

Synthetic Control A weighted blend of control regions created to best match the treatment region’s historical performance patterns.

Incremental Factor The ratio of incremental value to spend, representing either incremental ROAS or epROAS depending on the target variable.

​

Test Types

​

Platform-Specific Terms

User-level Treatment Used in platform studies (like Meta’s Conversion Lift), where individual users are randomly selected to not see ads.

Regional Treatment Used in Dema’s geo-testing, where entire geographic regions have their marketing spend modified.

​

Time Periods

Pre-treatment Period The baseline period before the test begins, used to verify group matching.

Post-treatment Window The observation period after the test ends to capture delayed effects of marketing.