Terminology

​

Test Structure Terms

​

Treatment and Control Groups

Treatment Group Regions where marketing spend is modified for the experiment. There are three types of treatment:

• Holdout treatment: Regions where ads are paused or reduced
• New channel treatment: Regions where new channel ads are added
• Scaling treatment: Regions where marketing spend is increased or decreased

Control Group Regions where marketing spend continues as usual (business as usual). Used as the baseline for comparing treatment effects.

​

Geographic Terms

Commute Zones Clusters of zip codes that represent real-world consumer movement patterns, accounting for how people shop across city boundaries. Regions Geographic areas (typically groups of zip codes) used to segment test and control groups.

​

Statistical Terms

​

Confidence and Precision

Confidence Interval The range of values within which the true incremental impact is likely to fall, indicating the precision of results. MDE (Minimum Detectable Effect) The smallest change in performance that can be reliably detected by the test given its design.

​

Experiment Metrics

Treatment Group Proportion The percentage of regions assigned to the treatment group (e.g., 13%). A lower proportion means fewer regions are affected by the test intervention. Estimated Lift The predicted percentage change in performance (e.g., -9.96% indicates an expected decrease in performance when ads are paused in a lift test). Minimum Detectable Effect (MDE) The smallest percentage change the test can reliably detect (e.g., -10% means the test can identify true effects of this size or larger). It describes the design’s sensitivity — not the lift the model measured. Average Detected Lift (shown in the app as Minimum potentially detectable lift) The average lift the synthetic-control model actually recovered during back-testing at the smallest effect size that reaches the power threshold — a measure of the design’s sensitivity (the smallest lift it could potentially detect), not what the test inferred. Distinct from the MDE: the MDE is the smallest effect the design can detect, while average detected lift is what the model measured when a known effect of that size was injected. Power The probability that the test detects a real effect of a given size when one is present. Higher power means the design is more likely to catch a genuine lift. Treatment Group Correlation How well the treatment and control groups match historically (e.g., 94.61%). Higher correlation (>90%) indicates well-matched groups for reliable comparison. P-value The probability of observing the measured difference if there were truly no effect (the null hypothesis). A low p-value means a gap this large would rarely occur by chance.

​

Model Quality Diagnostics

These metrics distinguish a genuinely reliable market split from one that fit the past too closely (overfitting). See Analyze a suggested experiment for how to read them together. False Positive Rate How often the model detected a lift when none was applied, across historical A/A back-tests. Good: at or below the experiment’s significance level (alpha). Lift Estimation Error How far the model’s detected lift was from a known lift injected during back-testing. Good: close to 0%. Out-of-sample Prediction Error On periods the model didn’t train on, how far its prediction was from actual sales, as a share of average sales. Cannot be inflated by overfitting. Good: low (e.g. under ~10%). Generalization Score In-sample error divided by out-of-sample error, scored from 0 to 1 (higher is better). Good: close to 1 (the held-out fit is about as good as the pre-period fit — it generalizes); near 0 signals overfitting. Out-of-sample Stability How consistent the out-of-sample error stays as the back-testing window slides through history, scored from 0 to 1. Higher is better: above 0.5 stable, 0.3–0.5 moderate, under 0.3 fragile. Effective Control Markets The effective number of control regions the synthetic control actually leans on (~N of M available). A low number means the comparison rests on few markets and is more fragile. Back-testing / Lookback Windows How many times the proposed match was re-tested on earlier slices of history; each window shifts back by one period. Typical: 3.

​

Analysis Methods

Synthetic Control A weighted blend of control regions created to best match the treatment region’s historical performance patterns. Incremental Factor The ratio of incremental value to spend, representing either incremental ROAS or epROAS depending on the target variable.

​

Test Types

​

Platform-Specific Terms

User-level Treatment Used in platform studies (like Meta’s Conversion Lift), where individual users are randomly selected to not see ads. Regional Treatment Used in Dema’s geo-testing, where entire geographic regions have their marketing spend modified.

​

Time Periods

Pre-treatment Period The baseline period before the test begins, used to verify group matching. Post-treatment Window The observation period after the test ends to capture delayed effects of marketing.