Skip to main content

​
Overview

Incrementality measurement determines the true causal impact of advertising by isolating conversions that would not have occurred without ad exposure. Per IAB/MRC Retail Media Measurement Guidelines, incrementality testing provides the most accurate measure of advertising effectiveness.

​
What is Incrementality?

Incremental lift represents sales or conversions directly caused by advertising, excluding those that would have happened organically. This differs from attribution, which assigns credit for conversions but doesn’t prove causation.
Example: If 100 people who saw an ad made purchases, but testing shows 70 would have purchased anyway, the incremental impact is 30 conversions (30% lift).

​
Testing Methodologies

​
1. Randomized Controlled Trials (RCTs)

The gold standard for incrementality measurement:
1

Random Assignment: Users randomly divided into test (see ads) and control (no ads) groups

2

Campaign Execution

Test group exposed to advertising while control group is held out
3

Measurement

Compare conversion rates between groups
4

Calculation

Incremental Lift = (Test Conversions - Control Conversions) / Control Conversions
Advantages:
  • Most accurate causal measurement
  • Eliminates selection bias
  • Clear statistical significance
Limitations:
  • Requires holdout group (lost opportunity)
  • Minimum sample size needed
  • May not reflect real-world conditions

​
2. Synthetic Control Methods

Creates artificial control group using historical data and machine learning:

Data Collection

Gather historical conversion patterns and user characteristics

Model Training

Build predictive model of expected conversions without advertising

Comparison

Compare actual results to synthetic control predictions

Lift Calculation

Measure difference between actual and predicted outcomes
Advantages:
  • No holdout group required
  • Can be applied retroactively
  • Continuous measurement possible
Limitations:
  • Requires robust historical data
  • Model accuracy affects results
  • Assumptions may not hold in all cases

​
3. Matched Market Tests

Compares similar geographic markets with different ad exposure:
  1. Market Selection: Identify comparable markets by demographics, sales patterns
  2. Test Design: Run campaigns in test markets, hold out control markets
  3. Analysis: Compare lift between matched market pairs
  4. Scaling: Extrapolate results to full population
Advantages:
  • Real-world conditions maintained
  • Good for regional campaigns
  • Can test different strategies
Limitations:
  • Finding truly comparable markets difficult
  • External factors may affect results
  • Geographic spillover possible

​
Implementation in Topsort

​
Enabling Incrementality Tests

Marketplaces can configure incrementality testing through:
{
  "test_configuration": {
    "methodology": "rct",
    "test_split": 0.8, // 80% test, 20% control
    "minimum_sample_size": 10000,
    "measurement_period_days": 30,
    "stratification": ["user_segment", "geographic_region"]
  }
}

​
Test Setup Process

1

Define Objectives

Define your primary KPI (sales, new customers, etc.), expected lift range, and required confidence level.
2

Calculate Sample Size

Use statistical power calculators, account for expected variance, and include buffer for incomplete data.
3

Configure Test Parameters

Set test/control split ratio, stratification variables, and measurement window.
4

Monitor Execution

Check randomization balance, track exposure rates, and validate data quality.
5

Analyze Results

Calculate incremental lift, determine statistical significance, and generate confidence intervals.

​
Reporting Incrementality

​
Standard Metrics

Reports include:
  • Incremental Conversions: Additional conversions caused by advertising
  • Incremental Revenue: Revenue directly attributable to ad exposure
  • iROAS: Incremental Return on Ad Spend (incremental revenue / ad spend)
  • Lift Percentage: Relative increase over baseline
  • Confidence Interval: Statistical range of true effect

​
Sample Report Format

INCREMENTALITY TEST RESULTS
━━━━━━━━━━━━━━━━━━━━━━━━━━
Test Type: Randomized Controlled Trial
Test Period: Oct 1 - Oct 31, 2024
Sample Size: 50,000 users (40,000 test / 10,000 control)

RESULTS:
─────────────────────────────
Test Group Conversion Rate: 4.2%
Control Group Conversion Rate: 3.1%
Incremental Lift: 35.5% (95% CI: 28.2% - 42.8%)
Statistical Significance: p < 0.001

Incremental Conversions: 440
Incremental Revenue: $44,000
iROAS: 4.4x

​
Best Practices

​
Test Design

  1. Pre-registration
    • Document hypothesis before testing
    • Define success metrics upfront
    • Commit to test duration
  2. Randomization Quality
    • Verify random assignment
    • Check for pre-test differences
    • Use stratification for balance
  3. Sample Size
    • Calculate required size for desired power
    • Account for attribution window
    • Include non-compliance buffer

​
Common Pitfalls to Avoid

Avoid these common mistakes: - Stopping tests early based on interim results - Changing test parameters mid-flight - Ignoring spillover effects between groups - Using insufficiently powered tests - Not accounting for seasonality

​
Advanced Considerations

​
Multi-Touch Incrementality

For campaigns with multiple touchpoints:
  1. Sequential Testing: Measure incremental impact of each additional exposure
  2. Interaction Effects: Assess how different ad formats work together
  3. Diminishing Returns: Identify optimal frequency caps

​
Long-term Effects

Measuring beyond immediate conversions:
  • Customer Lifetime Value: Track incremental CLV over time
  • Brand Metrics: Survey-based measurement of awareness/consideration
  • Halo Effects: Impact on non-advertised products

​
Cross-Channel Coordination

When running omnichannel campaigns:
  • Coordinate test/control groups across channels
  • Measure total incremental impact
  • Identify channel interaction effects

​
Integration with Attribution

​
Complementary Insights

Attribution Answers

“Which ads get credit for conversions?”

Incrementality Answers

“How many conversions were caused by ads?”

​
Combined Reporting

Best practice includes both metrics:
  • Attribution for tactical optimization
  • Incrementality for strategic decisions
  • Reconciliation of differences

​
API Access

​
Requesting Test Results

// Fetch incrementality test results
const testResults = await fetch("/api/incrementality/results", {
  method: "POST",
  body: JSON.stringify({
    campaign_id: "camp_123",
    test_id: "test_456",
    include_confidence_intervals: true,
    breakdown_by: ["product_category", "user_segment"],
  }),
});

​
Response Format

{
  "test_summary": {
    "methodology": "rct",
    "test_group_size": 40000,
    "control_group_size": 10000,
    "measurement_period": "2024-10-01 to 2024-10-31"
  },
  "results": {
    "incremental_lift": 0.355,
    "confidence_interval": [0.282, 0.428],
    "p_value": 0.0001,
    "incremental_conversions": 440,
    "incremental_revenue": 44000,
    "iroas": 4.4
  },
  "quality_checks": {
    "randomization_balance": "pass",
    "sample_size_adequate": true,
    "statistical_power": 0.95
  }
}

​
Frequently Asked Questions

  1. How long should incrementality tests run?
    • Minimum 2-4 weeks to capture full purchase cycle, longer for considered purchases.
  2. What’s the minimum sample size needed?
    • Depends on expected lift and baseline conversion rate. Generally 10,000+ users per group.
  3. Can incrementality be measured without holdouts?
    • Yes, using synthetic controls or matched markets, though RCTs remain most accurate.
  4. How often should incrementality be tested?
    • Quarterly for ongoing campaigns, or when significant changes occur in strategy or market conditions.
Last updated: