Skip to content
Toposort Logo
Knowledge Base

Incrementality Measurement

Overview

Incrementality measurement determines the true causal impact of advertising by isolating conversions that would not have occurred without ad exposure. Per IAB/MRC Retail Media Measurement Guidelines, incrementality testing provides the most accurate measure of advertising effectiveness.

What is Incrementality?

Incremental lift represents sales or conversions directly caused by advertising, excluding those that would have happened organically. This differs from attribution, which assigns credit for conversions but doesn’t prove causation.

Testing Methodologies

1. Randomized Controlled Trials (RCTs)

The gold standard for incrementality measurement:

  1. Random Assignment: Users randomly divided into test (see ads) and control (no ads) groups
  2. Campaign Execution: Test group exposed to advertising while control group is held out
  3. Measurement: Compare conversion rates between groups
  4. Calculation: Incremental Lift = (Test Conversions - Control Conversions) / Control Conversions

Advantages:

  • Most accurate causal measurement
  • Eliminates selection bias
  • Clear statistical significance

Limitations:

  • Requires holdout group (lost opportunity)
  • Minimum sample size needed
  • May not reflect real-world conditions

2. Synthetic Control Methods

Creates artificial control group using historical data and machine learning:

Data Collection

Gather historical conversion patterns and user characteristics

Model Training

Build predictive model of expected conversions without advertising

Comparison

Compare actual results to synthetic control predictions

Lift Calculation

Measure difference between actual and predicted outcomes

Advantages:

  • No holdout group required
  • Can be applied retroactively
  • Continuous measurement possible

Limitations:

  • Requires robust historical data
  • Model accuracy affects results
  • Assumptions may not hold in all cases

3. Matched Market Tests

Compares similar geographic markets with different ad exposure:

  1. Market Selection: Identify comparable markets by demographics, sales patterns
  2. Test Design: Run campaigns in test markets, hold out control markets
  3. Analysis: Compare lift between matched market pairs
  4. Scaling: Extrapolate results to full population

Advantages:

  • Real-world conditions maintained
  • Good for regional campaigns
  • Can test different strategies

Limitations:

  • Finding truly comparable markets difficult
  • External factors may affect results
  • Geographic spillover possible

Implementation in Topsort

Enabling Incrementality Tests

Marketplaces can configure incrementality testing through:

{
  "test_configuration": {
    "methodology": "rct",
    "test_split": 0.8,  // 80% test, 20% control
    "minimum_sample_size": 10000,
    "measurement_period_days": 30,
    "stratification": ["user_segment", "geographic_region"]
  }
}

Test Setup Process

  1. Define Objectives

    • Primary KPI (sales, new customers, etc.)
    • Expected lift range
    • Required confidence level

  2. Calculate Sample Size

    • Use statistical power calculators
    • Account for expected variance
    • Include buffer for incomplete data

  3. Configure Test Parameters

    • Test/control split ratio
    • Stratification variables
    • Measurement window

  4. Monitor Execution

    • Check randomization balance
    • Track exposure rates
    • Validate data quality

  5. Analyze Results

    • Calculate incremental lift
    • Determine statistical significance
    • Generate confidence intervals

Reporting Incrementality

Standard Metrics

Reports include:

  • Incremental Conversions: Additional conversions caused by advertising
  • Incremental Revenue: Revenue directly attributable to ad exposure
  • iROAS: Incremental Return on Ad Spend (incremental revenue / ad spend)
  • Lift Percentage: Relative increase over baseline
  • Confidence Interval: Statistical range of true effect

Sample Report Format

INCREMENTALITY TEST RESULTS
━━━━━━━━━━━━━━━━━━━━━━━━━━
Test Type: Randomized Controlled Trial
Test Period: Oct 1 - Oct 31, 2024
Sample Size: 50,000 users (40,000 test / 10,000 control)


RESULTS:
─────────────────────────────
Test Group Conversion Rate: 4.2%
Control Group Conversion Rate: 3.1%
Incremental Lift: 35.5% (95% CI: 28.2% - 42.8%)
Statistical Significance: p < 0.001


Incremental Conversions: 440
Incremental Revenue: $44,000
iROAS: 4.4x

Best Practices

Test Design

  1. Pre-registration

    • Document hypothesis before testing
    • Define success metrics upfront
    • Commit to test duration

  2. Randomization Quality

    • Verify random assignment
    • Check for pre-test differences
    • Use stratification for balance

  3. Sample Size

    • Calculate required size for desired power
    • Account for attribution window
    • Include non-compliance buffer

Common Pitfalls to Avoid

Advanced Considerations

Multi-Touch Incrementality

For campaigns with multiple touchpoints:

  1. Sequential Testing: Measure incremental impact of each additional exposure
  2. Interaction Effects: Assess how different ad formats work together
  3. Diminishing Returns: Identify optimal frequency caps

Long-term Effects

Measuring beyond immediate conversions:

  • Customer Lifetime Value: Track incremental CLV over time
  • Brand Metrics: Survey-based measurement of awareness/consideration
  • Halo Effects: Impact on non-advertised products

Cross-Channel Coordination

When running omnichannel campaigns:

  • Coordinate test/control groups across channels
  • Measure total incremental impact
  • Identify channel interaction effects

Integration with Attribution

Complementary Insights

Attribution Answers

“Which ads get credit for conversions?”

Incrementality Answers

“How many conversions were caused by ads?”

Combined Reporting

Best practice includes both metrics:

  • Attribution for tactical optimization
  • Incrementality for strategic decisions
  • Reconciliation of differences

API Access

Requesting Test Results

// Fetch incrementality test results
const testResults = await fetch('/api/incrementality/results', {
  method: 'POST',
  body: JSON.stringify({
    campaign_id: 'camp_123',
    test_id: 'test_456',
    include_confidence_intervals: true,
    breakdown_by: ['product_category', 'user_segment']
  })
});

Response Format

{
  "test_summary": {
    "methodology": "rct",
    "test_group_size": 40000,
    "control_group_size": 10000,
    "measurement_period": "2024-10-01 to 2024-10-31"
  },
  "results": {
    "incremental_lift": 0.355,
    "confidence_interval": [0.282, 0.428],
    "p_value": 0.0001,
    "incremental_conversions": 440,
    "incremental_revenue": 44000,
    "iroas": 4.4
  },
  "quality_checks": {
    "randomization_balance": "pass",
    "sample_size_adequate": true,
    "statistical_power": 0.95
  }
}

Frequently Asked Questions

  1. How long should incrementality tests run?
    • Minimum 2-4 weeks to capture full purchase cycle, longer for considered purchases.
  2. What’s the minimum sample size needed?
    • Depends on expected lift and baseline conversion rate. Generally 10,000+ users per group.
  3. Can incrementality be measured without holdouts?
    • Yes, using synthetic controls or matched markets, though RCTs remain most accurate.
  4. How often should incrementality be tested?
    • Quarterly for ongoing campaigns, or when significant changes occur in strategy or market conditions.