build: 2026-10-19 20:32 UTC

Experimentation and Causal Inference

Christopher J. Collins

 

Chapter 6 — Experimentation and Causal Inference

Why before-and-after averages mislead, and what a credible engineering experiment actually requires.

Christopher J. Collins · Chapter Overview

The question that actually matters

Review latency fell after the AI reviewer rolled out. Success?

Plausible alternatives to "the tool worked":

  • a quieter release period

  • simpler repositories adopted first

  • reviewer staffing changed

  • CI got faster in the same window

  • novelty effect on newly-observed engineers

  • low-performing teams opted out

The causal question is not whether the metric changed — it is whether it would have changed differently without the intervention.

Potential outcomes and the missing counterfactual

For unit \(i\):

\[Y_i(1) = \text{outcome under treatment}, \quad Y_i(0) = \text{outcome without}\]
\[\tau_i = Y_i(1) - Y_i(0)\]

Only one potential outcome is ever observed.

Every experimental and quasi-experimental design is a disciplined strategy for estimating the missing one.

The DAG makes assumptions visible

causal dag

  • T — adoption of the intervention

  • M — mediator (allocated reviewer time)

  • Y — outcome

  • U — confounder (team skill, repository maturity)

Randomising T cuts the arrow into it. Adjusting for U blocks the backdoor. Neither happens by accident.

Define the treatment — precisely

"Use AI review" is not a treatment. A treatment specifies:

  • vendor, model, model version

  • prompt and policy configuration

  • repositories and file types in scope

  • advisory vs. blocking findings

  • ranking and suppression rules

  • human-review requirements

  • exposure start, end, and fallback

Without treatment fidelity, results cannot be reproduced or generalised.

Assignment unit vs. analysis unit

The assignment unit receives the treatment. The analysis unit contributes the outcome. They can differ.

Engineer / PR

efficient, contamination-prone

Team / repository

reduces spillover, costs power

Engineering Change

matches multi-repository outcomes

Time interval

switchback designs

Choose against the mechanism, the interference risk, and the decision — not the convenience.

A toolkit of designs

Experiment and causal study lifecycle

Randomised: RCT · cluster-randomised · switchback · staggered rollout.

Quasi-experimental: difference-in-differences · interrupted time series · synthetic control · matching · regression discontinuity.

Randomisation is preferred when feasible; quasi-experimental designs earn their keep when it is not.

Difference-in-differences, in one line

\[\widehat{\tau}_{DiD} = (\bar{Y}_{T,\text{after}}-\bar{Y}_{T,\text{before}}) - (\bar{Y}_{C,\text{after}}-\bar{Y}_{C,\text{before}})\]

Key assumption: parallel trends — absent treatment, treated and control would have moved similarly.

Inspect pre-trends. Test alternative windows. Report sensitivity to plausible violations. Don’t ship without them.

Primary outcomes and guardrails

Pre-declare a small number of primary outcomes:

  • quality-adjusted cycle time

  • human review effort

  • severe escaped-defect rate

  • cost per accepted Engineering Change

Pre-declare guardrails: severe defects · security escapes · rollback rate · author understanding · cognitive reload · tool trust · infrastructure cost.

A faster process is not successful if a guardrail deteriorates beyond tolerance.

Power, effect size, and the honest MDE

Statistical significance is not practical value. With enough data, trivia become "significant".

Report:

  • the absolute and relative effect

  • the uncertainty interval

  • the minimum detectable effect — before collecting data

  • the economic consequence

A negative result at an unstated MDE is not evidence of no effect. It is evidence of insufficient data.

Power analyses belong in the pre-registration, not the discussion.

Exposure, compliance, and interference

Assignment is not exposure. Interference is not a nuisance.

  • Record: assignment · eligibility · actual exposure · intensity · fallback · cross-over · protocol deviation

  • Intent-to-treat estimates the effect of offering treatment

  • CACE estimates the effect among users — a different question

  • Shared reviewers, shared libraries, and peer influence violate SUTVA by default

Spillover can be harm. It can also be the point — knowledge transfer from AI to team.

The worked example, distilled

An AI reviewer rolls out to half of 240 teams. Six weeks later:

Estimate

Value

What it answers

Naive before-after

−1.9 h

Nothing serious

Difference-in-differences

−1.1 h

Effect of offering the tool

Complier-average (CACE)

−1.55 h

Effect among users

Same data. Three answers. Report the wrong estimand and you mis-price the rollout.

Common failure modes

  • Before-and-after attribution — the metric moved; therefore the tool

  • Self-selection — early adopters differ; adoption confounds treatment

  • Underpowered rare outcomes — short tests can’t rule out severe-defect harm

  • Metric substitution — swap the hard outcome for a proxy, then optimise the proxy

  • Peeking — repeated naive checks inflate false positives

  • Ignoring noncompliance — dilutes the treatment, hides operational failure

  • Global averages — hide harm to high-risk populations

  • Unrecorded concurrent changes — attribution becomes impossible

The registry and the platform

Every experiment registered before launch:

  • hypothesis · intervention · assignment/analysis unit · eligible population

  • primary outcomes · guardrails · MDE · stopping rules · decision rule

The platform behind it:

  • eligibility · deterministic assignment · configuration delivery

  • exposure logging · metric + guardrail computation

  • external-events registry · sequential monitoring

  • analysis pipelines · decision records · audit + reproducibility

Assignment must be stable, versioned, and independent of the outcome pipeline.

Key takeaways

  • Causal evaluation asks what would have happened without the intervention

  • Treatment, assignment, analysis, exposure, and outcomes must be defined precisely

  • Randomisation is preferred; quasi-experimental designs are valuable when it isn’t

  • Clustered engineering systems require clustered designs and standard errors

  • Primary outcomes, MDEs, and guardrails belong in the pre-registration

  • External factors, noncompliance, interference, and heterogeneity are central, not peripheral

  • Experimentation is durable platform infrastructure — not a one-off analysis

Next

Chapter 7 — Applications

AI reviewers · merge queues · autonomous agents · portfolio decisions — the framework applied to consequential engineering choices.