Running an Engineering Experiment
Workshop — 90 minutes
Running an Engineering Experiment
A hands-on workshop derived from Chapter 12 of Engineering Intelligence.
Pick a playbook; run the design; write the pre-registration.
90 minutes · 4 hands-on segments · one signed pre-reg per team
Learning outcomes
By the end of the ninety minutes each team will:
Apply a playbook — AI reviewer, merge queue, PR size, CI acceleration, review routing — to a real intervention on your own team
Write a pre-registration — primary outcome, guardrails, MDE, stopping rule, interpretation table
Name and avoid the top failure modes — wrong unit, missing exposure, interference, novelty, seasonality
You will leave with a signed document you could actually launch.
What this workshop is not
This is not a causal-inference lecture. We are not teaching DiD, IV, or synthetic controls today.
We are teaching how to drive one experiment end-to-end from a playbook — the operational document, not the estimator.
If you need the inference technique, the sibling workshop covers it.
The problem
A director walks in. "We’re rolling out the AI reviewer to all 180 backend teams next month. Can you A/B test it?"
You nod. You should not have nodded.
Assignment unit? Reviewers are shared across teams
Compliance? Teams may install but never open the findings
Interference? One team’s PR is another team’s review
Guardrails? What breaks if the tool is wrong
Decision rule? At what effect size do we adopt
"Let’s just A/B test it" is not a design. It is a wish.
A playbook forces the choices
Every playbook names: decision · hypothesis · treatment · control · assignment unit · analysis unit · eligibility · primary outcome · guardrails · exposure · external factors · statistical design · MDE · stopping rules · interpretation risks.
Nothing on the list is optional.
The experiment lifecycle
The playbook governs the design phase. If design is wrong, no amount of clever analysis rescues it later.
The twelve playbooks (menu)
You will pick one of these in Segment 1:
1 | AI pull-request reviewer |
2 | Ranked vs. unranked AI findings |
3 | Merge queue introduction |
4 | CI acceleration |
5 | Risk-based specialist review |
6 | Pull-request size guidance |
7 | Monorepo consolidation |
8 | Automated context restoration |
9 | Human-first vs. AI-first review |
10 | Autonomous maintenance agent |
11 | Branch-protection policy |
12 | Review-queue prioritization |
Segment 1 — pick and hypothesise (15 min)
Task. Choose a playbook. Choose an intervention on your team.
Write, on one sheet:
Decision — what commits or refuses on the basis of this experiment
Treatment — the exact versioned thing
Control — the exact status-quo it replaces
Hypothesis — direction and mechanism, in one sentence
Rule: if the decision does not commit the org to adopt / revise / reject, pick a different intervention.
Report back: the decision sentence.
Assignment vs analysis unit
Assignment unit — receives the treatment (team, repository, engineer, time block)
Analysis unit — contributes an outcome (Engineering Change, PR, service)
They can differ. The mismatch is where most designs die.
Pull-request assignment breaks when reviewers are shared. Team-level assignment costs power. Engineering Change is usually the right analysis unit for workflow interventions.
Exposure is not assignment
Assignment is policy. Exposure is use.
Team A is assigned the AI reviewer
Only 71% of authors open a finding on any given PR
Reporting the average outcome across Team A confuses mandating the tool with using it
Instrument both. The gap between them is the story.
Sample size and MDE
The minimum detectable effect decides whether the experiment can possibly answer your question.
For a two-sample z-test on means:
Work backwards from the smallest effect worth acting on:
effort improvement < 5% → probably not worth vendor cost
< 3% → within noise of a Monday morning
If your throughput can’t reach n in a quarter, redesign.
Guardrails
Guardrails are the outcomes that must not get worse.
They exist because a well-optimised primary outcome can silently break the system.
Merge queue: queue age, urgent-change latency, CI cost
AI reviewer: escaped defects, security findings, cognitive reload, trust calibration
CI acceleration: flaky-test rate, cache correctness, missed defects
Autonomous agent: unrelated-file mods, license violations, escalation rate
Every playbook names its guardrails. Copy that list. Do not shorten it.
Segment 2 — design (25 min)
For your chosen intervention, write:
Assignment unit — and why that unit (interference, shared reviewers, shared infra)
Analysis unit — usually Engineering Change; justify if different
Exposure instrumentation — how you will know it was actually used
Eligibility and exclusions — who is in, who is out, why
Sample size / MDE — declare \(\sigma\), the smallest useful effect \(\delta\), compute n
Duration — n divided by your real throughput
Report back: the assignment unit, and the compliance rate you are willing to tolerate before the ITT number becomes uninterpretable.
Pre-registration — what goes in
Primary outcome — one, quality-adjusted, at the Engineering Change level
Guardrail outcomes — the ones the playbook names
Analysis plan — estimator, adjustment variables, subgroups
Stopping rule — how you handle interim looks
Expected effect direction — sign, rough magnitude, mechanism
Interpretation table — what each outcome pattern means
Pre-analysis plans belong before data, not in the discussion section.
The interpretation table
Decide, before launching, what every outcome means. Four cells:
| Outcome | Reading | Action |
|---|---|---|
Win | Primary improves ≥ MDE, guardrails hold | Adopt; roll out; register decision record |
Null | Primary within ± MDE of zero, guardrails hold | Reject at current cost; revisit at price change or model refresh |
Regression | Primary worse, or a guardrail breached | Reject; investigate mechanism; do not retry the same design |
Can’t tell | CI wider than MDE, or exposure < threshold | Extend, redesign, or abandon — do not report as null |
Stopping rules and peeking
Engineering experiments run continuously. Teams peek.
Repeated naive tests at \(\alpha = 0.05\) inflate false positives fast (1 look 5%, 5 looks ~14%, 10 looks ~19%).
Pick one, pre-declare, do not deviate:
Fixed horizon — commit to n; do not analyse before
Group-sequential (O’Brien-Fleming) — strict early, conventional late
Always-valid (mSPRT / e-value) — safe under continuous monitoring
Segment 3 — write the pre-reg (25 min)
Draft, on one sheet:
Primary outcome — expression and window
Guardrails — from the playbook, verbatim
Analysis plan — estimator, adjustment, ITT + secondary CACE
Stopping rule — fixed / OBF / always-valid, with the numbers
Expected direction — sign and rough magnitude
Interpretation table — the four cells above, filled in
Sign it. Names on the sheet. That signature is the pre-registration.
Report back: read your interpretation table aloud. The room can push back on ambiguous cells.
The failure modes
Wrong unit — PR metrics for Engineering-Change-level effects
Multiple mechanisms — new model + prompt + UI + policy shipped together
Missing exposure — assignment recorded, actual use unknown
Short window — speed measured; quality and learning still to arrive
Interference — treated and control units affect each other (shared reviewers, shared CI capacity)
Novelty — the effect is being new, not being better
Seasonality — the effect is a release freeze
No decision threshold — significance without practical significance
External factors
Scope to affected entities and time windows, not global notes.
release and planning cycles | reorganizations |
holidays and staffing | new tools or model releases |
incidents and outages | repository migrations |
policy and compliance | vendor or platform failures |
Every one of these has ended a well-designed experiment. Record them from day one.
Segment 4 — red-team (15 min)
Swap pre-registrations with the next team.
For their design, break it. Look for:
Interference — their assignment unit is porous
Novelty — the effect will decay in six weeks
Seasonality — the pilot window is not representative
Non-random assignment — who volunteered, who was chosen
Missing exposure — assignment ≠ use
Ambiguous interpretation cell — the "can’t tell" quadrant is not defined
Return the sheet with a red-line list.
Rule: the receiving team must address each red-line — accept, mitigate, or defend — before the pre-reg is final.
Pre-experiment readiness
Do not launch until:
treatment and control are technically stable
assignment and exposure are instrumented
primary outcomes and guardrails are available
baseline variance and power are understood
security, legal, and ethical reviews are complete
stopping and rollback rules are defined
concurrent interventions are registered
The cost of a delayed launch is almost always smaller than the cost of a study that has to be redone.
Worked example — the AI-review pre-reg
180 backend teams. Vendor V, model M @ 2024-08, prompt P rev 12. Assign by team; analyse by Engineering Change. Stratified by baseline review latency and language.
Baseline \(\sigma = 42\) minutes per change; MDE = 8%.
Result: −6.4% effort (CI 2.1%–10.7%); escapes unchanged.
Below the pre-declared 8% MDE. Real, but too small at current licensing cost. Interpretation-table cell: Can’t tell → revisit at next model refresh.
Post-experiment decision record
Whatever the outcome, write:
estimated causal effects and intervals
practical significance vs. pre-declared threshold
guardrail status
heterogeneous effects
external-factor sensitivity
economic interpretation and limitations
adopt · revise · reject · extend
rollout and monitoring plan
The output of a good experiment is organizational knowledge — even when the intervention fails.
What you go home with
A signed pre-registration for one experiment you can actually run
A red-line list from a peer team, with each item addressed
A decision the pre-reg commits you to, per interpretation cell
A launch date — or a written reason the design is not yet ready
Close
Playbooks make the choices explicit before the data arrives.
Read: Chapter 12 of Engineering Intelligence.
Pair with: the causal-inference workshop when the estimator matters.
Return: when the experiment ends. Bring the decision record.