build: 2026-10-19 20:32 UTC

Engineering Intelligence Maturity Assessment

Workshop — 2 hours

 

Engineering Intelligence Maturity Assessment

A self-assessment workshop derived from Appendix J of Engineering Intelligence.

2 hours · self-scoring · roadmap outputs

Purpose

By the end of this workshop your team will:

  • Have scored your organisation across the eight EI dimensions

  • Know which capability closes the biggest decision-risk gap

  • Have a next-quarter investment plan grounded in the maturity ratchet — not in tool fashion

What we mean by maturity

Not: how many dashboards you have.

Yes: what class of decision your evidence system can defensibly support.

Every level closes a specific gap. You cannot skip levels; each is a prerequisite for the practices at the next.

The six levels

maturity ratchet

L0 → L5, each ratchet adds specific capability.

Level 0 — Fragmented telemetry

"Teams answer the same question three different ways."

  • Metrics defined locally

  • Cross-system linkage missing

  • Activity counts dominate

  • Data quality mostly implicit

Cost: every strategic conversation starts with definition arguments.

Level 1 — Descriptive visibility

"A leader can open a dashboard without arguing about definitions."

  • Major sources ingested

  • Standardised team and repo metrics

  • Basic lineage and quality monitoring

Missing: mechanism. You can see patterns, not explain them.

Level 2 — Connected Engineering Change evidence

"End-to-end questions become answerable for the first time."

  • Reconstruction of intent → outcome

  • Confidence bands on joins

  • Analysts stop reasoning about pull requests, start reasoning about Changes

Cost of missing this level: multi-repo coordination is a rumour, not a measure.

Level 3 — Governed decision intelligence

"A metric change is a release, not a query edit."

  • Metrics have semantic versions

  • Consequential measures carry uncertainty

  • Decision records reference metric versions

  • Prohibited uses are enforceable

Cost of missing this: a metric shift and a real change are indistinguishable.

Level 4 — Experimental learning system

"'We rolled it out and things got better' is no longer evidence."

  • Assignment + exposure instrumentation

  • Guardrails and stop rules

  • Pre-analysis plans and reproducibility packages

  • Negative results shipped

Cost of missing this: your AI evaluations rest on before-after averages.

Level 5 — Adaptive engineering organisation

"The operating model itself is a subject of measurement."

  • Policies as versioned interventions

  • Learning graph survives reorganisations

  • Bounded automated policy with audit + appeal

  • AI agents with identity, authority, provenance

Rare. Not many organisations. That’s OK — L3 is the pragmatic target.

The eight assessment dimensions

Score each on 0–5 separately:

  1. Decision orientation

  2. Source evidence and contracts

  3. Engineering Change semantics

  4. Identity, ownership, organisational context

  5. Metrics and governance

  6. Experimentation

  7. Economics

  8. AI participation

A single averaged score hides critical weaknesses.

Segment 1 (30 min) — self-score

Individually score your organisation across the eight dimensions.

One rule: score demonstrated operating capability, not documented policies or purchased tools.

If you can’t cite a decision that was made or reversed because of the evidence, that dimension is not at L3.

Segment 2 (25 min) — reconcile

Compare scores across the group.

You will disagree. That’s the useful part.

The disagreements are usually about interpretation of "demonstrated." Push each other to name the decision.

Segment 3 (25 min) — identify the binding constraint

For each dimension where you scored 0-2, ask:

  • Which decision class becomes unreachable at this level?

  • What is the smallest capability that would unlock the next level?

Not: what tool to buy. Yes: what would allow us to answer one specific question we cannot answer today.

Segment 4 (20 min) — pick one

You get one investment for next quarter.

Choose the dimension where:

  • The next-level unlock has the largest decision consequence

  • The path to that unlock is bounded and known

Use the pilot-selection rubric: sponsored, consequential, evidence-reachable.

Reference: companion/notebooks/20-pilot-selection.py.

Anti-patterns to name and avoid

  • Maturity theatre — scoring high without operating evidence

  • Average-score optimism — hiding an L1 dimension inside an L3 average

  • Tool-based maturity — believing a purchase implies capability

  • Roadmap by connector — sequencing by data availability, not decision value

  • Enterprise-wide first step — trying to boil the ocean instead of proving one loop

Ninety-day plan template

  • Week 1-2: Register the decision question. Name the sponsor.

  • Week 3-6: Build minimum evidence path (source + reconstruction if needed)

  • Week 7-10: Governed metric v1.0 with uncertainty and guardrails

  • Week 11-12: Register the use, produce the decision record

What ships: one Class-2 decision supported by governed evidence. Not a platform.

What you go home with

  • Signed self-assessment scorecard (per dimension)

  • One next-quarter investment written as a bounded pilot proposal

  • A no list — dimensions you deliberately are not investing in yet

  • A review date (typically 90 days out) to reconvene

Reading and running

  • Read: Appendix J of Engineering Intelligence

  • Read: Chapter 15, Implementation and Rollout

  • Run: companion/notebooks/20-pilot-selection.py

  • Run: companion/notebooks/19-portfolio-allocation.py when you’re negotiating capacity

Thank you

Reconvene in 90 days.

Bring the decision you made. Bring the decision you almost made and reversed. Those are what a maturity system is for.