Engineering Intelligence

The fundamental unit of analysis should not be the developer, commit, pull request, ticket, deployment, or AI suggestion. Each is only a partial artifact of a larger process.

The central unit is the Engineering Change: a vendor-independent representation of an intended modification and its implementation, validation, delivery, operation, and observed consequences.

An Engineering Change connects:

This unit allows organizations to measure the complete change system rather than optimizing isolated tools or stages.

Engineering capacity is not adequately represented by headcount or hours. The scarcest resource is often expert attention: the ability to understand, decide, validate, coordinate, diagnose, and accept responsibility.

Attention is consumed by:

The most valuable interventions do not merely increase visible activity. They release or redirect scarce attention toward higher-value decisions.

This leads to a critical economic distinction:

Capacity becomes valuable only when it appears in usable form, reaches constrained valuable work, and is absorbed by downstream systems capable of converting it into outcomes.

Engineering work moves through queues for implementation, review, specialist approval, CI, test environments, deployment, incident response, and organizational decisions.

Queue behavior is nonlinear. Near high utilization, small changes in arrival rate, variability, or service capacity can produce disproportionately large changes in waiting time.

This means:

Engineering Intelligence therefore measures arrival, service, wait, work in progress, variability, utilization, and lifecycle state together.

Engineering investments should be evaluated against credible alternatives, not against zero.

A complete business case includes:

The framework explicitly separates:

This prevents modeled engineering time from being reported as realized cash savings.

A metric is not a number or SQL query. It is a versioned analytical product with:

A metric that is useful for team workflow improvement may be invalid for cross-team comparison. A metric valid for tool evaluation may be prohibited for individual performance decisions.

The question is never simply, “Is this metric valid?” It is:

Descriptive telemetry can show that two things moved together. It cannot, by itself, establish that an intervention caused the outcome.

Engineering Intelligence supports a continuum of evidence:

Experiments require more than a feature flag. They require:

Assignment must remain distinct from exposure. Adoption must remain distinct from effectiveness.

AI-assisted engineering should not be evaluated through suggestion volume, acceptance rate, generated code, or anecdotal enthusiasm alone.

AI may reduce authoring effort while increasing:

The correct unit is the net effect on the Engineering Change and the surrounding system.

A useful economic expression is:

AI systems also require provenance: provider, model, configuration, prompt, retrieval context, tool permissions, initiating actor, delegation chain, latency, and cost.

As agents become more autonomous, the Engineering Change graph must include machine identity, delegated authority, validation responsibility, and acceptance accountability.

Architecture influences the cost and risk of change through:

Architecture should not be reduced to static complexity counts. It should be evaluated by how structure changes the effort, coordination, safety, and reversibility of Engineering Changes.

The framework combines repository, service, runtime, deployment, ownership, and incident graphs to evaluate architectural constraints over time.

A credible Engineering Intelligence platform preserves a traceable path from source evidence to decision.

The reference architecture includes:

The architecture separates:

This separation is essential because each carries different uncertainty and authority.

An unrestricted evidence system will damage trust and eventually damage the evidence itself.

Engineering Intelligence therefore includes controls for:

The framework presumes that ordinary engineering telemetry is unsuitable for individual compensation, promotion, discipline, or termination decisions unless a separate formal process establishes exceptional validity, necessity, fairness, legal authority, transparency, and due process. In most organizations, such uses should remain prohibited.

Engineering Intelligence becomes useful when embedded in an operating cadence.

A complete loop is:

The book provides standard templates for each part of this loop.

Organizations can develop the capability incrementally.

A successful Engineering Intelligence program is not defined by the number of connectors, dashboards, metrics, or AI features.

It is visible when the organization can repeatedly:

The ultimate product is not telemetry. It is an organization that becomes better at learning how it engineers.

Begin with:

This path focuses on investment, capacity, risk, causal evidence, portfolio choices, and value realization.

Begin with:

This path focuses on building an evidence system that connects workflow instrumentation to useful decisions.

Begin with:

This path focuses on constructs, validity, reliability, uncertainty, causal design, replication, and reporting.

Begin with:

This path treats architecture as a determinant of coordination cost, validation burden, change risk, and reversibility.

Begin with:

This path focuses on net attention, validation burden, model and agent provenance, quality guardrails, causal evaluation, economics, and governance.

Begin with:

This path focuses on purpose limitation, access, identity, retention, automated decisions, correction, appeal, and audit.

Begin with:

This path explains how measurement can improve the work system without reducing people to activity counts.

The book uses several recurring distinctions.

Observed evidence is directly recorded from a source system.

Reconstruction links evidence into entities, relationships, or lifecycle state.

Inference estimates something not directly observed.

Prediction estimates an unknown or future result.

Causal evidence estimates how an intervention changed an outcome relative to a counterfactual.

Decision is the organizational action selected from alternatives.

These categories should not be collapsed.

A practical reading and application sequence is:

Appendix G provides reusable records for this process.

The book should not be used as justification for:

No metric, model, benchmark, or organizational practice is context-free.

The definitions and standards in this book are designed to make context explicit: population, workflow, architecture, risk, work mix, time, ownership, source quality, treatment version, and decision consequence.

Readers should adapt implementations while preserving these semantic and scientific constraints.

Some foundations in this book draw on mature research in measurement, queueing, causal inference, organizational learning, and software delivery. Other areas—particularly long-term AI effects, autonomous engineering agents, skill formation, digital twins, and prescriptive engineering policy—remain active research questions.

The book distinguishes durable principles from emerging possibilities and treats uncertainty as information rather than an editorial weakness.

Capitalized terms such as Engineering Change refer to controlled concepts defined in Appendix H. Tool-specific terminology should be mapped to these concepts rather than used as a substitute for them.

The reference templates are deliberately technology-neutral. They may become:

Their value comes from preserving purpose, ownership, evidence, versions, uncertainty, conditions, and review—not from the storage format.

The book succeeds when it helps an organization replace vague claims such as “engineering is faster” or “the AI tool is productive” with more useful statements:

That is the language of an engineering organization capable of learning.

Modern software organizations generate extraordinary amounts of operational data. Source-control platforms record every commit, branch, review, and merge. Continuous-integration systems retain build results and test histories. Issue trackers capture work planning and defect reports. Deployment systems know what reached production, when, and through which path. Observability platforms reveal how software behaved after release. AI coding and review tools add another stream of comments, suggestions, decisions, and generated artifacts.

Despite this abundance, many organizations still cannot answer basic questions with confidence:

The obstacle is not a lack of metrics. It is a lack of models.

A metric describes an observation. A model explains how observations relate, identifies mechanisms, supports prediction, and allows competing interventions to be evaluated. Pull-request lead time, for example, is useful as a descriptive measure. It becomes operationally valuable only when an organization can distinguish waiting from active work, control for change complexity, identify the constrained stage, and estimate whether an intervention would improve quality-adjusted delivery rather than simply accelerate merging. This decision-oriented view is consistent with the broader measurement principle that validity attaches to a specific interpretation and use, not merely to a number’s reproducibility. See [messick1995] and [hand2004].

Engineering Intelligence therefore begins with a change in objective:

Those resources include money and compute, but the most constrained resource in most engineering organizations is expert human attention. Build capacity can be purchased. AI inference can be scaled. Experienced judgment, architectural context, security expertise, and sustained cognitive focus remain comparatively scarce. Cognitive-load and interruption research provide part of the theoretical basis for treating attention as constrained and for distinguishing uninterrupted reasoning from fragmented work. See [sweller1988] and [mark2008].

A dashboard can reveal that review latency increased. An Engineering Intelligence system should help determine why it increased, whether the increase is harmful, which populations are affected, what intervention is likely to work, and whether the intervention caused an improvement.

This requires a closed learning loop:

The loop contains six capabilities:

The final step is essential. Measurement that does not change a decision is, at best, descriptive. At worst, it creates surveillance, metric gaming, and administrative burden without improving the engineering system. The risk that measures become distorted when converted into targets is well established in the work associated with Campbell’s Law and Goodhart’s Law. See [campbell1979] and [goodhart1984].

Software engineering has repeatedly adopted measures that are easy to compute but dangerous to optimize directly.

Lines of code can reflect implementation volume, generated code, verbosity, duplication, or unnecessary complexity. Commit count depends on personal workflow and repository policy. Pull-request count is sensitive to slicing conventions. Story points are locally defined and frequently distorted when aggregated. Utilization rewards visible busyness while penalizing the slack required for learning, incident response, mentoring, and creative problem solving.

Even stronger frameworks must be used carefully. Delivery-performance measures such as deployment frequency, lead time for changes, change-failure rate, and restoration time are valuable system-level outcomes and are central to the software-delivery research synthesized in [forsgren2018]. The SPACE framework emphasizes that productivity cannot be represented by a single dimension and includes satisfaction and well-being, performance, activity, communication and collaboration, and efficiency and flow. See [forsgren2021-space]. Developer Experience research further emphasizes feedback loops, cognitive load, and flow state. See [noda2023-devex] and [fagerholm2012].

Engineering Intelligence builds on these foundations. It does not attempt to replace them with one universal score. Instead, it adds four capabilities:

The result is less like a scorecard and more like an observability and decision system for the engineering organization.

A pull request is a convenient platform object, but it is rarely the complete unit of engineering work.

A single product change may require:

Conversely, one pull request may combine several unrelated changes. Pull-request boundaries are shaped by repository topology, team conventions, code ownership, and tooling limitations. Treating every pull request as an equivalent unit therefore introduces substantial measurement error. The broader relationship between organizational coordination and technical structure is established in socio-technical research, including [conway1968], [cataldo2006], and [herbsleb2007].

This book defines an Engineering Change as follows.

The accepted outcome may be production deployment, a completed migration, a security remediation, an internal platform capability, a retired system, or another explicitly defined terminal state. Not every Engineering Change produces user-visible code.

The relationship between intent and implementation is many-to-many:

This graph-oriented representation matters for multi-repository and multi-team work. Cost and latency should be attributed to the complete change graph, while local measures can still be retained for individual pull requests, builds, approvals, and deployments. The entity-activity-agent structure is compatible with general provenance models such as W3C PROV, while the Engineering Change model adds domain-specific semantics. See [moreau2013].

Software delivery is neither purely technical nor purely organizational. Outcomes emerge from interactions among people, tools, architecture, incentives, policies, and external constraints. This framing is consistent with the organizational and socio-technical foundations in [conway1968], [cataldo2006], and [herbsleb2007].

Consider a two-line configuration change. Its implementation complexity may be trivial. Yet the change could modify authentication behavior for every product, require review from three security teams, trigger a coordinated global deployment, and create a severe regulatory risk if incorrect. Code size is low; potential blast radius and coordination complexity are high.

A different change may add thousands of lines of generated client code. Its raw size is large, but its reasoning burden, novelty, and operational risk may be low.

Engineering Intelligence therefore represents a change as a multidimensional vector rather than a single scalar complexity score. Relevant dimensions include:

These dimensions influence both expected effort and uncertainty. They also help prevent invalid comparisons. A workflow intervention should not be judged by comparing raw cycle time across fundamentally different change populations. The modularity principles in [parnas1972] and the team cognitive-load perspective in [skelton2019] provide important foundations for understanding why boundaries and ownership affect change difficulty.

Traditional cost accounting often starts with labor hours multiplied by loaded compensation. That is necessary but incomplete.

An hour of uninterrupted architectural work is not equivalent to twelve five-minute interruptions distributed across a day. A review request that arrives after the author has mentally left the task may require expensive context reconstruction. A senior security engineer’s attention may be more constrained than the monetary cost assigned to the same amount of general engineering time. A queue of unresolved reviews can create attention residue even when no one is actively reviewing. These distinctions are consistent with cognitive-load theory, interruption research, and developer-productivity studies. See [sweller1988], [mark2008], and [meyer2014].

Engineering Intelligence therefore distinguishes at least four forms of human effort:

The fourth category is frequently invisible. It appears as longer completion time, repeated reading, reopened files, duplicated analysis, forgotten assumptions, and hesitant decisions. It is one reason why elapsed time cannot be naively converted into labor cost.

The cost model developed later in this book will therefore include direct effort, queueing delay, context-recovery cost, scarce-expertise consumption, tool cost, infrastructure cost, expected quality loss, and cost of delay. The distinction between elapsed flow time, service time, utilization, and queue delay draws on queueing and operations research, including [little1961], [kingman1961], and [hopp2011].

AI coding and review systems change both engineering work and the observability of that work.

An AI reviewer may identify a defect, generate a patch, explain a security issue, suggest a test, summarize a large change, or route a concern to a human expert. The human may accept the suggestion, modify it, reject it, ignore it, create a follow-up issue, or independently discover the same problem.

Treating the AI merely as a software license obscures its role. The AI participates in the engineering system and produces measurable work products and decision prompts. Engineering Intelligence therefore models AI contributions explicitly. Empirical work on AI coding assistants shows that interaction, usability, comprehension, debugging, and outcomes vary materially by task and context. See [peng2023], [vaithilingam2022], and [barke2023].

This does not mean equating model output with human judgment. It means recording the distribution of work:

This framing creates the possibility of measuring cognitive offload: human reasoning or production work avoided because a reliable AI contribution performed it first. It also reveals cognitive reload: additional human work created by noisy, ambiguous, or untrustworthy AI output.

A high suggestion-acceptance rate is not sufficient evidence of offload. Developers may accept trivial suggestions, mechanically apply changes they do not understand, or ignore valuable comments because they lack time. Conversely, a rejected comment may still create value by prompting inspection. The correct object of study is the causal effect of AI participation on human effort and quality-adjusted outcomes. Repository-level agent benchmarks such as [jimenez2024] are useful for capability evaluation, but they do not replace organization-specific causal and production-outcome studies.

Every operational metric in this book is evaluated through six questions:

This discipline prevents metric proliferation. Large enterprises can generate thousands of measures; only a subset deserve to become operational objectives. Measurement theory and validity research emphasize that a measure must be evaluated against its intended interpretation and use. See [stevens1946], [messick1995], and [hand2004].

Metrics will be classified along several axes:

No individual-level metric should automatically be treated as a performance score. Measurements designed to understand system constraints can become harmful when repurposed to rank people. This changes behavior, damages trust, and invalidates the measurement process itself. See [campbell1979], [goodhart1984], and [messick1995].

Engineering organizations frequently introduce tools and policies globally, compare before-and-after averages, and attribute any movement to the intervention. This is rarely sufficient.

Tool adoption may coincide with seasonal release cycles, reorganizations, staffing changes, product launches, incidents, changes in repository mix, or broader adoption of generative AI. Teams choose tools non-randomly. Early adopters differ from late adopters. Measurement definitions change. Novelty effects decay. Work may be shifted outside the observed boundary. These are standard threats to causal identification and validity described in [shadish2002], [imbens2015], and [hernan2020].

Engineering Intelligence treats experimentation as infrastructure, not an occasional analytics project.

A strong evaluation specifies:

Randomized controlled experiments are valuable when feasible, but software organizations often require alternatives: staggered rollouts, switchback experiments, difference-in-differences, matched controls, regression discontinuity, interrupted time series, synthetic controls, hierarchical models, and Bayesian sequential analysis. The methodological foundations are covered in [rubin1974], [holland1986], [shadish2002], [angrist2009], [imbens2015], [hernan2020], and [kohavi2020].

The objective is not statistical sophistication for its own sake. It is to avoid confident but incorrect conclusions about expensive organizational changes.

A mature implementation separates event capture, identity resolution, change reconstruction, feature computation, causal analysis, and presentation.

Near-real-time events are appropriate for lifecycle state, queue depth, tool interactions, CI results, and deployment transitions. Batch computation is appropriate for repository topology, dependency graphs, historical baselines, code-complexity features, ownership networks, and model retraining. Some derived outcomes, such as realized defects or incident impact, arrive days or weeks after a merge.

The architecture must therefore support event time, late-arriving data, changing entity relationships, historical recomputation, and versioned metric definitions. A number displayed without the definition and model version that produced it is not reproducible evidence. These architectural requirements are grounded in event-log, stream-processing, provenance, and reliability principles described in [kleppmann2017], [akidau2018], [moreau2013], and [beyer2016].

The remainder of the book develops five connected layers.

Foundations define the Engineering Change, engineering economics, value, cognition, AI participation, complexity, and blast radius.

Measurement provides a catalogue of direct and derived measures for flow, quality, cost, attention, coordination, architecture, and AI contribution.

Instrumentation describes event schemas, identity resolution, stream and batch architecture, knowledge graphs, feature stores, data quality, privacy, and governance.

Experimentation covers randomized and quasi-experimental designs, causal inference, power, sequential testing, heterogeneous effects, and external confounding factors.

Applications demonstrates how the framework can evaluate AI review tools, merge queues, branch protection, stacked changes, repository strategies, CI optimizations, autonomous agents, and portfolio-level engineering investments.

The goal is practical. Every major construct must be measurable, useful for a decision, and capable of empirical validation.

At first glance, the cost of a pull request seems straightforward. Count the time spent by the author and reviewers, multiply by loaded compensation, add the software and infrastructure costs, and report the total. This calculation is useful, but it is not a complete economic model.

Several important costs remain hidden:

The correct economic question is not simply, "How much labor was recorded?" It is:

This is an activity-based view of engineering economics. Costs are attributed to the activities and constraints that created them rather than being inferred from a single duration or output count.

Four terms must be kept separate.

Cost is the consumption of resources associated with an Engineering Change or process.

Price is the amount paid to acquire a tool, service, or labor input. Price may be easy to observe while cost remains hidden. A low-priced tool can create expensive human validation work.

Value is the benefit created or protected by the outcome. Value may include revenue, retention, reduced operational risk, compliance, faster learning, strategic option value, or avoided future effort.

Investment is current cost incurred to improve future capability. Platform engineering, test infrastructure, codebase simplification, and knowledge transfer are often investments rather than immediate feature-delivery costs.

Confusing these terms leads to poor optimization. A process can become cheaper while destroying value. A change can be expensive while producing exceptional returns. A platform investment can reduce current throughput while increasing future capacity.

Let an Engineering Change be denoted by . Its total economic cost can be represented as:

where:

This decomposition is intentionally additive for explanation, but the components are not always independent. A tool may reduce authoring time while increasing quality. More review may increase direct human cost while reducing expected defect cost. A slower rollout may increase delay but reduce operational risk.

A useful model therefore records both component values and the assumptions connecting them.

The simplest human-cost estimate is:

where is the set of participants, is the number of hours participant spent on the change, and is that participant’s loaded hourly cost.

Loaded cost commonly includes salary, benefits, payroll taxes, facilities, management overhead, and other allocated expenses. Organizations may use standardized rates by role or level to reduce sensitivity and protect individual compensation data.

This estimate is useful for budgeting, but three refinements are required.

Tool cost includes both fixed and variable components.

Let a tool have annual fixed cost , usage-based cost , and allocated population . A simple per-change allocation is:

where is an allocation factor representing the share of licensed capacity attributed to the change.

Per-seat allocation is often sufficient for financial reporting but weak for experiment analysis. To compare two AI review workflows, capture marginal usage:

The full economic cost of an AI tool must also include the human work it creates:

where is the value of human effort genuinely avoided or quality improvement created.

A cheap model producing many low-value comments can be more expensive than a costly model producing a small number of trusted findings. Cost per comment is therefore rarely useful. Cost per validated defect prevented, or cost per hour of quality-adjusted human attention saved, is more meaningful.

Infrastructure cost includes more than cloud invoices. A change can trigger builds, tests, ephemeral environments, security scans, artifact storage, deployment waves, and production observation.

For Engineering Change :

Useful event-level inputs include:

Shared infrastructure requires allocation rules. The rule should match the decision. For cost optimization, marginal cloud cost may be appropriate. For total-cost comparison, allocated platform and staffing cost may be needed. Mixing marginal and fully allocated cost in the same analysis produces misleading results.

Coordination cost is often the dominant hidden cost of large enterprise changes.

It arises from:

A simple coordination model might use:

where:

This should not be interpreted as a universal formula. It is an example of a model whose coefficients can be learned from historical changes.

Coordination cost is nonlinear. Adding a second team may be inexpensive if the relationship is routine and interfaces are stable. Adding a fifth team can increase the number of communication paths and scheduling constraints dramatically.

The theoretical number of pairwise communication paths among participants is:

Real organizations do not require every participant to communicate directly, but the equation illustrates why coordination structures, clear ownership, and interface contracts matter.

Waiting is not free merely because no one is actively working.

A change delayed in review may postpone:

Define the instantaneous value-delivery rate of change as . If delivery is delayed from time to , the cost of delay is:

In practice, value curves are rarely known precisely. Organizations can use categories:

Cost of delay should not be applied indiscriminately to manufacture urgency. Its purpose is prioritization and trade-off analysis. When uncertain, report a range and sensitivity analysis.

Quality cost should be modeled prospectively and retrospectively.

Retrospective quality cost includes observed rework, rollback, support, incident response, customer remediation, and follow-up changes.

Prospective quality cost is expected loss:

where is the set of relevant failure modes, is the probability of failure given the change, and is its impact.

The expected-cost model allows review and testing investments to be evaluated economically. Suppose an additional security review costs $600 and reduces the estimated probability of a $200,000 loss from 1.0% to 0.4%. The expected loss reduction is:

Under those assumptions, the review produces $1,200 of expected risk reduction for $600 of direct cost. The estimate is uncertain, but the structure makes the decision explicit.

Observed defect rates often underrepresent security, privacy, financial, and availability risk because severe events are rare. Historical averages can therefore create false confidence.

Risk analysis should combine:

For rare events, expected value alone can be inadequate. Two changes may have the same expected loss while one carries catastrophic tail risk. Decision support may therefore include quantiles, conditional value at risk, or explicit policy constraints.

A change violating a regulatory requirement is not made acceptable merely because its estimated expected loss is low.

Figure 4 illustrates the same phenomenon for the cost of an Engineering Change: the distribution is right-skewed, and the mean lies well above the median. A policy tuned to the average change under-invests in the tail; a policy tuned to the tail over-invests in the routine. Reporting both statistics, together with the shape of the distribution, avoids either failure mode.

Cost analysis becomes dangerous when disconnected from value.

Let be the discounted value created by Engineering Change . A basic net value is:

Return on engineering investment can be expressed as:

These formulas are simple; estimating value is not.

Engineering value commonly falls into six categories:

Not every change should be forced into a precise dollar value. The purpose of valuation is to improve comparisons, expose assumptions, and support portfolio decisions. Qualitative value classes and bounded estimates are preferable to fabricated precision.

Consider an organization processing 100,000 pull requests per year. It is evaluating an AI reviewer with an annual license and infrastructure cost of $1.2 million.

The observed treatment effects from a controlled rollout are:

Net human time saved per pull request is five minutes:

Expected defect savings are:

Ignoring delay value, total estimated benefit is $1.85 million. Subtracting the $1.2 million tool cost yields an estimated annual net benefit of $650,000.

This result is incomplete until uncertainty and heterogeneity are considered. The effect may differ by language, repository, change complexity, team experience, or comment type. The defect-cost distribution may be heavy-tailed. Adoption may be incomplete. The seven-minute review reduction may be concentrated in trivial changes.

A publication-quality evaluation should therefore report:

The point-estimate of $650,000 is a starting hypothesis, not a conclusion. Reported in isolation it is nearly indistinguishable from an educated guess; reported with the accompaniments above, it becomes a claim that other engineering leaders can inspect, replicate, or dispute. Chapter 6 develops the causal machinery required to turn such an evaluation into evidence that survives contact with an audit or a change of leadership.

A platform team proposes requiring a database specialist to approve every schema-changing pull request.

Historical data shows 4,000 such changes per year. Each approval requires an average of 12 specialist minutes and creates an average queueing delay of 5.5 hours. The specialist’s loaded cost is $190 per hour. The approval is estimated to prevent eight production incidents annually, each costing an average of $45,000.

Direct review cost:

Expected incident savings:

On direct cost and expected quality benefit, the policy appears favorable. But the queueing delay affects 4,000 changes, and the specialist becomes a single constrained resource. The correct decision may be to retain specialist review only for high-risk migrations while automating classification and standard checks for routine changes.

This illustrates a recurring pattern: the optimal solution is often not "add review" or "remove review," but route scarce attention according to predicted risk and uncertainty.

Different cost components require different methods.

Averages answer questions about total allocation. Marginal values answer questions about change.

The average cost of a CI system includes fixed platform engineering and infrastructure. The marginal cost of one additional test run may be only compute and storage. When deciding whether to run a new test, marginal cost is often appropriate. When comparing operating models or vendors, total allocated cost matters.

Likewise, reducing average review time by ten minutes does not necessarily free ten minutes of usable capacity. The saved fragments may be too dispersed to reallocate. A constrained reviewer queue may show a large nonlinear benefit from a small capacity increase, while an unconstrained queue may show almost none.

Economic analyzes should state explicitly whether they use:

Choosing incorrectly is the more common failure mode than computing incorrectly. A team that reports "we saved five million dollars" using fully allocated cost is describing a spreadsheet reallocation, not real cash freed for other work. A team that treats avoidable cost as marginal will systematically under-invest in capacity because it counts the savings but not the queuing that produced them. Naming the cost category next to every economic claim is therefore the smallest change with the largest effect on the honesty of the resulting decision.

Engineering economic models contain uncertain inputs. The proper response is not to hide uncertainty but to propagate it.

Suppose annual benefit is:

where is annual volume, is minutes saved, is validation minutes added, is hourly labor cost, and are control and treatment defect probabilities, and is defect cost.

Each parameter can be represented as a distribution rather than a point. Monte Carlo simulation can then estimate the probability that benefit exceeds cost.

A decision report might state:

This is more useful than a single number because it identifies which additional evidence would improve the decision.

A practical scorecard should avoid collapsing everything into one number. Recommended views include:

Executive reporting should emphasize distributions, trends, and decisions rather than league tables. Operational reporting should permit drill-down to the Engineering Change graph and raw evidence while respecting privacy and access controls.

Pull requests are attractive analytical units because they are visible, structured, and supported by mature APIs. They contain timestamps, authors, commits, reviewers, comments, checks, and merge outcomes. For many local questions, the pull request is exactly the right unit.

It becomes misleading when used as a universal proxy for engineering work.

A customer-visible capability may require a web change, a mobile change, an API update, a shared-library release, a database migration, infrastructure configuration, security approvals, a feature-flag rollout, documentation, and post-deployment verification. Each artifact may appear in a different repository or system. Measuring them independently hides the actual coordination, critical path, and outcome.

The reverse problem also occurs. A pull request may combine a refactor, dependency update, and product feature that should be analyzed separately. Repository conventions, not engineering intent, determine the boundary.

A reliable measurement system must therefore distinguish between observable artifacts and the logical change those artifacts collectively realize.

Let an Engineering Change be represented as:

where:

This representation separates what the organization intended from what its tools observed.

The natural representation is a temporal property graph:

where is the set of nodes, the set of edges, node attributes, edge attributes, and temporal information.

A graph representation enables questions that flat pull-request tables cannot answer well:

Each of these is a traversal over the Engineering Change graph rather than a query against a flat activity log. The graph therefore does more than reshape the data: it makes coordination, dependency, and provenance first-class citizens of the analysis. Subsequent chapters use this representation for cost accounting, blast-radius estimation, and experimental unit-of-analysis decisions.

Every metric depends on a boundary. The Engineering Change boundary determines which activities, costs, outcomes, and failures are attributed together.

A robust identity scheme should provide a stable change identifier, parent-child relationships, aliases from external systems, versioned intent, explicit lifecycle state, confidence when membership is inferred, and merge-split history.

An opaque organization-wide ID such as EC-01J8…​ is preferable to a ticket number or repository-local key. External identifiers become aliases.

A generic lifecycle is proposed, accepted for work, active, under verification, ready for release, releasing, observing, and accepted, abandoned, or rolled back.

Real workflows are not strictly linear. Work may return from review, pause, fork, or deploy incrementally. State should therefore be derived from events rather than assumed from a waterfall sequence.

A state transition can be represented as:

The transition function should be versioned. Changing the definition of "active" or "ready for release" can materially alter flow metrics.

The strongest reconstruction uses explicit identifiers propagated through the toolchain.

A change ID can be attached to issue metadata, branch names, commit trailers, pull-request fields, CI environment variables, artifact metadata, deployment manifests, feature flags, telemetry attributes, and incident records.

A deployment could emit the same identifier as an OpenTelemetry resource attribute:

Explicit propagation produces high-confidence linkage and allows production telemetry to connect back to intent.

In existing estates, most changes must initially be inferred.

For artifacts and , define a linkage probability:

where indicates common Engineering Change membership, is a feature vector, and is a logistic function.

Features may include exact ticket matches, explicit references, shared authors and reviewers, temporal proximity, repository dependency, coordinated deployment, shared feature flags, semantic similarity, common design documents, service dependencies, and incident linkage.

A graph clustering or correlation-clustering method can then propose Engineering Change groups.

A reconstruction system must answer, "Why are these artifacts linked?"

For each edge or membership decision, retain source systems, contributing signals, model version, feature values, explanation, confidence, creation time, correction time, and human override history.

Explainability is necessary for trust, correction, and debugging. It also reveals systematic data-quality problems.

Multi-repository work should be analyzed as a dependency graph, not a bag of pull requests.

Let be pull requests in Engineering Change , and let edges represent ordering or compatibility dependencies. The critical path is the longest-duration path through the directed acyclic portion of the graph.

Useful graph measures include repository count, dependency depth, fan-in, fan-out, ownership boundaries, critical-path length, parallelism ratio, synchronization points, cycles, and release-order constraints.

Cycles deserve special attention. Two changes that each require the other can create compatibility layers, coordinated merges, or manual sequencing. The coordination cost may far exceed implementation complexity.

In a monorepo, one pull request may touch many services and teams. In a polyrepo environment, the same intent may produce a chain of pull requests. Raw pull-request counts and sizes are therefore not comparable without topology context.

The Engineering Change abstraction distinguishes intrinsic complexity, topological complexity, and organizational complexity. This supports experiments that estimate the economic effect of repository layout and ownership structure.

The Engineering Change does not replace all planning entities. An initiative is usually broader and longer-lived. A ticket may describe one task or defect. A release is a packaging concept that may include many changes. A pull request is a source-control collaboration object.

The relationships are many-to-many:

A strong platform preserves each entity and connects them rather than forcing one system to become the universal source of truth.

The domain model above shows the Engineering Change with the attributes and derived quantities it must expose to support cost accounting, blast-radius estimation, and cohort selection later in the book. The cardinalities matter: because a single Engineering Change may accumulate Effort from many Participants and produce exactly one Outcome, aggregation queries that treat any of these relationships as one-to-one will systematically under- or over-count.

Membership can change as work evolves. An artifact may initially belong to a prototype and later be incorporated into a production change. A shared-library release may serve several Engineering Changes.

The graph therefore needs temporal membership:

An artifact can participate in multiple changes with different roles. Cost allocation should reflect role and causality rather than duplicating the full cost into every parent.

When one artifact supports several Engineering Changes, possible allocation methods include equal allocation, usage-weighted allocation, causal attribution, Shapley-value approximations, primary-change ownership with secondary references, or reporting shared-platform cost separately.

No method is universally correct. The allocation method must match the decision.

Engineering Change instrumentation should be event-driven. Source systems emit normalized events into an immutable log. Identity-resolution services map users, repositories, services, teams, and tools. A reconstruction service creates and updates graphs. Stream processors maintain near-real-time lifecycle state, while batch jobs recompute topology, semantics, and historical corrections.

Core event domains include planning, source control, AI, CI, deployment, production, and organization.

A normalized event envelope should include:

Near-real-time computation is appropriate for current state, review and CI queues, aging changes, experiment exposure, AI interactions, deployment progression, and operational alerts.

Batch computation is appropriate for repository topology, semantic clustering, historical reconstruction, ownership networks, code-complexity analysis, model training, incident attribution, and cost recomputation.

The two paths must converge on versioned state. A batch correction should update the graph without erasing the evidence that produced the previous state.

Recommended data-quality measures include explicit ID coverage, inferred membership coverage, percentage above confidence thresholds, orphan artifact rate, contradictory-link rate, late-event rate, source-event loss, identity ambiguity, reconstruction correction rate, event-to-state latency, and metric recomputation lag.

Coverage and confidence should be displayed alongside derived measures.

Reconstruction quality can be evaluated as an information-retrieval problem.

Pairwise precision and recall measure whether artifact pairs are correctly grouped. Cluster metrics such as B-cubed precision and recall, adjusted Rand index, and normalized mutual information can evaluate whole-group quality.

Ranking metrics such as mean reciprocal rank and NDCG are useful when the system proposes candidate related artifacts.

Evaluation should be stratified by monorepo versus polyrepo, single-team versus cross-team, work type, explicit versus missing ticket IDs, repository activity level, and long-running versus short changes.

Users should be able to attach or detach an artifact, merge or split changes, correct intent, confirm an outcome, or mark a relationship as invalid.

Corrections become labeled data for active learning. The system should prioritize uncertain or high-value cases for review rather than asking users to validate every graph.

The correction interface must show why a link exists and what downstream reports will change. Silent mutation destroys trust.

Engineering Change reconstruction combines data across many systems and can expose sensitive organizational behavior. Governance requirements include purpose limitation, role-based access, retention rules, aggregation thresholds, audit logs, model transparency, and explicit prohibition of unsupported individual ranking.

Human identity is often unnecessary for system analysis. Pseudonymous identifiers and role-level aggregation should be used whenever the decision does not require named attribution.

A platform team wants to attribute cost and outcome to an Engineering Change titled "add refund reason codes to checkout." The reconstruction service assembles evidence from six systems: the intent ticket, three pull requests across two repositories, a feature-flag configuration change, and a deployment record.

Deterministic linkage — where the pull-request description or commit trailer carries refs REFUND-8842 — connects two of the three pull requests to the intent with confidence 1.0. The third pull request modifies the payment-integration service and shares an author but no explicit reference; a probabilistic model links it with confidence 0.62 based on temporal co-occurrence, semantic similarity of description, and shared file paths with the other two changes.

The feature-flag change is emitted by a separate system without repository references. It links to the intent via a normalized flag name (checkout.refund_reason_codes_enabled) that matches text in the intent description; confidence 0.88.

Total human effort attributed:

Total expected effort: 24.1 hours. A conservative interpretation reports 20.3 hours (deterministic only) as the floor and 26.4 hours (all evidence, unweighted) as the ceiling. The versioned reconstruction stamps every metric derived from this Engineering Change with reconstruction model v3.4, so if the linkage model is later revised the effort estimate can be recomputed rather than retconned.