AI-Assisted Engineering

A compiler transforms source deterministically. An AI reviewer judges, ranks, and influences decisions. An agent inspects, edits, tests, and opens PRs.

Every contribution now carries an actor type:

Evaluation requirements scale with autonomy. A completion can be judged at suggestion level; an autonomous migration agent needs task, system, and operational evaluation.

Every AI contribution should retain: change and artifact IDs, actor, model, provider, version, prompt, policy, context selection, tools, input scope, output, confidence, cost, human actions, downstream revisions, governance.

Without provenance, you cannot explain why behavior changed after a model, prompt, or index update.

AI work is rarely accepted unchanged. A model proposes, a human edits, another model reviews, a reviewer requests more.

Represent contribution as a derivation graph: GENERATED, EDITED, VERIFIED, REJECTED, REPLACED, INSPIRED, DUPLICATED.

Four distinct attribution objects:

Only the last supports value claims — and it normally requires experimentation.

Suggestion acceptance is an interaction measure, not an outcome. Evaluation needs:

At the task level: compare quality-adjusted human effort and accepted outcome — not generated lines.

Findings are information-retrieval + decision-support objects.

Severity-weighted recall matters more:

Ranking quality matters because human attention is limited. Use NDCG, precision at k, and severe-finding recall at k so the most important issues appear first.

A labeled corpus of Engineering Changes with known defects, non-defects, severity, category, context, and expected evidence.

Golden sets should include:

AI output must be interpreted and verified.

High output volume can raise \(VB\) even when inference cost is low.

Proxies: finding views, code navigation after a finding, time to action, independent reproduction, discussion, suggestion modification, rejection reason.

The goal is calibrated trust, not maximum trust.

Trust depends on precision, consistency, explanation quality, confidence honesty, domain fit, and recovery from errors.

Measure: reliance conditional on correctness, rejection of incorrect findings, acceptance of correct high-value findings, behavior change after visible errors, calibration by category and confidence band.

Offload: reduced reasoning, search, production, or verification while preserving outcomes. Reload: inspecting noisy output, reconstructing hidden assumptions, correcting artifacts, recovering from misplaced trust.

Estimate experimentally. Stratify by task type, expertise, and risk.

Numerator: value added, human cost avoided, expected quality benefit. Denominator: AI cost + validation cost.

Agent evaluation adds: planning quality, tool selection and permissions, state tracking, recovery from failed steps, verification discipline, boundedness, reproducibility, side effects, final task success.

A successful pull request is not sufficient if the agent used unsafe permissions, modified unrelated files, or produced hidden operational risk.

From human intent → autonomous plan → permitted execution surface → human review of evidence.

Each boundary is a place where instrumentation, permissions, and audit trails must be defined.

Risk routing must consider both task risk and model uncertainty.

AI behavior changes because of model upgrades, prompt edits, retrieval changes, repository evolution, or policy configuration.

Monitor: finding volume, precision proxies, action rates, cost, latency, category mix, failure modes, guardrail outcomes.

12,400 PRs, twelve weeks, treated vs. matched control cohort.

Net: −0.4 min — a 2.3% reduction, not the 19% headline.

Escaped-defect rate falls 8% (significant). Quality-retention factor \(Q_r = 1.05\) makes QCOI marginally net-positive.

The case for the tool depends on the quality effect, not the raw time saving.