How to Build a Governed AI Evaluation System: Constrained Optimization and Four-Layer Triangulation

A Systems Engineering Blueprint for Principal PMs, Evaluation Engineers, and AI Architects

The architectural objective is not to design an ungameable metric. It is to build a verification system in which no single proxy—benchmark, judge, telemetry, or expert audit—has enough unilateral authority to clear a deployment on its own.

Prerequisite Reading: For an introduction to why AI metrics get gamed and the three classic failure modes (verbosity, refusal inflation, and sycophancy), start with Part 1: Goodhart’s Law in Generative AI: When Better Evals Produce Worse Products.

The Problem with Composite Scoring

When engineering teams recognize that optimizing for an isolated metric produces pathological behavior, their initial instinct is often mathematical: blend every competing priority into an exponential composite scoring formula.

Score = (Accuracy ^ 1.2) * (1 / Latency) * (1 - Toxicity_Penalty)

This mathematical instinct fails in production. Multiplying disparate dimensions introduces arbitrary weighting through mathematical structure:

• Why should latency decay exponentially while toxicity subtracts linearly?
• How does an engineering team defend weighting a 200ms latency increase against a 1.5% drop in citation accuracy?
• In practice, squads tweak exponents and scaling constants until the composite formula approves the model checkpoint they intended to ship.

Composite scores create an illusion of rigor while obscuring dangerous trade-offs. A high overall score routinely masks unacceptable localized degradations—such as severe sycophancy on high-risk queries or refusal spikes on ambiguous customer prompts.

To build an evaluation harness that resists Goodhart drift, teams must replace composite formulas with constrained optimization and multi-layer triangulation.

Constrained Optimization: Target + Failure Envelopes

Constrained optimization replaces weighted arithmetic with explicit step-functions. The architecture establishes a clear separation between the primary optimization objective and non-negotiable operational boundaries.

flowchart TD
    Target["<b>Primary Yield Target (Optimization Goal)</b><br/><b>Maximize Verified Task Completion</b> on valid in-domain requests"]

    Target ==>|"Bound by Non-Negotiable Failure Envelopes"| Envelopes

    subgraph Envelopes ["Failure Envelopes (Hard Constraints)"]
        direction TB
        E1["<b>Safety & Legal Boundary:</b> Zero ungrounded claims or regulatory violations"]
        E2["<b>Calibrated Refusal Floor:</b> CRQ baseline — no evading valid enterprise tasks"]
        E3["<b>Latency SLA Budget:</b> P95 response time ≤ 3.5 seconds"]
        E4["<b>Token Density Floor:</b> Ratio ≥ 0.65 (eliminates filler preambles)"]
        E5["<b>Unit Margin Ceiling:</b> Inference cost ≤ $0.02 per resolved session"]
        E1 ~~~ E2 ~~~ E3 ~~~ E4 ~~~ E5
    end

    style Target fill:#0f172a,stroke:#3b82f6,stroke-width:2px,color:#ffffff
    style Envelopes fill:#111827,stroke:#ef4444,stroke-width:2px,color:#f8fafc
    style E1 fill:#1f2937,stroke:#f87171,stroke-width:1px,color:#fca5a5
    style E2 fill:#1f2937,stroke:#f87171,stroke-width:1px,color:#fca5a5
    style E3 fill:#1f2937,stroke:#f87171,stroke-width:1px,color:#fca5a5
    style E4 fill:#1f2937,stroke:#f87171,stroke-width:1px,color:#fca5a5
    style E5 fill:#1f2937,stroke:#f87171,stroke-width:1px,color:#fca5a5

1. The Primary Yield Objective

The engineering squad focuses on a single goal: maximizing verified task completion on valid, in-domain customer requests. Every prompt modification, fine-tuning run, and retrieval parameter tweak aims to increase this yield.

2. Non-Negotiable Failure Envelopes

The primary target is bound by hard boundary conditions that cannot be violated under any circumstances:

• Safety & Regulatory Boundary: Zero ungrounded legal, clinical, or policy violations on high-stakes workflows.
• Calibrated Refusal Floor: The system cannot inflate completion rates by guessing on ambiguous inputs or refusing legitimate edge cases.
• Latency SLA Budget: P95 response times must remain within strict human-interaction tolerances.
• Token Density Floor: Output cannot inflate length or introduce redundant preamble boilerplate to score higher on automated judge rubrics.
• Unit Economics Ceiling: Cost per verified resolution must remain within gross margin targets.

The Golden Rule of Failure Envelopes: You cannot trade off a non-negotiable boundary to buy higher accuracy. If a candidate prompt or model checkpoint boosts task completion by 8% but increases P95 latency by 1.8 seconds or triggers defensive refusals on valid edge cases, it fails the release gate immediately.

Decision Rights Matrix

Who determines whether an accuracy increase justifies an increase in operational latency or inference cost? Distributing decision rights across accountable functional owners prevents hidden engineering compromises:

The Four-Layer Triangulation Engine

No single measurement layer provides complete visibility into generative AI quality. A robust governance system relies on Four-Layer Triangulation, treating every layer as an imperfect proxy with distinct structural blind spots.

flowchart TD
    subgraph L1 ["Layer 1: Offline Capability ('Can the model do the job?')"]
        direction LR
        L1_A["<b>Measurement:</b> Golden datasets, schema tests, extraction precision"]
        L1_B["<b>Operational Rule:</b> Dynamic refreshes from live production incident logs"]
    end

    subgraph L2 ["Layer 2: Adversarial Bounds ('How does the system break?')"]
        direction LR
        L2_A["<b>Measurement:</b> Injected flawed premises, sycophancy probes, edge stress"]
        L2_B["<b>Operational Rule:</b> Derived from sanitized customer support escalations"]
    end

    subgraph L3 ["Layer 3: Production Telemetry ('What do real users experience?')"]
        direction LR
        L3_A["<b>Measurement:</b> Re-prompt rates, Time-to-Action (TTA), copy-paste discard rate"]
        L3_B["<b>Operational Rule:</b> Multi-signal negative divergence tracking across cohorts"]
    end

    subgraph L4 ["Layer 4: Human Outcome Auditing ('Did it deliver business value?')"]
        direction LR
        L4_A["<b>Measurement:</b> Double-blind reviews by accredited domain practitioners"]
        L4_B["<b>Operational Rule:</b> Statistical inter-rater reliability to audit automated judges"]
    end

    L1 ==> L2 ==> L3 ==> L4

    style L1 fill:#0f172a,stroke:#38bdf8,stroke-width:2px,color:#f1f5f9
    style L2 fill:#0f172a,stroke:#f59e0b,stroke-width:2px,color:#f1f5f9
    style L3 fill:#0f172a,stroke:#a855f7,stroke-width:2px,color:#f1f5f9
    style L4 fill:#0f172a,stroke:#10b981,stroke-width:2px,color:#f1f5f9
    style L1_A fill:#1e293b,stroke:#0284c7,stroke-width:1px,color:#e2e8f0
    style L1_B fill:#1e293b,stroke:#0284c7,stroke-width:1px,color:#e2e8f0
    style L2_A fill:#1e293b,stroke:#d97706,stroke-width:1px,color:#e2e8f0
    style L2_B fill:#1e293b,stroke:#d97706,stroke-width:1px,color:#e2e8f0
    style L3_A fill:#1e293b,stroke:#9333ea,stroke-width:1px,color:#e2e8f0
    style L3_B fill:#1e293b,stroke:#9333ea,stroke-width:1px,color:#e2e8f0
    style L4_A fill:#1e293b,stroke:#059669,stroke-width:1px,color:#e2e8f0
    style L4_B fill:#1e293b,stroke:#059669,stroke-width:1px,color:#e2e8f0

Layer 1: Offline Capability Evaluations ("Can it do it?")

• Role: Evaluates baseline competency across curated golden datasets (schema compliance, retrieval recall, entity extraction, factual consistency).
• Structural Vulnerability: Susceptible to benchmark leakage, synthetic data artifacts, and static distribution decay.
• Operational Control: Test suites must be updated dynamically from production incident logs and regulatory changes rather than arbitrary calendar schedules.

Layer 2: Adversarial Failure Envelopes ("How does it break?")

• Role: Probes boundary conditions by injecting contradictory evidence, out-of-domain edge cases, and flawed user premises to verify refusal calibration and anti-sycophancy.
• Structural Vulnerability: Synthetic adversarial suites risk over-indexing on synthetic edge cases that never occur in real production workflows.
• Operational Control: Adversarial test cases must be derived directly from sanitized production escalations and customer support post-mortems.

Layer 3: Production Behavioral Telemetry ("What actually happens?")

• Role: Monitors live interaction signals (session completion, edit distance, copy-paste events, Time-to-Action, re-prompt rates).
• Structural Vulnerability: Telemetry is inherently ambiguous. End users adapt their behavior around system flaws:
• High Copy-Paste: Can signal an exceptional response, or an output so fragmented that the user copied it into an external editor to fix it manually.
• Low Re-Prompt Rate: Can indicate first-turn resolution, or an exhausted user who gave up and abandoned the tool.
• Counterfactual Value: When an enterprise AI assistant prevents a contractual concession or adverse clinical event, the averted risk emits no anomalous clickstream event.
• Operational Control: Never treat clickstream telemetry as ground truth. Track negative trend divergences across multiple telemetry streams simultaneously.

Layer 4: Human Expert Outcome Auditing ("Did it improve the customer's job?")

• Role: Periodic double-blind reviews of sampled production interactions conducted by accredited domain experts (clinicians, attorneys, underwriters).
• Structural Vulnerability: Expensive, slow, and subjective. Flawed review rubrics simply institutionalize organizational bias.
• Operational Control: Use human calibration not as an inline gate for daily traffic, but as a meta-evaluator to audit the automated judges. Calculate inter-rater reliability between automated judges and expert panels using task-appropriate statistical measures (such as Cohen’s $\kappa$ or Gwet’s AC1). When agreement falls below baseline, pause automated promotions until judge rubrics are re-anchored.

The Evidence Independence Audit

Triangulation is an illusion if all four layers share the same structural assumptions.

If the offline golden set, the automated LLM judge, the adversarial edge tests, and the human annotator rubrics all draw from the same prompt templates, vendor model family, or corporate policy interpretations, four evidence streams produce one shared blind spot.

Before treating multi-layer evidence as verification, run an Evidence Independence Audit:

1. Model Diversity: Does the automated judge utilize a different model family and architecture than the generator model? Frontier models trained on similar preference corpora often share identical verbosity and sycophancy biases; true independence requires distinct training distributions.
2. Methodological Diversity: Does the system combine deterministic programmatic assertions (regex, JSON grammar constraints) with probabilistic LLM evaluators and behavioral signals?
3. Incentive Diversity: Are human raters compensated based on evaluation accuracy and depth rather than review volume?
4. Data Isolation: Are golden evaluation sets strictly segregated from fine-tuning corpora and prompt-tuning workflows?

Calibrated Refusal Quality: Balancing Precision and Recall

A frequent failure mode in production AI is optimizing exclusively for accuracy or exclusively for safety.

• Mandating that refusals cannot exceed 2% causes models to hallucinate answers on ambiguous or ungrounded queries to avoid triggering a refusal penalty.
• Mandating zero hallucination causes models to defensively refuse legitimate, safe user requests.

To eliminate this perverse incentive, govern by Calibrated Refusal Quality (CRQ):

$$\text{CRQ} = \text{Refusal Precision} \times (1 - \text{False Refusal Rate})$$

Where:

• Refusal Precision $= \frac{\text{Appropriate Refusals}}{\text{All Actual Refusals}}$ (Did the model refuse only when the query was genuinely invalid or unanswerable?)
• False Refusal Rate $= \frac{\text{False Refusals}}{\text{Valid, In-Domain Requests}}$ (Did the model reject legitimate users attempting valid tasks?)

CRQ penalizes gaming in both directions. A model that refuses all hard queries fails on False Refusal Rate; a model that guesses indiscriminately fails on Refusal Precision.

Dual-Track Architecture: Assurance vs. Discovery

Applying rigid failure envelopes to early-stage product development kills innovation before value is discovered. Product organizations must maintain a Dual-Track Architecture:

flowchart TD
    Work["<b>Incoming AI Workflow</b>"] --> Decision{"Does the team know what<br/>'good' looks like for this workflow?"}

    Decision ==>|"YES: Established Tasks"| Assurance["<b>Capability Assurance Track</b><br/>─────────────<br/>• Enforce non-negotiable failure envelopes<br/>• Automated 4-gate CI/CD deployment blocking<br/>• Shadow production traffic evaluation<br/>• Strict latency and unit economics ceilings"]

    Decision ==>|"NO: New Paradigms"| Discovery["<b>Product Discovery Track</b><br/>─────────────<br/>• Reversible opt-in customer cohorts<br/>• Workflow shadowing & heuristic exploration<br/>• Structured qualitative Decision Journals<br/>• Rapid iteration without premature gating"]

    style Work fill:#1e293b,stroke:#64748b,stroke-width:1px,color:#f8fafc
    style Decision fill:#334155,stroke:#94a3b8,stroke-width:2px,color:#f8fafc
    style Assurance fill:#0f172a,stroke:#3b82f6,stroke-width:2px,color:#93c5fd
    style Discovery fill:#0f172a,stroke:#10b981,stroke-width:2px,color:#a7f3d0

1. The Capability Assurance Track

• Context: Established, high-volume production workflows where acceptable latency, safety bounds, and task definitions are clear (e.g., invoice extraction, policy question answering).
• Operating Model: Enforces hard failure envelopes, automated CI/CD deployment gates, and shadow traffic validation.

2. The Product Discovery Track

• Context: Early-stage category exploration, novel multi-agent interaction paradigms, or complex workflows where "good" is not yet understood.
• Operating Model: Uses lightweight guardrails, opt-in customer cohorts, workflow shadowing, and Decision Journals:
• Decision Journals: For high-stakes, low-volume workflows (e.g., M&A due diligence, complex medical diagnostics) where statistical telemetry is too sparse to evaluate, teams record the prompt context, model output, human practitioner decision, and downstream business outcome in structured qualitative logs.
• Reversible Cohorts: Expose novel capabilities to small, opt-in user segments with full audit logging and immediate rollback mechanisms, allowing breakthrough value to emerge without risking enterprise-wide operational disruption.

The 4-Gate CI/CD Release Pipeline

To operationalize these controls into an automated software delivery process, structure deployment pipelines around four sequential verification gates. A candidate prompt, model checkpoint, or routing policy must clear each gate sequentially before promoting to production.

flowchart TD
    G1["<b>GATE 1: Mechanistic & Syntactic Validation</b><br/>• Constrained decoding grammars (JSON/regex)<br/>• Latency SLA budget (P95 ≤ 3.5s)<br/>• Token density floor (ratio ≥ 0.65)"]
    G2["<b>GATE 2: Adversarial Failure Envelopes</b><br/>• Injected flawed premises (anti-sycophancy)<br/>• Calibrated Refusal Quality (CRQ baseline)<br/>• Citation entailment & conflict tests"]
    G3["<b>GATE 3: Constrained Task Yield Evaluation</b><br/>• Verified completion on golden test sets<br/>• Concurrent failure envelope enforcement<br/>• Zero cross-metric trade-offs allowed"]
    G4["<b>GATE 4: Shadow Traffic & Telemetry Monitoring</b><br/>• Live production shadow routing<br/>• Re-prompt spikes & copy-paste discard tracking<br/>• Tenant-isolated evaluation perimeters"]
    Prod[("<b>Production Canary Release</b>")]

    G1 ==>|"Pass Gate 1"| G2
    G2 ==>|"Pass Gate 2"| G3
    G3 ==>|"Pass Gate 3"| G4
    G4 ==>|"Deploy to Canary"| Prod

    style G1 fill:#1e293b,stroke:#38bdf8,stroke-width:2px,color:#f8fafc
    style G2 fill:#1e293b,stroke:#f59e0b,stroke-width:2px,color:#f8fafc
    style G3 fill:#1e293b,stroke:#6366f1,stroke-width:2px,color:#f8fafc
    style G4 fill:#1e293b,stroke:#10b981,stroke-width:2px,color:#f8fafc
    style Prod fill:#047857,stroke:#34d399,stroke-width:2px,color:#ffffff

Gate 1: Mechanistic and Syntactic Validation

Before running quality evaluations, test programmatic invariants:

• Constrained Decoding Grammars: Verify that outputs conform strictly to required JSON schemas or structural formats.
• Latency SLA Budget: Reject any candidate exceeding P95 latency limits under simulated concurrent load.
• Token Density Floor: Measure raw information density against total word count. If candidate outputs inflate boilerplate preambles to game length bias, the build fails immediately.

Gate 2: Adversarial Failure Envelopes

Subject the candidate to edge cases designed to trigger the failure modes its primary objective tends to induce:

• Inject provably false user premises to verify anti-sycophancy.
• Test against ambiguous, out-of-domain, or ungrounded queries to calculate Calibrated Refusal Quality (CRQ).
• Test citation entailment against contradictory or outdated corpus documents.

Gate 3: Constrained Task Yield Evaluation

Evaluate the candidate against the versioned golden evaluation suite:

• Verified task completion must meet or exceed the active production baseline.
• All failure envelopes (refusal quality, latency, unit cost) must remain within their negotiated boundaries concurrently.
• No single metric can be compromised to purchase an improvement in another.

Gate 4: Shadow Traffic and Divergence Telemetry

Route a risk-calibrated fraction of live production traffic to the candidate in shadow mode:

• Monitor behavioral divergence signals: re-prompt spikes, elevated edit distance, and abnormal copy-paste patterns.
• Log all edge cases within tenant-isolated evaluation perimeters to maintain data privacy.
• If shadow telemetry matches or exceeds active baseline metrics across the observation window, promote the candidate to canary release.

Core Operating Directives for AI Engineering Leaders

1. Enforce Hard Failure Envelopes over Composite Formulas: Reject any model candidate that achieves higher accuracy by broadening its failure envelopes. A 6% improvement in task completion is invalid if it introduces defensive refusals or unacceptable latency.
2. Audit the Auditor Continuously: Measure inter-rater agreement between automated LLM judges and human domain experts. When statistical agreement drops below baseline, halt automated deployments until judge rubrics are recalibrated.
3. Treat Telemetry as an Early Warning System, Not Ground Truth: Use clickstream telemetry to identify divergence, not proof of quality. When high copy-paste volume coincides with high re-prompt rates, assume users are manually repairing broken outputs.
4. Retire Static Benchmarks Dynamically: Refresh evaluation datasets based on production incident logs, customer escalations, and regulatory updates rather than fixed calendar intervals.
5. Decouple Product Discovery from Capability Assurance: Keep constraints lightweight during early exploration to discover user value; enforce rigorous failure envelopes when scaling validated workflows into production.
6. Align Organizational Incentives to Business Outcomes: Measurement architecture addresses the symptom; incentive structure addresses the cause. If product managers, engineers, and data annotators are evaluated on response-level proxy scores rather than downstream workflow completion, no governance architecture will prevent gaming.

One-Line Synthesis

Durable AI governance does not seek an ungameable metric; it enforces a constrained optimization system where no single proxy can unilaterally clear a production release.

The ideas in this post are my own — they emerged from questions I asked while learning applied AI concepts and putting them to work in my job and my projects. The prose was developed with AI assistance.