How to Build a Governed AI Evaluation System: Constrained Optimization and Four-Layer Triangulation

A Systems Engineering Blueprint for Principal PMs, Evaluation Engineers, and AI Architects

Production verification requires an architecture where no single proxy (golden test set, LLM judge, user telemetry, or expert audit) holds enough authority to clear a deployment on its own.

Prerequisite Reading: For an introduction to why AI metrics get gamed and the three classic failure modes (verbosity, refusal inflation, and sycophancy), start with Part 1: Goodhart’s Law in Generative AI: When Better Evals Produce Worse Products.

Why Do Single Composite Scores Fail in Production?

When engineering teams recognize that optimizing for a single metric produces perverse behavior, their first instinct is often mathematical: compress every competing priority into a single utility formula. Because automated CI/CD gates, hyperparameter sweeps, and leaderboards demand a single sortable rank (argmax Score), teams build composite scoring formulas:

$$ \begin{aligned} \text{Composite Score} = &\ \left(\text{Accuracy}^{1.2}\right) \\ &\ \times \left(\frac{1}{\text{Latency}}\right) \\ &\ \times \left(1 - \text{Penalty}_{\text{toxicity}}\right) \end{aligned} $$

This instinct fails the moment it hits production. When teams mash accuracy, latency, safety, and cost into one magic score, they might as well average apples, sports cars, and tax audits. First, there is no real-world exchange rate between completely different metrics: why should a 200ms latency increase be offset by a 1.5% drop in citation accuracy? Engineering teams cannot justify trade-offs like that without inventing arbitrary conversion factors. Second, composite scoring hides critical failures: a high score on easy invoice lookups can mathematically conceal severe safety failures on complex compliance edge cases, letting a model checkpoint pass release gates while silently breaching company policy. Finally, teams inevitably game the exponents: whenever the composite formula blocks a feature the squad wants to ship, someone tweaks the weights until the build turns green. To build an evaluation harness that survives production pressure, teams must replace composite formulas with constrained optimization and multi-layer triangulation.

Constrained Optimization: Target + Failure Envelopes

Constrained optimization replaces weighted arithmetic with explicit step-functions. Rather than blending competing objectives into a single number, engineers formulate the evaluation as a constrained optimization problem:

$$ \begin{aligned} \text{maximize} \quad & \text{Verified Task Yield} \\ \text{subject to} \quad & \text{Safety and Legal Violations} = 0 \\ & \text{Calibrated Refusal Quality} \ge \text{Baseline}_{\text{CRQ}} \\ & \text{P95 Latency} \le 3.5\text{ s} \\ & \text{Token Information Density} \ge 0.65 \\ & \text{Inference Cost per Session} \le \$0.02 \end{aligned} $$

The mathematical separation is strict: the engineering squad pursues a single objective (maximizing verified task completion on valid, in-domain customer inputs) while all competing business priorities act as hard boundaries.

Under this setup, metrics fall into two distinct classes:

1. The Primary Yield Target: Every prompt change, fine-tuning pass, context pruning rule, and retrieval tweak focuses on raising task yield on legitimate customer requests.
2. Non-Negotiable Failure Envelopes: Statutory compliance, latency budgets (P95 $\le 3.5$s), calibrated refusal floors, minimum token density, and cost caps per session act as binary pass/fail gates.

The Operational Invariant: You cannot buy task yield by borrowing from a failure envelope. If a candidate prompt or model checkpoint boosts task completion by 8% but adds 1.8 seconds to P95 latency or triggers defensive refusals on valid edge cases, it fails the release gate immediately.

Decision Rights Matrix: Policy Definition vs. Day-to-Day Execution

In enterprise production environments, Risk, Legal, and Compliance do not attend daily engineering standups or approve individual prompt edits. If you require Legal and Compliance to review every prompt change manually, daily deployments grind to a halt.

Instead, enterprise governance operates through pre-authorized policy envelopes:

1. Upfront Boundary Setting: Oversight teams (Legal, Risk, Finance, and domain experts) set non-negotiable thresholds, statutory guardrails, and margin limits ahead of time.
2. Automated Squad Enforcement: The AI product and engineering squad translates those boundaries into code checks and automated CI/CD gates.
3. Autonomous Execution with Exception Escalation: The squad has full authority to ship daily as long as all pre-authorized envelopes pass together. The squad brings oversight leads back into the loop only when asking to widen an envelope or approve an exception.

The Four-Layer Triangulation Engine

No single measurement layer gives complete visibility into generative AI quality. A reliable governance setup relies on Four-Layer Triangulation, treating every layer as an imperfect proxy with its own blind spots.

The Evidence Independence Audit

Triangulation is an illusion if all four layers share the same structural assumptions.

If the golden test set, the LLM judge, the adversarial tests, and the human review rubrics all draw from the same prompt templates, vendor model family, or policy assumptions, four evidence streams produce one shared blind spot.

Before treating multi-layer evidence as verification, engineering leads run an Evidence Independence Audit across four dimensions. First, for model independence, the LLM judge must run on a different model family and architecture than the generator; frontier models trained on similar preference data often share identical verbosity and sycophancy biases. Second, for methodological diversity, teams combine deterministic code checks (AST parsing, JSON schema validation, and regex rules) with LLM judges and runtime telemetry so code checks catch syntax bugs while judges catch meaning drift. Third, for incentive alignment, operations leads pay audit panels for catching subtle errors rather than paying per ticket, because piece-rate pay produces superficial approvals. Fourth, for data isolation, platform engineers keep golden test sets strictly separated from fine-tuning and prompt-tuning data so teams never train on the test.

Calibrated Refusal Quality: Balancing Precision and Recall

A frequent mistake in production AI is optimizing exclusively for accuracy or exclusively for safety. When leadership mandates that refusals stay below 2%, models start guessing on ambiguous or ungrounded queries just to avoid the refusal penalty. Conversely, when leadership demands zero hallucinations at all costs, models defensively refuse legitimate, safe user requests.

To remove both perverse incentives, govern refusals using Calibrated Refusal Quality (CRQ):

$$ \begin{aligned} \text{CRQ} = &\ \text{Refusal Precision} \\ &\times (1 - \text{False Refusal Rate}) \end{aligned} $$

Where:

• Refusal Precision $= \frac{\text{Appropriate Refusals}}{\text{All Actual Refusals}}$ (Did the model refuse only when the query was genuinely invalid or unanswerable?)
• False Refusal Rate $= \frac{\text{False Refusals}}{\text{Valid, In-Domain Requests}}$ (Did the model reject legitimate users attempting valid tasks?)

CRQ penalizes gaming in both directions. A model that refuses all hard queries fails on False Refusal Rate; a model that guesses blindly fails on Refusal Precision.

Dual-Track Architecture: Assurance vs. Discovery

If a product team applies rigid failure envelopes to early-stage prototypes, they kill exploration before finding what works. Product organizations avoid that trap by running a Dual-Track Architecture:

The Capability Assurance Track

High-volume, standardized production workflows (such as invoice data extraction, tier-1 dispute triage, or policy Q&A) run under formal Capability Assurance. Here, the task scope is clear, latency targets are fixed, and safety failure modes are well understood. Capability Assurance enforces hard automated failure envelopes, blocks non-compliant builds in CI/CD, and requires shadow traffic checks before cutting over live traffic.

The Product Discovery Track

When squads explore new multi-agent workflows, open-ended synthesis tasks, or unfamiliar UX patterns, strict metric gates choke off learning. Product Discovery replaces automated pipeline blocks with two lightweight guardrails. In high-stakes, low-volume workflows like M&A due diligence or oncology review where sample sizes are too small for automated judges, squads keep structured decision journals that log the prompt context, the model output, the clinician or counsel decision, and the final outcome. For interactive features, squads ship to reversible cohorts of opt-in users with full event logging and instant rollback switches, gathering real usage signal without putting company-wide SLAs at risk.

The 4-Gate CI/CD Release Pipeline

To turn these controls into an automated delivery process, platform engineers structure deployment pipelines around four sequential verification gates. A candidate prompt, model checkpoint, or routing rule must clear each gate in order before reaching production.

Gate 1: Mechanistic and Syntactic Validation

Before spending GPU budget on semantic evaluations, the pipeline runs deterministic code checks. Decoding grammars enforce strict schema rules (checking JSON structure, field types, and required keys). Load tests verify that P95 latency stays within SLA budgets under peak concurrency. Finally, a density filter blocks verbose preambles, halting the build if the ratio of informative tokens to total tokens drops below baseline.

Gate 2: Adversarial Failure Envelopes

Once syntax checks pass, the pipeline runs synthetic adversarial probes aimed at the exact shortcuts the target metric rewards. Injected contradictions test whether the model caves in to sycophancy, out-of-domain queries test Calibrated Refusal Quality (CRQ), and conflicting source documents test citation accuracy. Any drop in refusal precision or sycophancy resistance stops the build immediately.

Gate 3: Constrained Task Yield Evaluation

Gate 3 runs the candidate against the versioned golden test set. Task completion must meet or beat the active production baseline while every operational envelope (latency, cost per session, and refusal quality) passes at the same time. The pipeline allows no metric trade-offs: it rejects an 8% gain in extraction recall if session inference cost crosses the \$0.02 cap.

Gate 4: Shadow Traffic and Divergence Telemetry

The final gate mirrors a slice of live production traffic to the candidate model in shadow mode. The monitoring service watches for behavioral shifts against the live baseline: spikes in user re-prompts, larger edit distances before submission, and unusual session lengths. Once shadow telemetry holds steady across the observation window, the pipeline promotes the candidate to canary rollout.

The PM's AI Evaluation Triage Matrix: Resolving Engineering Trade-Offs in Sprint Reviews

During release triage and sprint planning, AI PMs constantly arbitrate metric trade-offs between engineers eager to ship model updates and stakeholders worried about production regressions.

Bring this triage matrix to sprint reviews to settle go/no-go debates:

Core Operating Directives for AI Engineering Leaders

In day-to-day practice, six operating habits keep an evaluation system honest. First, enforce hard failure envelopes instead of composite formulas, and reject any model candidate that buys higher accuracy by loosening latency or refusal bounds. Second, audit the LLM judge continuously against domain experts; whenever inter-rater agreement ($\kappa$) drops below baseline, pause automated releases until evaluation engineers recalibrate the judge rubric. Third, treat clickstream telemetry as an early warning signal rather than ground truth: when high copy volume coincides with high re-prompt rates, assume users are fixing broken outputs by hand.

Fourth, refresh golden test sets directly from production incidents, customer escalations, and regulatory updates rather than waiting for quarterly calendar reviews. Fifth, keep early Product Discovery separate from Capability Assurance so prototypes stay fast while mature workflows stay locked down. Finally, tie team incentives to completed customer workflows rather than isolated proxy scores; if leadership rewards PMs, engineers, and raters for single response scores, teams will eventually game the harness regardless of how well it is built.

The Adversarial Stress-Test: Five Critical Inquiries

To verify that this evaluation architecture holds up under enterprise skepticism and operational stress, we subject constrained optimization, four-layer triangulation, and the 4-gate CI/CD pipeline to five inquiries:

1. What is the strongest argument against this approach?

When a team runs a four-gate pipeline (syntax checks, adversarial stress tests, full golden test set sweeps, and shadow traffic observation) on every change, deployments take much longer. For squads shipping multiple times a day, a pipeline that takes hours or days to clear slows iteration and tempts engineers to route around the harness.

The Practical Resolution: Match gate depth to the blast radius of the change. Run fast deterministic checks (Gate 1) on every commit, run adversarial suites and golden test sets (Gates 2 and 3) on staging pull requests, and use asynchronous shadow traffic (Gate 4) to gate canary promotion so daily developer commits never block on multi-hour runs.

2. Under what circumstances would this fail?

This evaluation architecture breaks down under three specific conditions. First, when regulations, legal precedents, or internal policies shift quickly, golden test sets go stale faster than domain experts can update them, giving false confidence against outdated rules. Second, while shadow mode works cleanly for stateless, single-turn tasks like extraction or summarization, multi-turn agents that write to external databases or call third-party APIs cannot run live actions in shadow mode without elaborate mocks that diverge from production. Third, Layer 4 depends on attorneys, clinicians, and underwriters to anchor the LLM judges; when expert panels get tired or grade from vague rubrics, their agreement drops and the calibration baseline drifts.

3. What would a skeptical customer, client, or colleague object to?

A skeptical VP of Engineering will point out that hiring evaluation engineers, paying domain expert panels, and running shadow traffic infrastructure can cost more to build and maintain than the core AI feature itself.

The Practical Resolution: Scale the evaluation harness to match the workflow's liability tier. Low-stakes internal drafting tools only need deterministic syntax checks (Gate 1) and a basic golden test set (Gate 3); customer-facing autonomous workflows justify the full four-layer engine because a single regulatory fine or clinical error costs far more than the annual evaluation budget.

4. Which alternative deserves more consideration?

Instead of building large evaluation suites to verify that a probabilistic model follows business rules, teams can wrap smaller models inside deterministic state machines: runtime AST parsers, hardcoded rule engines, and strict egress filters that sanitize every output before it reaches a user.

5. What am I assuming that may not be true?

This architecture rests on three assumptions that deserve scrutiny. First, it assumes user clickstream telemetry gives an honest signal of product quality without hiding silent workarounds or users giving up outside the app. Second, it assumes LLM judges from different model providers have truly independent error patterns, even though shared web training data and preference datasets often bake the exact same blind spots into competing models. Third, it assumes domain experts have the time and focus to write and maintain ground-truth rubrics month after month.

One-Line Synthesis

Durable AI governance binds task optimization to hard operational failure envelopes, verified across four independent evidence layers where no single metric can clear a release on its own.