The PM Who Can't Explain the Failure: Why Shipping AI Requires Proof, Not Just a PRD

Product & Strategy 2026-10-30 • ~23 min read

The PM Who Can't Explain the Failure

Why Shipping AI Requires Proof, Not Just a PRD

Series Notice: Part III of the AI Product Leadership Series

← Part I: The New Product Sense: Knowing When AI Should Stop: Designing Asymmetric Escalation Boundaries and the Friction Vector.

← Part II: Why AI Product Sense Requires Systems Judgment: Why AI Is Collapsing the Boundary Between Product Strategy and System Architecture.

Part III (This Monograph): The PM Who Can't Explain the Failure: Why Shipping AI Requires Proof, Not Just a PRD.

What "94% Accuracy" Leaves Out

When a product team brings an AI feature to a launch review, the conversation almost always centers on a single score. The team ran the model against an internal test set, it hit 94% accuracy, and now they want sign-off to put it in front of customers.

Signing off on that single number could be dangerous because this could mean different things to different stakeholders. To the machine learning engineer, it might mean the model matched the expected answer on 94% of the rows in the test set. The product manager might define it as the feature working roughly nine times out of ten. Operations might assume that only 6% of incoming volume would land in their manual review queue, and Finance might look at it through a headcount savings perspective based on automating the other 94%. Legal might be waiting for someone to explain what happens inside the 6% where the model gets it wrong.

If 93% of the refund requests or compliance checks coming into your queue every day are routine approvals, a script that blindly approves everything without reading a word will score 93% accuracy. A top-line average cannot tell you whether the model learned the hard edge cases or just rode the easy majority.

Before greenlighting a launch, I ask product teams a much simpler question:

"When this feature gets it wrong, who takes the hit, what does that mistake cost us, and how will we know it happened before a customer tells us?"

If the PM answers by pointing back to the 94% score on the test set, they are not ready to ship. An average tells you how often the system succeeded on a past sample. It tells you nothing about which customers absorb the failures or how much each mistake costs to clean up.

Most teams do not cut corners on purpose. They get blindsided after launch because they evaluate probabilistic models using the same habits that worked for deterministic SaaS features. In practice, five bad habits show up repeatedly.

Launch Blind Spot	What the Team Does Wrong	How It Breaks in Production
1. Benchmark Substitution	The team treats public leaderboard scores as proof the model fits their workflow	The model passes general knowledge tests but fails on messy internal data
2. Average-Score Camouflage	The team hides expensive failure clusters inside a high overall average	The system handles easy requests while failing on complex, high-value ones
3. Prompt-as-Architecture	Engineers put `"CRITICAL: NEVER DO X"` into a prompt instead of writing code checks	Prompt instructions fail as soon as users phrase things in unexpected ways
4. Blindly Trusting "The Evals Passed"	The PM accepts a green test report without checking who graded the answer key	Small samples and bad labels in the test set hide real production risks
5. Mistaking Fluency for Fact	Reviewers assume a confident, well-written response means the answer is true	Polished prose lulls human reviewers into approving bad outputs

The Five Launch Blind Spots: Comparing pre-launch slide deck assumptions with production failure modes across data, control, and human review.

1. Benchmark Substitution

Model vendors compete by publishing charts showing their latest checkpoint beating the field on academic reasoning and coding benchmarks. None of those scores prove the model can handle your company's badly formatted PDFs or custom billing rules. Public benchmarks measure general capability, whereas a launch decision requires testing on your own messy inputs.

2. Average-Score Camouflage

A 95% overall accuracy score looks comfortable on a slide until you break it down by customer tier and find that most of the 5% error rate lands on your largest enterprise accounts. Averages smooth over where the damage happens. If you automate cheap, routine tickets while misfiring on complex transactions, you create far more support cost than you save.

3. Prompt-as-Architecture

When an AI feature violates a business rule during testing, the quickest patch is to type a warning in all-caps into the system prompt:

"IMPORTANT: Always verify account eligibility before issuing a credit.
Never approve requests over the limit."

Natural language prompts steer how a model writes, but they cannot enforce business rules. If a rule cannot be broken, engineers must enforce it in regular application code with a database lookup or an API check before the system takes action.

4. Blindly Trusting "The Evals Passed"

In traditional software, when engineering says the integration tests passed, the PM moves on. With AI, when engineering says "the evals passed," the PM needs to ask what is inside the test set. How many examples did the team test: forty, or four thousand? Did engineers make up synthetic test prompts at their desks, or did they sample real production logs? Do your own domain specialists even agree on the right answer key? If a PM has never looked inside the test set, they have no idea what standard the model just cleared.

5. Mistaking Fluency for Fact

Vendors train language models to write smooth, convincing sentences. A model will state a wrong figure or cite an outdated policy using the exact same calm, authoritative tone it uses when it is right. When an interface puts fluent AI drafts in front of busy human reviewers without showing which facts the system actually verified, reviewers quickly stop checking the source material and start rubber-stamping the output.

The PRD Is Not Enough: Pairing Feature Specs with a Release Contract

To fix these blind spots, we have to change what we write down before engineering starts building.

Take a typical requirement from a product spec for an AI feature:

"The assistant shall accurately summarize customer disputes
and recommend the appropriate resolution."

There is no way for QA or engineering to write a definitive pass/fail test for that sentence. What counts as "accurate" when the ticket history for a customer contradicts itself? Which dispute types are in scope? How many bad recommendations can we tolerate per week before the feature loses money, and what signal forces the system to hand the ticket back to a person? Without those boundaries, the requirement is just a wish.

In standard software, a PRD describes deterministic rules: when a user clicks a button, the service writes a record and updates the screen. The code either executes the rule or throws an exception. AI products still need a traditional PRD to specify user permissions, UI states, and audit logging. What a PRD cannot cover is the statistical behavior of the model itself. To govern that, product leaders need to pair the PRD with a Release Contract.

Dimension	The Traditional PRD (Feature Spec)	The AI Release Contract (Launch Proof)
Core Question	What features and workflows must engineering build?	What evidence shows the system is safe enough to turn on?
What It Governs	Deterministic rules: permissions, state changes, audit logs, UI flows	Model behavior: accuracy across segments, error caps, escalation rates
How Teams Test It	Unit, integration, and end-to-end tests in CI	Expert-graded test sets, checks on high-risk segments, replaying past failures
Pass/Fail Bar	Binary pass/fail against acceptance criteria for the feature	Minimum accuracy floor and hard caps on expensive mistakes
How It Fails	Crashes, timeouts, broken UI states, unhandled exceptions	Silent bad decisions, confident fabrications, overloaded queues for human reviewers
When Engineers Change It	Code review and automated CI test runs	Re-running the test set and shadow-testing before shifting live traffic

A PRD tells engineering what to build. A Release Contract tells the business under what conditions the model is allowed to run on its own.

Pricing the Mistakes: Where Accuracy Bars Should Come From

When teams do set an accuracy bar for a launch, they usually pick a round number like 90% or 95% because it sounds high. In practice, your required accuracy bar has nothing to do with round numbers; it comes directly from the unit cost of a wrong answer.

The financial math of any automated workflow comes down to comparing what you save when the model is right against what you lose when it is wrong:

$$ \begin{aligned} \text{Net Value} = &\ \underbrace{(\text{Success Rate} \times \text{Savings per Right Answer})}_{\text{Labor and Turnaround Savings}} \\ &- \underbrace{(\text{Error Rate} \times \text{Cost of a Mistake})}_{\text{Remediation, Unwind, and Support Cost}} \end{aligned} $$

If an internal tool saves \$5 of staff time every time it categorizes a document correctly, and fixing a miscategorized document also takes \$5 of staff time, a model that is right 85% of the time pays for itself immediately.

Now compare that to a workflow where a correct decision saves \$10 in review time, while a wrong decision (like approving an invalid chargeback or sending a broken contract term to a client) costs \$1,000 to unwind. At 95% accuracy, every 100 transactions will generate \$950 in labor savings from the 95 right answers, while the 5 mistakes cost \$5,000. The feature loses over \$4,000 on every 100 runs despite hitting a 95% accuracy bar. To break even on that workflow, the model has to be right more than 99% of the time.

Before picking an accuracy bar, the PM needs to separate mistakes by what they actually cost:

Outcome Type	What Happened	How to Treat It in the Release Contract
Correct Automation	The AI handled a valid request without human help	The Upside: Saves manual processing time and speeds up turnaround
Unnecessary Escalation	The AI was unsure and handed a routine case to a human reviewer	Friction Cost: Costs a few minutes of review time, without downstream harm
Expensive Mistake	The AI approved or executed a bad action on its own	Direct Loss: Must stay below the financial breakeven rate
Severe Harm	The AI broke a compliance rule or exposed customer data	Hard Floor: The PM cannot leave this to model probability; engineering must block it in code or require human sign-off

When mistakes are cheap to fix, you can ship at 85% accuracy and improve the model from live feedback. When a single bad output wipes out the savings of a hundred good ones, a high overall average will not protect your P&L.

When the AI Says "I Don't Know": Sizing the Human Fallback

Once a team realizes how much a wrong answer costs, the most common fallback is to tell engineering: "Whenever the model isn't 99% confident, have it abstain and route the ticket to a person."

On a whiteboard, if you route uncertain cases to human reviewers, it looks like a free safety net. In day-to-day operations, every abstention becomes a ticket in a real person's queue.

If you tune a model to be extremely cautious so it rarely makes an error, two operational problems surface quickly. First, if the model punts 80% of incoming requests to humans, the automation barely reduces your manual workload. Second, if a model update or a shift in user traffic suddenly causes the AI's escalation rate to jump from 10% to 35% in a single week, your operations team gets buried:

$$ \begin{aligned} \text{Cautious Model Update} &\longrightarrow \text{Escalation Rate Spikes} \\ &\longrightarrow \text{Human Review Queue Backs Up} \\ &\longrightarrow \text{Turnaround SLAs Missed} \\ &\longrightarrow \text{Reviewers Skim and Rubber-Stamp} \end{aligned} $$

Anyone who has managed an operations queue knows what happens when human reviewers face a 500-ticket backlog on a Friday afternoon: they stop checking the details and start clicking "Approve" just to clear the queue. An unbudgeted spike in AI escalations degrades the exact human oversight you were counting on for safety.

That is why a Release Contract has to track Accuracy (how often the AI is right when it acts) alongside Coverage (how much volume it resolves without help) and Queue Capacity (how many escalations your operations team can realistically review per day without missing turnaround targets).

What Goes Into a One-Page Release Contract

You do not need a fifty-page governance binder to run this well. Before turning on an AI workflow, the PM and engineering lead can align on a one-page Release Contract covering six points:

Contract Section	The Question the PM Must Answer	Example Target in the Release Contract
1. Scope Boundaries	What is the AI allowed to touch, and what bypasses it?	Handles standard tier-1 requests; enterprise and regulated accounts route directly to staff.
2. Segment Accuracy	Does it work across our hardest segments, not just the easy average?	Must clear accuracy bars independently on noisy inputs, edge cases, and high-value accounts.
3. Error Cost Cap	What is the maximum rate of bad approvals before we lose money?	Expensive false approvals stay below the financial breakeven rate; zero severe compliance misses on the test set.
4. Daily Escalation Limit	How many daily escalations can our operations team absorb?	Daily escalation volume stays within existing staff capacity so turnaround times hold steady.
5. Speed & Unit Cost	Does the feature run fast enough and cheaply enough for our margins?	Hard ceiling on P95 latency and total cost to run the model per resolved task.
6. Rules for Partial Rollout	If the model passes on easy segments but fails on hard ones, what ships?	Pre-agreed rule allowing launch on passing segments while keeping failing segments manual.

The sixth row, Rules for Partial Rollout, is often the most useful part of the document. Without it, launch meetings turn into binary arguments between shipping a flawed model everywhere or delaying the whole project. When the team agrees on segment rules upfront, a PM can turn on automation for the 60% of routine requests where the model cleared the bar, while keeping the harder 40% on manual workflows until engineering fixes the remaining issues.

Why Launch Sign-Off Expires

When you ship a traditional software feature, the code behaves the same way six months later unless someone deploys a change. An AI feature can degrade in production even when your code and model weights stay untouched, because the environment around the model keeps moving.

Customers upload new file formats or change how they phrase requests. Internal teams update pricing tiers, refund rules, or help-center articles, leaving the search index out of sync with current policy. Over time, human reviewers also get used to the AI being mostly right and stop scrutinizing borderline recommendations.

$$ \begin{aligned} \text{Green Launch Sign-Off} &\longrightarrow \text{Live Inputs and Document Formats Shift} \\ &\longrightarrow \text{Silent Error Drift Across Automated Actions} \\ &\longrightarrow \text{Segment-Level Pullback and Re-Testing} \end{aligned} $$

Because pre-launch test results go stale, a Release Contract needs three ongoing operational checks:

Sample Decisions the AI Made on Its Own: When the AI escalates a request to a person, a human reviewer checks it. When the AI resolves a request automatically, nobody looks at it unless the customer complains. To catch silent errors, the operations team needs to pull a small random sample (even 1% to 2%) of fully automated decisions every week and grade them.
Alert When Escalation Volume Swings: If your system normally escalates 15% of volume to human reviewers, and this week it escalates 35% (or drops to 2%), something changed in the incoming traffic or the pipeline. When teams track the escalation rate daily, they often catch upstream data problems days before accuracy metrics dip.
Put Kill Switches on Each Segment: If an AI feature starts misbehaving on one request type, the team should not have to deploy code in an emergency to stop it. Product and operations should be able to route a specific customer segment or request type back to manual handling through a simple config flag.

When Things Break: Looking Past "The Model Hallucinated"

When a bad AI output reaches a customer, the initial bug report almost always says the same thing: "The model hallucinated."

If you stop the investigation there, you treat the language model like an unpredictable black box and ignore the rest of the software stack. In practice, most production incidents happen in the plumbing around the model rather than in the model's weights:

System Layer	What Actually Went Wrong	How Engineering Fixes It
1. Search & Context	The search step pulled an outdated policy document, or cut off the bottom of a long page to save latency.	Fix how documents are split, keep the search index fresh, or expand the context limit.
2. Model Reasoning	The prompt contained the right facts, and the model still drew the wrong conclusion.	Upgrade the model checkpoint, simplify the instruction, or split one complex step into two.
3. Tool & API Calls	The model tried to call an internal service, passed a bad parameter, or hit an unhandled timeout.	Add strict schema checks and retry logic around tool calls.
4. Guardrail Code	A check existed in code, but it only verified that an ID field was non-empty rather than checking if the ID was valid in the database.	Replace surface-level text checks with database lookups before the system runs an action.
5. Reviewer Screen	The interface buried risk warnings in small gray text, making it easy for a rushed human reviewer to click approve.	Redesign the review screen so high-risk fields require explicit confirmation.

Four of those five layers (search, tool calls, guardrail code, and the reviewer screen) are ordinary software engineering and product design problems. When a PM insists on tracing an incident across all five layers instead of blaming "hallucinations," most "AI failures" turn out to be application bugs that engineering can fix directly.

Who Owns the Launch Call?

When an AI feature fails in production and responsibilities are fuzzy, every team points to their own slice of the workflow. Engineering points out that they built the exact flow described in the spec. Data science notes that the model cleared the offline F1 score target in their notebook. Product says engineering told them the test set passed, and Legal asks why nobody looped them in before the team turned on live customer traffic.

Clear decision rights prevent that stall:

Role	What They Own	Their Say in the Launch
Product Manager	Defines scope boundaries, the Release Contract, limits on error cost, and fallback plans.	Makes the business go/no-go call based on whether the system meets every requirement in the Release Contract.
Engineering & ML	Builds the pipeline, guardrail code, latency budgets, and monitoring alerts.	Certifies technical stability and proves that the guardrail code holds up.
Domain Experts / Ops	Grade the answer key for the test set and staff the escalation queues.	Confirms the test set matches real work and the operations team can absorb the escalation rate.
Legal, Risk & Compliance	Sets non-negotiable boundaries around regulation, privacy, and customer harm.	Holds veto power if the system breaches legal or safety limits.

The PM does not need to write test scripts or draft legal policy. The PM's role is to bring those pieces together into a single release decision, making sure test results by segment back up every capability claim and that the operations team is staffed for the escalations before live traffic shifts.

How This Changes High-Pressure Launch Decisions

You can see the value of this setup clearly when a team hits a deadline crunch. When a model provider gives 30 days' notice that they are retiring the model version your product runs on, engineering has to test the replacement checkpoint quickly. Suppose the new model scores higher on public benchmarks, but on your internal test set, overall accuracy dips slightly, and on one complex, high-value request type, the new model makes several expensive mistakes. At the same time, executive leadership is pushing to expand the feature to all customers this quarter.

Without a Release Contract, the PM is stuck between two bad options: cave to schedule pressure and ship the regression to everyone, or halt the entire rollout and miss the quarter's goals.

With a Release Contract already in place, the PM has a third option: 1. Ship where the evidence holds: Cut over to the new model for the 60% of routine volume where it passed every accuracy and cost bar in the Release Contract. 2. Fence off the failing segment: Turn off automated execution for the complex segment that failed on the test set, routing those cases to human reviewers while engineering checks whether the regression came from how the system reads documents or from the guardrail code. 3. Price the temporary staffing gap: Show leadership the concrete numbers: when you route that extra volume to the operations team for sixty days, you need a temporary overtime budget, which costs a small fraction of what the company would lose by letting the model auto-approve bad transactions on that segment.

The PM does not need to lecture leadership on model internals. They just need to show which segment is ready to run, which segment needs human coverage, what that temporary coverage costs, and what result on the test set will let them turn full automation back on.

The Adversarial Stress-Test: Five Critical Inquiries

Before adopting this approach, it helps to weigh the five strongest objections you will hear from engineering, finance, and operations.

1. What is the strongest argument against this approach?

The Objection: "Our operations team already makes mistakes on 7% of requests and costs a lot to run. If we delay an AI launch for weeks to build segment test sets and a Release Contract, we waste money we could be saving right now. Why hold AI to a 99% standard when humans operate at 93%?"
The Counter-Analysis: If machine mistakes and human mistakes behaved the same way, holding AI to a stricter bar would make no sense. They differ in two ways. First, human errors on routine queues are often cautious (such as escalating a case unnecessarily), whereas an unguarded automation error can trigger an unrecoverable payout or customer-facing mistake. Second, human mistakes are mostly random one-off slips caused by fatigue, while a model bug is systematic: if a checkpoint misreads a specific template or policy clause, it will repeat that exact error on 100% of matching cases at scale. You do not need perfection to beat a human baseline, but you do need segment testing to make sure the model's errors do not cluster inside one expensive blind spot.

2. Under what circumstances would this fail?

Gaming the Test Set (Goodhart's Law): A Release Contract becomes useless paperwork as soon as teams start curating the test set just to make the dashboard green. If teams drop messy edge cases from the test set, test only on clean synthetic examples, or tune prompts to memorize the test rows, the Release Contract will report passing scores while live traffic breaks.

3. What would a skeptical colleague object to?

The Pushback on Shipping Speed: "Building expert-graded test sets and a Release Contract makes sense for a regulated bank, but it will slow a fast-moving product team to a crawl."
The Resolution: Match the weight of your testing to the cost of a mistake:
Tier 1 (Shadow Mode): The AI runs silently in the background while humans do the actual work. You can ship shadow mode on day one with minimal testing and use the logged predictions to build your test set from real traffic.
Tier 2 (Human-in-the-Loop Copilot): The AI drafts a response or highlights fields, and a human reviewer checks it and clicks send. This requires moderate testing focused on source citations and keeping reviewers from rubber-stamping outputs.
Tier 3 (Full Autonomy): The AI acts directly on customer accounts or moves money without human review. Here, upfront testing against a Release Contract is much cheaper than cleaning up a public incident.

4. Which alternative deserves more consideration?

Keep Decisions in Deterministic Code, Use AI Only to Parse and Draft: Before building an autonomous agent that makes final approval or routing decisions, consider whether you can keep the final decision rule inside ordinary deterministic code while you use the language model strictly to extract fields or summarize documents for a human reviewer. You give up full automation on ambiguous cases, but you remove the risk of a model guessing wrong on the core transaction.

5. What am I assuming that may not be true?

Domain Experts Agree on the Answer Key: A Release Contract assumes your internal specialists can grade the test set consistently. In many real-world workflows, two experienced reviewers will legitimately disagree on the right handling for 10% to 15% of borderline cases.
You Can Undo Mistakes Later: When the operations team samples 2% of automated actions each week, that check assumes you can absorb or reverse the week's errors if you spot drift on Friday. In workflows where actions cannot be undone (like external emails or instant payouts), weekly sampling only tells you how much money you already lost.
Company Incentives Reward Caution: This approach assumes leadership will back a PM who narrows a launch scope when a segment fails on the test set. In organizations where quarterly goals reward shipping "AI features" by a deadline regardless of unit economics, teams face strong pressure to ignore denominator problems and ship anyway.

One-Line Synthesis

Strong AI product leadership comes down to three practical habits: prove where a system is safe to run, know when that proof expires, and price the human fallback before live traffic crosses the line.