Before You Fund an AI Platform: Prove the Workflow First

Part 1 of 3 — AI Platform Budget Decisions

Platform investment justified by a misdiagnosed constraint produces the same outcome as no investment at all. The only difference is the bill.

The Pattern That Keeps Repeating

A product organization runs four squads. Four squads build four independent prompt pipelines. Someone proposes centralizing them. The business case projects $600k in productivity savings. Leadership approves it.

Six months later, squads are bypassing the platform using discretionary vendor credits. The platform team has become an intake bottleneck. The CFO is asking why software expenditure rose while delivery velocity declined.

The platform did not fail because shared infrastructure lacks value. It failed because the product team answered the wrong question first.

The right first question is not "Should we build a platform?" It is "Do we actually have a capability problem?"

Is Your Value Ceiling a Capability Problem or a Workflow Problem?

Here is how the misdiagnosis typically happens. An invoice extraction feature stalls at 22% monthly active adoption. The squad blames the model: "Extraction accuracy is 84%. We need a shared fine-tuning pipeline and a retrieval layer."

That may be wrong. And committing platform capital before testing that assumption is how organizations build infrastructure that nobody uses.

Two Starting Points: Brownfield vs. Greenfield

Product managers face this decision at two completely different moments in the product lifecycle:

1. The Brownfield Diagnostic (Improving an Existing AI Feature): You already deployed a scrappy local prompt wrapper or MVP. Adoption has stalled (e.g., an invoice extraction feature sitting at 22% monthly active adoption). The squad blames model accuracy: "Extraction is only 84%. We need platform budget, fine-tuning, and a vector database to hit 95%." The test proves whether accuracy is genuinely the adoption blocker before you spend capital fixing it.
2. The Greenfield Feasibility Test (Simulating AI in a Manual Workflow): You have a manual or rule-based process with no AI in production. The proposal asks for $250k to build an automated intelligence pipeline. Before writing a line of model plumbing, you run a Wizard of Oz test: insert human operators behind the interface to simulate near-perfect automated extraction and instant turnaround. If users still don't adopt the workflow when the output is artificially flawless, automated AI will not rescue it either.

The Capability Sensitivity Test

Whether testing an existing feature or simulating a new one, the diagnostic mechanism is the same. Hold model capability constant (or simulate it). Insert a human operator to correct or generate the outputs. Watch what happens to user task completion and retention.

If adoption rises substantially with verified output, improved capability is a credible value driver. Worth investigating.

But the test is not a clean experiment. A human operator does far more than fix model output. Three confounding interventions enter simultaneously:

• Contextual repair: The operator adds missing domain context, infers customer intent, and fixes ambiguous document formatting that the model missed.
• Ergonomic compensation: The operator navigates around awkward review interfaces, clunky validation steps, and poorly presented evidence—compensating for UX debt the model never touched.
• Decision threshold modification: The operator changes the implicit risk tolerance required to commit an action. Users may be completing tasks because a human vouched for the output, not because the output improved.

High adoption after human correction demonstrates that task quality matters. It does not prove that model accuracy alone is the binding constraint.

Name the result accurately. Call it a capability sensitivity test—not a capability vs. workflow diagnostic—to avoid overstating what the experiment shows.

If operators must correct every extraction manually before users complete the workflow, no model improvement will rescue the product. Platform engineering cannot substitute for interaction design or incentive alignment.

When Is the Signal Strong Enough to Act On?

The sensitivity test gives you a direction. It does not give you a green light. Platform investment should begin only when production evidence satisfies three conditions—all three, not a majority.

Retention Sensitivity Confirmed. The sensitivity test has shown that user retention responds to accuracy improvement, not exclusively to interface changes. This must be measured against real user behavior, not operator survey data.

Operational Feedback Volume. Production usage generates structured, correctable override data from daily operator interventions at sufficient volume to support evaluation curation. Without this, a platform has no learning signal to compound.

Engineering Capacity Constraint. Squad roadmaps are demonstrably blocked by the maintenance cost of probabilistic plumbing—not by customer discovery uncertainty or workflow redesign backlog. If squads are blocked by unclear product direction, a platform will not unblock them.

Why Thresholds Get Gamed

Thresholds create incentives. When platform funding, headcount, and engineering autonomy depend on whether these conditions are met, self-reported data becomes unreliable in two predictable directions:

• The autonomy-seeking squad will downplay maintenance friction and underreport operator overrides to avoid central intake queues.
• The maintenance-dumping squad will exaggerate rework hours and override counts to shift overhead onto a platform budget.

The fix is to measure through automated gateway telemetry rather than squad surveys:

• Feedback volume: Count automated diff counts between raw model outputs and final database commits, tracked at the API gateway layer—not from engineer estimates.
• Plumbing overhead: Audit Git commit histories for infrastructure boilerplate and vendor SDK migrations versus direct customer feature delivery.

Until all three conditions hold under objective telemetry, the AI feature belongs inside the product squad. Rewriting a squad's local prompt pipeline costs weeks. Maintaining an unvalidated enterprise platform abstraction across four product lines—and unwinding it when the abstraction proves wrong—costs quarters.

What This Test Does Not Tell You

The capability sensitivity test answers one question: is there a signal worth chasing? It does not tell you what to build, what to buy, or whether the economics support a shared platform at all.

Those are the next two questions—and they carry equal weight.

• Part 2 covers the architecture ownership decision: what to build, what to buy, and how to federate domain intelligence without turning the platform team into a bottleneck. → What to Build, Buy, and Federate in an Enterprise AI Platform
• Part 3 covers the financial case: how to model incremental platform value against the best realistic local alternative, using phased cash flows and real-options logic instead of point-estimate ROI. → Underwriting the Return on an AI Platform

One-Line Synthesis

Prove the constraint before funding the cure. A capability sensitivity test that confounds model accuracy with operator ergonomics and workflow incentives is not a diagnosis—it is an expensive assumption.

Related Reading

• What to Build, Buy, and Federate in an Enterprise AI Platform — Part 2: The architecture ownership decision and the three-tier federated model.
• Underwriting the Return on an AI Platform — Part 3: Phased cash flows, real-options logic, and the six failure modes.
• Policy-as-Code for Autonomous Agents — Implementing deterministic verification perimeters around probabilistic reasoning engines.
• The AI Pricing Paradox: Why Cost-Plus SaaS Models Collapse — How probabilistic compute costs destabilize traditional subscription pricing.

The ideas in this post are my own — they emerged from questions I asked while learning applied AI concepts and putting them to work in my job and my projects. The prose was developed with AI assistance.