Before You Fund an AI Platform: Prove the Workflow First

Part 1 of 3: AI Platform Budget Decisions

Platform investment justified by a misdiagnosed constraint produces the same outcome as no investment at all. The only difference is the bill.

The Pattern That Keeps Repeating

A product organization runs four squads. Four squads build four independent prompt pipelines. Someone proposes centralizing them. The business case projects $600k in productivity savings. Leadership approves it.

Six months later, squads are bypassing the platform using discretionary vendor credits. The platform team has become an intake bottleneck. The CFO is asking why software expenditure rose while delivery velocity declined.

The platform did not fail because shared infrastructure lacks value. It failed because the product team answered the wrong question first.

The right first question is not "Should we build a platform?" It is "Do we actually have a capability problem?"

Is Your Value Ceiling a Capability Problem or a Workflow Problem?

Here is how the misdiagnosis typically happens. An invoice extraction feature stalls at 22% monthly active adoption. The squad blames the model: "Extraction accuracy is 84%. We need a shared fine-tuning pipeline and a retrieval layer."

That may be wrong. And committing platform capital before testing that assumption is how organizations build infrastructure that nobody uses.

Two Starting Points: Brownfield vs. Greenfield

Product managers face this decision at two completely different moments in the product lifecycle:

1. The Brownfield Diagnostic (Improving an Existing AI Feature): You already deployed a scrappy local prompt wrapper or MVP. Adoption has stalled (e.g., an invoice extraction feature sitting at 22% monthly active adoption). The squad blames model accuracy: "Extraction is only 84%. We need platform budget, fine-tuning, and a vector database to hit 95%." The test proves whether accuracy is genuinely the adoption blocker before you spend capital fixing it.
2. The Greenfield Feasibility Test (Simulating AI in a Manual Workflow): You have a manual or rule-based process with no AI in production. The proposal asks for $250k to build an automated intelligence pipeline. Before writing a line of model plumbing, you run a Wizard of Oz test: insert human operators behind the interface to simulate near-perfect automated extraction and instant turnaround. If users still don't adopt the workflow when the output is artificially flawless, automated AI will not rescue it either.

The Capability Sensitivity Test

Whether testing an existing feature or simulating a new one, the diagnostic mechanism is the same. Hold model capability constant (or simulate it). Insert a human operator to correct or generate the outputs. Watch what happens to user task completion and retention.

If adoption rises substantially with verified output, improved capability is a credible value driver. Worth investigating.

But the test is not a clean experiment. A human operator does far more than fix model output. Three confounding interventions enter simultaneously:

• Contextual repair: The operator adds missing domain context, infers customer intent, and fixes ambiguous document formatting that the model missed.
• Ergonomic compensation: The operator navigates around awkward review interfaces, clunky validation steps, and poorly presented evidence, compensating for UX debt the model never touched.
• Decision threshold modification: The operator changes the implicit risk tolerance required to commit an action. Users may be completing tasks because a human vouched for the output, not because the output improved.

High adoption after human correction demonstrates that task quality matters. It does not prove that model accuracy alone is the binding constraint.

Name the result accurately. Call it a capability sensitivity test (not a capability vs. workflow diagnostic) to avoid overstating what the experiment shows.

If operators must correct every extraction manually before users complete the workflow, no model improvement will rescue the product. Platform engineering cannot substitute for interaction design or incentive alignment.

When Is the Signal Strong Enough to Act On?

The sensitivity test gives you a direction. It does not give you a green light. Platform investment should begin only when production evidence satisfies three conditions: all three, not a majority.

Retention Sensitivity Confirmed. The sensitivity test has shown that user retention responds to accuracy improvement, not exclusively to interface changes. This must be measured against real user behavior, not operator survey data.

Operational Feedback Volume. Production usage generates structured, correctable override data from daily operator interventions at sufficient volume to support evaluation curation. Without this, a platform has no learning signal to compound.

Engineering Capacity Constraint. Squad roadmaps are demonstrably blocked by the maintenance cost of probabilistic plumbing, rather than customer discovery uncertainty or workflow redesign backlog. If squads are blocked by unclear product direction, a platform will not unblock them.

Why Thresholds Get Gamed

Thresholds create incentives. When platform funding, headcount, and engineering autonomy depend on whether these conditions are met, self-reported data becomes unreliable in two predictable directions:

• The autonomy-seeking squad will downplay maintenance friction and underreport operator overrides to avoid central intake queues.
• The maintenance-dumping squad will exaggerate rework hours and override counts to shift overhead onto a platform budget.

The fix is to measure through automated gateway telemetry rather than squad surveys:

• Feedback volume: Count automated diff counts between raw model outputs and final database commits, tracked at the API gateway layer, rather than from engineer estimates.
• Plumbing overhead: Audit Git commit histories for infrastructure boilerplate and vendor SDK migrations versus direct customer feature delivery.

Until all three conditions hold under objective telemetry, the AI feature belongs inside the product squad. Rewriting a squad's local prompt pipeline costs weeks. Maintaining an unvalidated enterprise platform abstraction across four product lines (and unwinding it when the abstraction proves wrong) costs quarters.

What This Test Does Not Tell You

The capability sensitivity test answers one question: is there a signal worth chasing? It does not tell you what to build, what to buy, or whether the economics support a shared platform at all.

Those are the next two questions, and they carry equal weight.

• Part 2 covers the architecture ownership decision: what to build, what to buy, and how to federate domain intelligence without turning the platform team into a bottleneck. → What to Build, Buy, and Federate in an Enterprise AI Platform
• Part 3 covers the financial case: how to model incremental platform value against the best realistic local alternative, using phased cash flows and real-options logic instead of point-estimate ROI. → Underwriting the Return on an AI Platform

The Adversarial Stress-Test: Five Critical Inquiries

1. What is the strongest argument against this approach?

Synthetic Wizard of Oz tests and manual operator patching introduce artificial human latency and interpersonal empathy that probabilistic models cannot replicate at runtime. If users respond positively to human-in-the-loop intervention, that success may reflect the warmth or adaptability of human problem-solving rather than a clean signal that improved raw model accuracy will unlock automated adoption.

The Practical Resolution: Strip interpersonal cues from the experiment by restricting human operators to asynchronous data patching. The operator interacts exclusively through the underlying database or backend API, modifying parsed JSON fields or candidate suggestions without altering UI copy, latency profiles, or user communication channels. If users experience identical interface mechanics with only the underlying output accuracy improved, adoption changes isolate capability sensitivity rather than human empathy.

2. Under what circumstances would this fail?

The capability sensitivity test fails in high-throughput or real-time workflows (such as high-frequency fraud detection or interactive voice agents) where human operators cannot physically patch errors at operational speed, making manual error simulation technically unfeasible.

The Practical Resolution: Replace live human patching with pre-computed golden synthetic injections in real-time or streaming pipelines. In fraud detection or sub-second voice interactions, teams replay historical production requests where authoritative ground truth was verified out-of-band. Simulating high-accuracy model responses from pre-cached canonical answers under real production latency budgets proves whether higher precision unlocks operational throughput before building bespoke real-time fine-tuning infrastructure.

3. What would a skeptical customer, client, or colleague object to?

Ambitious engineering teams and platform leads will object that running manual sensitivity tests across multiple product squads creates artificial delays in infrastructure delivery, arguing that modern foundation models advance so quickly that building a unified platform ahead of demand is necessary to capture scale advantages.

The Practical Resolution: Enforce strict time-boxing on sensitivity tests (maximum two sprints per candidate feature) and provide thin off-the-shelf routing templates. Fast-tracking manual capability audits prevents infrastructure gridlock while ensuring that platform squads only build shared capabilities for workflows with proven demand elasticity. Framing the test as risk mitigation protects engineering leads from building expensive centralized services that squads ultimately abandon.

4. Which alternative deserves more consideration?

Rapid canary rollout of frontier models using off-the-shelf third-party API gateways without upfront bespoke platform architecture. Testing improved accuracy directly by hot-swapping leading frontier endpoints provides immediate behavioral telemetry without paying the overhead of simulated human workflows.

The Practical Resolution: Treat frontier model canary swapping as an initial filter within the sensitivity framework rather than a replacement. Upgrading to a frontier model via a lightweight proxy tests whether off-the-shelf generalist capability moves the adoption needle at near-zero development cost. If an expensive frontier model still fails to lift user adoption, it conclusively proves the bottleneck is interaction design, workflow friction, or incentive misalignment rather than platform infrastructure.

5. What am I assuming that may not be true?

This approach assumes that product squads and platform engineering have cleanly aligned incentives to accurately diagnose failure modes, when in reality squads often seek autonomy from centralized queues while platform architects are incentivized to justify centralized headcount and capital allocation.

The Practical Resolution: Institute objective, automated telemetry thresholds managed by an independent architecture review committee. Measure operational override volume through gateway-level diff counters (comparing raw model generation against final committed database state) and audit engineering commit histories directly. Decoupling funding decisions from subjective squad self-reporting eliminates political gaming between autonomy-seeking product teams and expansion-minded platform groups.

One-Line Synthesis

Platform capital must follow empirical constraint isolation: validating whether product adoption is bound by model accuracy requires testing workflow ergonomics and user incentives before committing shared engineering budget.

Related Reading

• What to Build, Buy, and Federate in an Enterprise AI Platform: Part 2: The architecture ownership decision and the three-tier federated model.
• Underwriting the Return on an AI Platform: Part 3: Phased cash flows, real-options logic, and the six failure modes.
• Policy-as-Code for Autonomous Agents: Implementing deterministic verification perimeters around probabilistic reasoning engines.
• The AI Pricing Paradox: Why Cost-Plus SaaS Models Collapse: How probabilistic compute costs destabilize traditional subscription pricing.