Governing the Black Box You Didn't Build: AI Governance When You Don't Own the Model

Most AI governance writing contains an unstated assumption.

It assumes you built the model. That you have access to the training data, the weights, the architecture, and the evaluation logs. That when something goes wrong, you can open the system and investigate what happened inside it.

That assumption describes a small minority of AI deployments.

The majority of organizations deploying AI in production today are wrapping third-party APIs. They send a prompt to an endpoint they do not control and receive a response from a model they have never seen. The model was trained on data they did not select. It may have been updated since the last time they evaluated it. When its behavior changes — and it will change — they will find out not from the vendor, but from their own users or their own audit.

When you deploy AI you didn't build, the governance problem is not how to audit the model. It is how to govern the surface you actually control — and how to be honest about what that surface cannot tell you.

That is a different governance problem than the one most frameworks are designed to solve.

What You Actually Lose

To build a realistic governance program for vendor AI, you need a precise inventory of what model ownership gives you that API access does not.

Training data visibility

You do not know what data the model was trained on. You do not know if it contains information about your customers, your industry, or your jurisdiction that could produce systematically biased outputs. You cannot audit training data composition, check for data quality issues, or verify that protected characteristics were handled appropriately during training.

This is not a theoretical concern. Several high-profile language model deployments have produced outputs that reflected biases embedded in training data that the deploying organization had no visibility into and no ability to audit.

Model internals

You cannot inspect the model's weights, its attention patterns, its internal representations, or its learned associations. Post-hoc explainability tools like SHAP require access to the model to compute feature attributions. You have no feature attribution layer. When a decision is challenged, you can explain what you sent to the model and what came back. You cannot explain what the model computed in between.

Update awareness

Vendors update models. Sometimes with notice, sometimes without. Sometimes the update improves performance on the dimensions you tested. Sometimes it changes behavior on dimensions you did not test. You will not know a behavioral shift has occurred until it appears in your output logs — assuming your monitoring is sensitive enough to catch it.

A model you don't own is a dependency that can change without a pull request, without a changelog, and without triggering any test in your CI pipeline. It is the only production dependency most engineering teams accept without version-pinning.

Causal traceability

The previous post on the correlation problem argued that causal structure is the prerequisite for genuine governance — specifically, the ability to answer what would have happened under different conditions. That capability lives inside the model. Without model access, you cannot build causal traces at the model layer. Your audit trail records what went in and what came out. It does not record why the model computed what it computed.

What Governance You Can Still Build

Acknowledging these losses does not mean vendor AI is ungovernable. It means governance has to move up the stack — from the model layer to the interface layer. The interface layer has three tractable governance handles.

Handle 1: Input Controls

Everything you send to the model is something you control. That surface is larger than it appears.

Prompt versioning. A prompt is software. It defines the model's behavior for your use case more directly than most configuration. Yet in most organizations, prompts are stored in application code without version control, without a review process, and without regression testing. Changing a prompt can change model behavior as dramatically as changing a model — and that change bypasses every model risk management review in place.

Governing prompts means treating them like code: version-controlled, reviewed before deployment, and regression-tested against a fixed behavioral benchmark after each change.

Schema validation. Before sending user-provided data to a model, validate that it conforms to expected structure and type constraints. This reduces the attack surface for prompt injection — where malicious input attempts to override the system prompt — and ensures the model is operating on data that has been verified to be structurally sound.

PII and sensitive data controls. Define explicitly what categories of data are permitted to leave your environment and enter a vendor's infrastructure. Implement stripping or tokenization at the boundary. Log what was stripped so the record exists for audit purposes even if the data itself was not transmitted.

The input boundary is a compliance boundary. Everything you send to a vendor model is data you have decided to share with a third party. That decision should be deliberate, logged, and governed.

Handle 2: Output Controls

Everything the model returns passes through your system before it affects a user or a decision. That passage is your validation window.

Schema enforcement. Require that model outputs conform to a defined structure before being accepted by downstream systems. If the model returns a JSON object, validate the schema. If it returns a decision recommendation, validate that it is within the set of permitted recommendation values. Reject and log anything that falls outside the schema.

Semantic range validation. Beyond structural validation, apply domain-specific checks. In lending, a model that recommends an interest rate outside your policy bands should be rejected regardless of whether the JSON is valid. In healthcare, a clinical recommendation that references a medication not in your approved formulary should trigger a hold. These checks are deterministic and auditable in a way the model's internal reasoning is not.

Confidence and abstention thresholds. If the model exposes confidence scores or logit probabilities, define minimum thresholds below which the output is routed to human review rather than automated action. If the model does not expose confidence signals, treat the absence of that signal as a risk factor in itself — not all vendor APIs are equally transparent about model uncertainty.

The output boundary is where probabilistic model behavior meets deterministic system behavior. Every acceptance decision at this boundary is a governance decision. It should be logged, thresholded, and auditable.

Handle 3: Decision Controls

The third handle is the most important and the most frequently neglected. After an output is validated and accepted, something still has to decide what to do with it. That decision layer is entirely yours — and it is where the most consequential governance controls live.

Deterministic decision rules. Regardless of what the model recommends, the final action should pass through a deterministic rule layer that enforces hard constraints. In a lending context: the model may recommend approval, but if the applicant's debt-to-income ratio exceeds a regulatory threshold, the deterministic rule overrides. The rule is auditable, replayable, and explainable in a way the model's recommendation is not.

This is the central pattern of composite intelligence: the model provides signal, the rule layer provides control. Neither alone is sufficient. Together, they produce a decision that is both informed and defensible.

Human review routing. Define the conditions under which a model's output triggers mandatory human review rather than automated action. High-stakes decisions, low-confidence outputs, outputs near policy boundaries, and outputs involving sensitive populations or edge cases should all have defined escalation paths. Those paths should be documented, logged, and auditable.

Decision logging at the point of action. Log the full decision context at the moment the action is taken: the model output received, the rule checks applied, the human reviewer if applicable, the final action taken, and the timestamp. This is the audit trail you can actually defend — because it records everything at the layer you control, without depending on model internals you cannot access.

The Comparison Table

The right side of that table is not small. An organization that governs inputs rigorously, validates outputs structurally and semantically, enforces deterministic decision rules, routes high-risk cases to human review, and logs all of this at the decision point has built a meaningful governance program — without ever touching the model.

What it cannot do is audit what the model computed, explain the model's internal reasoning, or detect behavioral shifts before they appear in production outputs.

Governance Residual Risk

Every mature risk program distinguishes between risks that can be controlled and risks that must be accepted. Vendor AI introduces a category of the latter that most governance frameworks do not explicitly name.

Call it governance residual risk: the portion of AI deployment risk that remains after an organization has applied all available interface-layer controls. It is not a sign of a weak governance program. It is the honest acknowledgement of what interface-layer governance cannot reach.

The specific components of governance residual risk for vendor AI are:

• Silent behavioral drift: The vendor updates the model. The update changes behavior in a way your regression tests do not detect. The change affects your users before you identify it.
• Training data liability: The model produces an output that reflects a bias embedded in training data you never saw. The liability accrues to the deploying organization.
• Undisclosed capability changes: The vendor improves or restricts model capabilities in ways not reflected in the API documentation. Your system's assumptions about what the model can and cannot do become incorrect.
• Third-party incident propagation: The vendor experiences a security incident, a data breach, or a model quality failure. The effects propagate to every organization using the API.

Governance residual risk is not a gap to close. It is a condition to disclose — to leadership, to regulators, and to auditors. An organization that does not acknowledge governance residual risk has not eliminated it. It has merely hidden it.

The practical implication is that vendor AI deployments require a different kind of risk acceptance documentation than internally built systems. The risk acceptance should explicitly identify the components of governance residual risk, describe the mitigations applied at the interface layer, and document the monitoring in place to detect adverse outcomes as early as possible.

How This Maps to the Architecture of Proof

The Architecture of Proof framework was designed with the assumption that governance starts at the perimeter and works inward through layers. That structure turns out to be well-suited to vendor AI governance, because the outer layers apply regardless of model ownership.

Layer 0 (Deterministic Perimeter): Applies directly. The physics-first pre-inference validation — checking data structure, detecting anomalies, enforcing input constraints — operates before the model is called and is fully within the deploying organization's control.

Layer 1 (Composite Intelligence): Applies. The orchestration pattern — rules informing models informing humans — works at the interface layer. The model becomes one signal in the composite, not the sole decision authority.

Layer 2 (Composite Accountability): Applies with modification. Accountability contracts can be written for what the model is expected to produce and what the organization is responsible for doing with that output. The contract boundary is at the API surface: the vendor is accountable for the model's output within stated parameters; the deployer is accountable for everything downstream.

Layer 3 (Control Tiers): Applies fully. Autonomy levels — act, ask, stop — are defined at the decision layer, which the deploying organization controls entirely. The model's output informs which tier the decision falls into, but the tier assignment and the resulting action are governed by the deployer.

Layer 4 (Escalation Protocols): Applies fully. Escalation triggers — confidence thresholds, semantic range violations, policy boundary proximity — are all defined and enforced at the interface layer.

Layer 5 (Audit Trails): Applies partially. Interface-layer logging is achievable and essential. Model-layer traceability is not. The audit trail is real but bounded.

Layer 6 (Causal Diagnostics): Applies partially. Root cause analysis at the decision layer is achievable — you can trace exactly what input was sent, what output was received, what rule was applied, and what action was taken. Root cause analysis at the model layer is not available. "The model computed this for reasons we cannot inspect" is an honest conclusion in some investigations.

The Practical Test

For any vendor AI deployment, the governance readiness question is not "Do we have a governance program?" It is: "Can we answer the following six questions from our logs alone, without contacting the vendor?"

1. What input did we send to the model for decision #4721?
2. What output did we receive?
3. What rules were applied to that output?
4. What action did the system take, and when?
5. If a human was involved, who reviewed it and what did they decide?
6. Has the model's behavior on this class of input changed in the last 30 days?

Questions 1 through 5 are answerable with interface-layer logging. Question 6 requires a behavioral regression testing program — a fixed evaluation set run against the model on a scheduled cadence, with outputs compared over time.

If an organization cannot answer all six from its own records, its audit trail has gaps. Those gaps will matter when a decision is challenged.

The audit trail you can defend is not the most complete one possible. It is the one you can produce, from your own systems, without relying on a vendor to cooperate with your investigation.

What This Changes for Product and Risk Teams

For AI product managers, vendor AI governance shifts the design question from "How do we build a trustworthy model?" to "How do we build a trustworthy system around a model we cannot fully audit?"

That shift has four immediate practical implications.

Treat the vendor relationship as a risk counterparty. The vendor is not just a technology provider. They are a party whose decisions — about model updates, data handling, and incident response — directly affect your risk exposure. The contract should reflect that: behavioral regression rights, update notification requirements, data handling obligations, and incident disclosure timelines.

Invest in behavioral benchmarks before deployment. Before going live with a vendor model, establish a fixed evaluation set that captures the decisions the model will be making in your environment. Run it before launch, store the results, and re-run it after every vendor update and on a regular cadence. This is the detection mechanism for silent behavioral drift.

Design the decision layer as the governance layer. The model is not your governance boundary. The decision layer is. Deterministic rules, human review routing, and action logging at the point of decision are not supplementary controls — they are the primary governance mechanism for a system where model internals are not accessible.

Disclose governance residual risk explicitly. In risk documentation, in vendor contracts, and in regulatory filings where applicable, name the components of risk that remain after interface-layer controls are applied. An organization that does this is not admitting weakness. It is demonstrating governance maturity — the ability to distinguish between what is controlled and what is accepted.

The strongest governance programs for vendor AI are not the ones that claim to control everything. They are the ones that control the interface rigorously, test behavioral stability continuously, and document what they cannot control with precision.

One-Line Synthesis

When you don't own the model, governance moves to the only layer you control — the interface — and honest risk programs name what that layer cannot reach.