Sunsetting AI: The Deprecation Playbook for Probabilistic Systems

Turning off an AI feature is harder than shipping it, and almost nobody plans for it.

Launching earns a press release. Retiring earns an awkward silence, so unprofitable tools stay in production while users work around them.

Every product team knows how to retire regular software. You check the traffic, send the deprecation notice, give customers a migration window, and delete the code. With an AI feature, the endpoint is the easy part. Customers have built habits around the drafts. The model vendor may retire your model on its own timeline. And your staff may no longer be able to do the work at the old manual pace.

Four problems show up again and again:

A retirement plan has to cover the people, the contracts, and the paper trail. Deleting the code is the smallest part of the job.

NIST's AI Risk Management Framework (NIST AI 100-1, function MANAGE 4.1) lists decommissioning, post-deployment monitoring, and change management as things organizations are expected to plan. This playbook turns that expectation into decisions a product leader can actually make.

How Do You Know It's Time to Turn It Off?

Most retirements start too late. Teams wait for a crisis, such as a vendor deadline or a budget cut, and then have two weeks to do what needed two quarters.

Watch a short list of signals, and look at the trend over two quarters instead of reacting to one bad month.

One signal is a prompt to investigate. Three moving the wrong way for two quarters is a prompt to run the decision below.

Decide First: Repair, Constrain, Replace, or Retire

A shutdown plan is not a product decision. Before you spend a sprint dismantling a feature, check whether it can deliver value in a smaller or cheaper form.

To keep this concrete, take a hypothetical product. A B2B software company shipped an assistant that pulls indemnity terms, governing law, and liability caps out of uploaded contracts. Eighteen months in, enterprise customers say it misses clauses in non-standard amendments, and the model bills for the heaviest accounts now exceed what those accounts pay.

These are not mutually exclusive. Many teams repair and constrain at the same time.

In the contract example, repair could mean fixing how the documents are split up before the model reads them, so liability caps stay attached to the definitions they depend on. Constrain could mean limiting the tool to standard agreements and sending unusual amendments straight to a lawyer, with a human approving every extraction before it is saved. This is the idea behind lowering a feature's autonomy so it can do less damage when it is wrong. Replace could mean a smaller model trained on the company's own contracts, or plain rules for the fields that never change. A smaller model brings its own hosting and security work, and rules miss anything phrased in an unexpected way, so both need testing on real documents before anyone counts the savings.

Retiring Does Not Have to Be a Switch

The gentlest form of retirement is grandfathering. You stop selling the feature to new customers, freeze changes, and let existing accounts move off at renewal. Costs stop growing right away, and nobody gets an abrupt notice. If the feature is cheap to keep and expensive to rebuild, maintenance mode is often a better answer than deletion.

Does the Math Work?

Run two checks, in this order.

First, check whether you are allowed. If a customer contract promises the feature through year-end, you can't turn it off until that is renegotiated. A good business case does not override a signed commitment.

Second, check whether it pays. Over a 12-month window, compare:

$$\text{Net Retirement Value} = \text{Savings You Will Actually Realize} - \text{Transition Costs} - \text{Replacement Costs} - \text{Expected Revenue Loss}$$

Savings you will actually realize means money that really stops leaving the building. Server costs that just move to another project don't count.

In the contract example, the team found \$140,000 a year in model and hosting savings. Moving customers over cost \$45,000. A partner to handle manual reviews cost \$35,000 a year. Expected renewal concessions were \$20,000. The base case came to \$40,000 in year one. These are made-up numbers, but they show the shape of the exercise.

Treat that \$40,000 as one scenario of three. Build a cautious case and an optimistic case too, and see whether the answer survives. Give each estimate to the person closest to it: Finance owns the cost savings, Customer Success owns the churn estimate, Engineering owns the transition cost. Be honest about two biases. Teams that built the feature tend to overstate how much users will miss it, and the same teams tend to understate how hard it will be to let go of it.

Match the Process to the Risk

Not every feature needs a review board. Pick a track before you plan anything else.

A rule is needed for who can move a feature into the lightweight track, or engineers will pick it every time. Make it the head of product or the engineering lead, and only for features with no outside callers.

Smaller teams can fold the roles together. In a 30-person company, the lead engineer covers the technical work, the head of product or a founder covers the decision and the customers, and operations plus outside counsel cover records and risk.

What Does a Product Leader Actually Own?

The full track has five gates. You don't run the technical work yourself, but you decide when each gate is passed.

graph LR
    G1[1. Decide] --> G2[2. Find who depends on it]
    G2 --> G3[3. Move people over]
    G3 --> G4[4. Keep the records]
    G4 --> G5[5. Close it out]
    subgraph Running from day one
        E[Save logs, prompt versions, and review records]
    end

Several of these gates have a trap in them.

Gate 1: Start Saving Evidence Now

The moment you decide to retire, ask engineering to freeze deletion of logs, saved prompts, and review records. Once servers are shut down or databases cleaned up, that evidence is gone. This is the single step teams most often get wrong, because it feels like something to handle later.

Gate 2: Don't Trust a Single Log Search

A three-month look at traffic logs will miss the finance team's year-end script. Combine the logs with the service catalog, the list of connected tools and agents, customer contracts, and plain conversations with account managers. Anything you can't confirm goes on a written list of open risks, with a named executive attached.

Gate 3: Test the Backup With Real People

A rule-based fallback can be tested in the background against live traffic. A human review queue cannot. You have to staff it, send it real work, and time how long it takes. Prove that the backup meets the targets you set before you cut the old feature off.

Decide up front whether you can turn the old feature back on if the backup struggles. If you are retiring for cost or cleanup reasons, keep a restore path for two to four weeks. If you are retiring for security, privacy, or customer harm, do not turn it back on under any circumstances. And if the vendor has already pulled the model, restoring it is not an option at all.

Gate 4: Keep the Story, Not a Replay

You can't promise to re-run a past decision and get the same answer. Hosted vendors change things behind the same model name, and even the PyTorch documentation says identical results aren't guaranteed across versions or hardware. What you can keep is the record of what happened, which is the idea behind decision traceability:

• the customer's input and when it arrived
• the prompt version and instructions in use
• the source text the model was shown
• what the model produced
• what the human reviewer did with it

Keeping all of that conflicts with privacy rules that say to delete personal data. Legal has to settle which records are needed and for how long. For high-risk systems, the EU AI Act (Article 26(6)) tells deployers to keep the logs they control for at least six months. Your use case may need longer, or may not fall under it at all.

Many older features were built with no such logging. If yours is one, write down what is missing, list the other records that might fill the gap (customer emails, exported documents, billing entries), and have legal sign a short memo accepting the remaining risk. Do not claim you can rebuild what was never saved.

Gate 5: Close the Loop

Finish the technical shutdown: revoke the feature's keys and credentials, remove it from tool and agent registries, take down the buttons and publish the notice, stop the servers and indexes so billing ends, and confirm that outside calls to the model have dropped to zero. Remove the retired feature's tests from the build pipeline, but keep company-wide safety checks such as personal-data filters. Save the test sets you built, because they will help you judge the next model.

Closing out the engineering work is not the same as finishing the program. Check back at 30 and 90 days to see whether the savings showed up, whether customers are coping, and whether the backup is holding.

Telling Customers

Enterprise customers will say what you would say: you sold me automation, and now my team is doing the work by hand while I pay the same price.

Lead with what happens to them, not with your cost problem. "Our inference bill is too high" is your problem, and no customer will find it persuasive.

Count a customer as migrated when their task works on the new setup, not when they acknowledge your email.

The People Side

The hardest part is rarely technical. After a year of using an automated tool, a team has changed.

If an assistant let one analyst handle twenty contracts a day instead of eight, switching it off makes a backlog until you add people. Reviewers who got used to skimming machine summaries read less carefully. Experienced staff lose some of their speed on messy documents. Newer hires may never have done the job by hand at all, and they miss the edge cases that the old hands would catch. These are four different problems and they need different fixes.

Put the cost of the manual work into the retirement budget. And don't wait for cutover day to find out what your team can do on its own. Run practice batches beforehand, with real work done by hand under a senior reviewer, so you know the true speed and error rate before the tool goes dark.

Selling the Decision Internally

A retirement fails more often in the meeting room than in the codebase.

Make the Next One Easier

The cheapest retirement is the one planned at launch. Most of the pain in this playbook, from the missing logs to the hunt for hidden callers, comes from features that shipped with no exit.

Before launch, route every model call through one internal gateway so swapping or removing a vendor happens in a single place. Log the full record of each decision from the first day. Add a line to enterprise contracts that reserves the right to change or retire the underlying model when the vendor does, with a sensible notice period. And write kill triggers into the original product spec, for example a minimum task completion rate and a maximum cost per task, with an automatic review if the feature misses either for two quarters in a row.

The Scorecard

Hold the program to a short list of checks.

Targets such as the length of the quiet window or the size of the test sample depend on the feature. Set them by risk, write them down, and don't copy a number from a template.

The Adversarial Stress-Test: Five Critical Inquiries

1. What is the strongest argument against this approach?

The process may cost more than the risk it protects against. Most AI features being retired are a summary button or a search assistant, not a lending model. A five-gate plan with records checks and sign-offs could push teams to skip the playbook and do nothing structured at all.

The answer is to use the tracks honestly. The lightweight track exists so that low-stakes tools can go in two days. The full track is for features where a surprise breakage or a missing record costs real money or creates legal exposure.

A second objection is that none of this is unique to AI. Reviewing a junior employee's work has the same asymmetry, and vendors have always retired products. That is fair. The difference is one of degree. AI output varies more, review costs more per item, and the vendor controls more of the stack.

2. Under what circumstances would this fail?

Four situations break it. A vendor shuts a model down with a deadline too short for any of this, which pushes you into the emergency track by default. Customers or internal teams have wired the output into spreadsheets or automations that never touched your logs and run once a year. The feature was built with no logging, so there is nothing to archive. And the team that understands the system has already been reassigned, so nobody owns the work.

3. What would a skeptical customer, colleague, or finance partner object to?

A customer will ask how they can trust your next AI feature when this one is being pulled. Answer with commitments they can hold you to: minimum support periods and written notice windows in future contracts.

A finance partner will say the churn forecast is a guess and easy to bend. They are right, which is why the three scenarios and the 90-day check on real invoices exist.

A legal colleague will say a reconstructed record is a story, not proof. That is partly true. It supports an audit and a defense, but it is not the same as a replay, and nobody should claim otherwise.

4. Which alternative deserves more consideration?

Handing the capability to the customer. For a few large accounts that depend heavily on the feature, giving them the prompts, the test set, and a supported way to run it themselves can turn a tense shutdown into a clean handoff. It only works for customers with the technical team to take it on.

5. What am I assuming that may not be true?

That a decision is made calmly and in advance. Plenty of retirements are forced by a vendor date, a budget cut, or a reorg, and the playbook then runs in compressed form.

That there is an acceptable fallback. For open-ended work, like reading messy medical records, no set of rules does the job, and there may not be enough qualified people to do it by hand. In those cases retiring the feature means leaving that line of work.

That the savings are real. Freed-up servers and staff often get absorbed elsewhere without showing up as savings.

That regulators and courts will accept a reconstructed record. The law on AI decision records is still moving, so what holds up today may need to change.

One-Line Synthesis

Plan the exit before you launch, because if you don't, your vendor, your costs, and your customers will eventually make the decision for you.