Reliability theater
The AI reliability tools you're being sold mostly verify code. Your risk is in everything else the model produces.
Most of what vendors now sell as AI reliability is a better code-review control. Agents write code, tests run against it, a gate holds the merge until the checks pass, and the pitch calls the result reliable AI. That is real progress for software delivery. It does nothing to the error rate of the AI output your business runs on: the summaries, extractions, classifications, and recommendations that feed decisions.
Bruce Schneier called security measures that make people feel safer without making them safer "security theater."[1] The same pattern now shows up in AI. Reliability theater is a control that produces the look of reliability without changing the error rate. It survives because the look is easy to demonstrate and the error rate is hard to see.

Green-check reliability
The flagship example is the passing build. Code is the one kind of AI output that comes with cheap, automatic checks: compilers, type checkers, linters, and test suites. Run the checks, get a green badge, and the output looks verified.
A green badge only proves the code passed the tests it was given, whether a person wrote them or an AI agent did. Researchers check test quality with mutation testing: they plant small bugs in the code and count how many the tests catch. In one published case, tests written by a language model covered every line and every branch of the code under test, and caught 4% of the planted bugs.[2] By the usual dashboard measure that suite was perfect. By the measure that matters it missed nearly everything.
That was one example from one study, not an industry rate. It shows what a green check can hide when nobody checks the checks.
The same holds for the pipeline around the code. Deployment success rates, rollback times, and pipeline uptime measure how reliably you ship. They say nothing about whether what you shipped gives right answers. An AI feature can deploy cleanly a dozen times a day and still be wrong on one output in ten.
When the model writes the tests
The first piece in this pair, [Your AI doesn't know the rules](https://www.dave-nix.com/post/your-ai-doesn-t-know-the-rules), argued that a language model doesn't know the rules of your business. It has no model of what correct means, only patterns of what correct-sounding output looks like.
A test has two parts: the setup, and the assertion that states what the right answer should be. That assertion is the part that encodes a rule. A 2024 study of 24 Java codebases found that tests generated by language models mostly assert what the code does, not what it should do.[3] Give the model buggy code and it tends to write tests that confirm the bug.
So the agent writes the code, writes the tests, and grades its own homework. The tests pass because they were written to describe the code they test. The more of the loop you hand to the model, the less the green badge tells you.
None of this makes test-gated agent pipelines worthless. They catch regressions, enforce standards, and keep a human reviewer from wasting time on code that doesn't compile. They are a good control for one stage of software delivery. The problem is selling that control as the answer to AI reliability.
Where the exposure sits
Code is the easy case, because some of it can be checked by machine. Most AI output in production has no equivalent check. Nothing compiles a contract summary. No test suite can confirm that an extracted invoice total is right, that a risk classification is defensible, or that a recommendation to a customer is sound. For that work, checking an output costs about as much as producing it, and only a person who knows the domain can do it.
That is where the business exposure sits. These outputs flow into decisions that are expensive or impossible to reverse: a payment approved, a claim denied, a well plan adjusted, a term accepted. Consumer web software grew up shipping fast and patching fast because a bad release cost an afternoon. Most enterprise AI work doesn't have an afternoon-sized failure.
Even the most regulated industries mostly aren't measuring this. In a 2026 study of 44 banks, only one in five used tools to formally monitor whether the human review controls on their generative AI were working.[4] Banks have model risk functions, regulators, and audit committees. If they mostly don't measure whether review catches errors, it's a safe bet most other firms don't either.
Why it's sold this way
Nobody needs bad intent to explain reliability theater. The business model explains it.
A reseller earns on what it resells, and vendor partner programs reward volume. In that model, a stack of tools is the product, and a process change that needs fewer tools is a smaller sale. Code tooling also sells itself: it demos well, it produces charts that go up and to the right (pass rates, merge throughput, cycle time), and it fits a budget line the buyer already has. Measuring the error rate on business outputs has none of those properties. It has no tool category, its first result is usually bad news, and the fix is often a process change rather than a purchase.
So the code story gets told, and the gap gets papered over with it. The salesperson pitching the stack usually believes it. The seller's work is done once the contract is signed. The error rate shows up afterward, and the buyer is the one who lives with it.
The buyer has its own reason not to look. The executive who sponsored an AI rollout has the most to lose from measuring it. Benefits reviews, post-launch ROI checks, and audits of real errors in production all risk showing that the project delivered less than the business case promised, so they get scheduled late, scoped narrowly, or skipped. Theater needs an audience that would rather not look closely, and a sponsor with a reputation riding on the result is that audience. The fix is to make measurement something a sponsor can survive, and a later piece in this series covers how.
What to ask instead
Before you buy anything sold as AI reliability, ask two questions of every system that produces outputs your business acts on. What is the error rate on those outputs? How would you know if it changed?
If nobody can answer the first, no tool you add will answer it for you. If nobody can answer the second, the answer you have today expires the next time the vendor updates the model, someone edits a prompt, or the inputs drift. A reliable system has a measured error rate and a way to notice when it moves. Everything else is staging.
Observability is the next place the same pattern shows up, and a piece in two weeks takes it on.
If you're being sold a reliability stack for 2027, I can help you work out what it measures before you sign.
Notes
1. Bruce Schneier, *Beyond Fear: Thinking Sensibly About Security in an Uncertain World* (Copernicus Books, 2003).
2. Guancheng Wang, Qinghua Xu, Lionel Briand, and Kui Liu, "Mutation-Guided Unit Test Generation with a Large Language Model," arXiv:2506.02954 (2025). The case is subject id_81 from the HumanEval-Java benchmark: 100% line and branch coverage, 4% mutation score. https://arxiv.org/abs/2506.02954
3. Michael Konstantinou, Renzo Degiovanni, and Mike Papadakis, "Can we find bugs using LLM-generated oracles?", arXiv:2410.21136 (2024). First posted as "Do LLMs generate test oracles that capture the actual or the expected program behaviour?" https://arxiv.org/abs/2410.21136
4. Risk.net Risk Benchmarking, model risk management study of 44 banks (2026), as reported in "Few banks formally evaluate GenAI human-in-the-loop controls," WatersTechnology. https://www.waterstechnology.com/node/7953165
© 2026 David Nix. Licensed under CC BY-NC-ND 4.0: share and quote with attribution; no modifications, no commercial redistribution.
