top of page

Your AI doesn't know the rules

Oct 16
3 min read

AI mistakes are built into the model.



Every large language model will make mistakes that no amount of tuning removes, because it has no model of the rules it works within. So the leadership decision is which work to give it. Where the rules are known, put them in software that can't break them. Where they aren't, pay for review, and test the reviewers.

The model has never seen the rulebook

Cory Doctorow calls today's AI "theory-free."[1] Older software started from an explicit model of the problem. A 1950s chess program knew there was a board, which pieces stood where, and which moves were legal. A large language model starts from none of that. It learns statistical patterns from an enormous number of examples and produces the likeliest next move.

Most of the time the likeliest move is a good one. Sometimes the model moves a piece onto a square its own piece already holds. The 1950s program, running on a sliver of today's computing power, never made that move.[1] The model has seen millions of games and no rulebook.

That's why the mistakes don't go away

A system with no model of its subject can't tell a plausible answer from a correct one. Doctorow argues the errors are intrinsic to how these systems work, not bugs waiting for a fix.[1] Better models and better prompts change how often it happens. They don't change why.

Courts keep the clearest record. Legal researcher Damien Charlotin tracks court decisions that address AI-fabricated content in filings: 2,149 worldwide as of October 5, 2026, and 854 of them involve lawyers.[2] Lawyers are trained to check citations. Fluent, confident output gets past them anyway.

Where the rules are known, write them down

Plenty of work now routed to AI follows rules a company already has: who qualifies for a discount, which invoices need a second approval, which claims meet a policy. A rules engine applies those rules the same way every time. It can't make the illegal move, each decision can be traced line by line, and it costs a fraction of a model call to run.

The five questions in The Price Is Not the Price test whether a model is the right tool at all. A workload that follows written rules is the easiest one to call.

Ask: Which of our AI workloads apply rules we already have written down?

Where they aren't, check the output and the checkers

Some work has no rulebook: drafting, summarizing, sorting messy requests. That is where a model earns its cost, and where every output that matters needs a person to check it before it acts.

Review fails in its own way. Doctorow points out that experts who check output quickly start approving it on reflex.[1] A reviewer who approves everything is no reviewer at all, and the approval rate won't show it. The test is to plant known-bad outputs in the queue and count how many the reviewers catch.

I tell clients that a control works only if following it is easier than skipping it. A planted error makes skipping visible, for a few minutes of setup a week.

Ask: When we plant an error, how often do our reviewers catch it?

If some of your AI workloads are guessing at rules you've already written down, I can help you find them and move them into software that can't break them.

Notes

  1. Cory Doctorow, "Swapping money for expertise," Pluralistic, October 6, 2026. https://pluralistic.net/2026/10/06/nonfungible/

  2. Damien Charlotin, "AI Hallucination Cases" database, last updated October 5, 2026. Counts legal decisions in which a court or tribunal found, or implied, that a party relied on hallucinated content; mere allegations are generally excluded. The lawyer count is the database's party-type filter across all years. https://www.damiencharlotin.com/hallucinations/



© 2026 David Nix. Licensed under CC BY-NC-ND 4.0: share and quote with attribution; no modifications, no commercial redistribution.

Recent Posts

See All

© 2026 David Nix.

bottom of page