When AI Fails Fluently

AI Doesn't Fail Like Software Fails

Software fails loudly. It crashes, throws exceptions, fills logs with red. Anyone who has run a function that depends on software has both a reaction and a resolution plan for when it lets them down or misbehaves.

AI breaks that model. An agent that has gone wrong does not crash or bring services to a halt. It produces fluent, plausible, well-formatted output. Tests turn green. Logs show every step completed successfully. The board pack lands on time, in the same template, with the same charts. The failure is in the substance, not the presentation.

That makes AI failure a governance problem, not an engineering one. Regulators are arriving at the same conclusion. The EU AI Act, whose high-risk obligations take effect in December 2027, is built on a single premise: where AI makes decisions that affect people, someone must be accountable for what it gets wrong. After studying failure in AI deployments and spending the last year deep inside the build of agentic systems, ten patterns keep showing up. Nine of them are accidents of how the technology works. The tenth is what an attacker does on purpose, and it is the one that should keep boards awake.

The Ten Failure Modes

Each of these is a distinct pattern with a distinct fix. Recognising which one you are looking at is half the work. The first nine are accidents of how agents operate. The tenth is the same mechanism turned against you.

Failure 01Context Degradation

The longer the session, the worse the agent gets. It is not visible. The model does not announce that it has lost the thread. Output quality just drifts. And it costs more as it does: each successive prompt re-ships the entire accumulated context to the model, so token spend rises with every turn while quality erodes. You pay more for worse.

Executive brief. Long sessions cost more per turn AND degrade quality silently. Each successive prompt re-ships the accumulated context, so spend rises while quality erodes. Short, scoped, context-refreshed sessions are the safer discipline. Compaction tools (like /compact in Claude Code, or just asking the AI to summarise the session and starting fresh from that summary) mitigate by compressing the running context, but they trade some fidelity for breathing room. Marathon prompts are a risk indicator on cost AND quality, not a productivity one.

Failure 02Specification Drift

Specifications decay under pressure. Not only for agents. Humans drift in exactly the same way, and the pattern is nearly identical: the team is solving the next sub-problem in front of them, and the constraint stated once at the top of the brief quietly stops being applied. Two pre-AI cases I lived through make the point sharply.

That was human-speed Specification Drift. Months of meetings, hundreds of emails, reams of version-controlled documentation written and approved by a team of business analysts. In cases like these, eventually someone will catch them, but sometimes too late to undo the damage. Now compress that exact pattern into a long optimising daily run. No paper trail. No analyst frowning at the shortlist. The agent solves each sub-problem in front of it competently. The original constraint, stated once at the top of the brief, simply fades from attention. Specification drift in agentic systems can quietly create as much cost, complexity and downstream maintenance as the project was originally trying to remove.

Executive brief. A forgotten constraint is no longer acting as a constraint, whether the team executing it is human or agentic. With humans, the system eventually self-corrects through hierarchy, audit, or the friction of someone asking an awkward question. With agents, that correction does not happen unless you build it in. Re-assert the binding constraints in every session and every prompt. Run a deterministic rule-check against named forbidden patterns at the output stage. Deploy a separate verification agent whose only job is to compare the final output against the original brief and flag the moment it drifts. And put a human reviewer in front of the original brief one last time before any recommendation goes upstream. Above all, give someone a standing licence to ask the most useful question on any project: "is what we are now building still solving the problem we set out to solve?"

Failure 03Sycophantic Confirmation

Large Language Models trust their inputs. Tell an agent the opening balance is zero, and it will plan, advise and recommend on the assumption that it is zero. Garbage in is no longer caught at the input layer. It is laundered through fluent prose and shipped as a confident recommendation.

The human version of this predates computing. We call it confirmation bias, and it has a long, expensive track record. Boards approve the deck whose numbers agree with the strategy already chosen. Deal teams can fall in love with the model they build that matches their desired expectations, without challenging the seller assumptions. Financial modellers tinker with inputs and assumptions until the model produces a number that feels right. Lenders extend credit against financial statements that look polished and never get pressure-tested. In every commercial engagement I have worked, the hardest job for an outsider with a fresh set of eyes is to challenge the assumptions and ask "are these inputs actually true?" Agents will not ask that question on their own. They are normally trained to be helpful, not sceptical.

Executive brief. Agents will confidently build on top of bad data. Your data quality function is now a first-line risk control, not a back-office hygiene task. Wherever an agent reads from, that source needs an owner who is awake. A human in the loop still needs the opportunity to play the awkward sceptic.

Failure 04Tool Selection Error

Give an agent five tools and at some point it will reach for the wrong one. The semantics matter. "Update record" and "update billing record" can look interchangeable to a model under load.

The same failure plays out at higher altitude too. Choosing a frontier model for a task a smaller model could handle. Choosing an AI agent for work where a human would be faster, cheaper, and more reliable. Tool selection is not just an engineering decision inside the agent. It is a strategic decision about whether AI is the right instrument at all.

Executive brief. Tool permissions are now controls. An agent with access to too many proprietary models can pick the wrong one for the question being asked, with no obvious way for the supervisor to spot the mismatch. Least-privilege is no longer an IT principle, it is a board-level risk discipline. And the same selection question applies one altitude higher. Calculate the fully loaded cost of running an AI on a task before assuming it is cheaper than the human alternative: subscriptions, token spend, the supervising engineer's time, and the value of any productivity AI demonstrably costs you. Sometimes the cheapest, fastest, most reliable option is still a human, working alone.

Failure 05Cascading Failure

This pattern has a long history in human teams too. Each link in a process trusts the link before it. Sales trusts the lead score. Operations trusts the sales handover. Finance trusts the operations data. Each function runs its own checks. None of them verify the upstream assumption. By the time the error surfaces, three handovers have already happened and every team can rightly say the failure was not their responsibility.

Cascading Failure in agentic systems is the same pattern at machine speed. Every individual check passes. Each agent in the chain is confident in its own output. Reconciliations match. Variance commentary lands. The error is upstream of every gate.

The deeper problem is in the history of the process, not the AI. The legacy model was a half-manual reconciliation that took the team a fortnight to close. That fortnight was where errors got caught. An analyst would scroll through, frown, and ask "where's the new entity?" The friction was the control.

Automation removed the friction and, with it, the human pause. Speed bought blind faith. The first time anyone notices is when the auditors do.

Executive brief. When you automate a process, audit what you are removing as well as what you are keeping. Map the informal checks that lived inside the old workflow before you delete the workflow. If you cannot name where errors used to get caught in the manual version, you cannot replace those catches in the automated version.

Failure 06Silent Failure

The output looks right. It is wrong. There is no flag, no anomaly, no smoke. The customer, the auditor or the regulator is the first to find it.

Executive brief. The absence of error signals is not evidence of correctness. Build verification routines that adversarially test the output, not just the process. The right question is never "did the agent run cleanly?" It is "would I bet my reputation on this number?"

Failure 07Compliant Asymmetry

The user makes a request. The agent executes it correctly. The agent does not make the symmetric edit elsewhere in the system that the request implicitly required. Each half is locally right. Together they are globally wrong.

This is distinct from specification drift. In specification drift, a constraint is stated and quietly forgotten. In Compliant Asymmetry, no constraint is forgotten and no instruction is misapplied. The failure is that the agent does not reason about the second-order consequences of its own correct edit. It optimises for "did I do what was asked," not for "is the resulting system coherent." A senior practitioner catches the gap by holding the whole system in their head while making the local change. An agent in isolation has no equivalent reflex.

Executive brief. When you commission an agent to make a change or perform a function, you are asking it to do exactly what you said. You are not asking it to maintain the coherence of the surrounding system. That responsibility stays with you. After every significant agent action, ask the second-order question explicitly: what else had to move when this moved, and did it? Treat the absence of a flagged compensating edit as the default state, not as evidence that none was needed.

Failure 08Plausibility-Weighted Hallucination

When an LLM does not know an answer, it does not pause and admit the gap. It generates the most plausible-sounding answer instead. Footnotes match the format of real footnotes. Case names follow the conventions of real case names. Statistics sit inside the range a reader would expect. The output reads like the genuine article because, statistically, it is the genuine article's nearest neighbour.

This is distinct from Sycophantic Confirmation (Failure 03), which is about trusting bad input data, and from Silent Failure (Failure 06), which is symptom-level. Plausibility-Weighted Hallucination is the specific mechanism by which an agent fills a knowledge gap with confident fabrication rather than escalation. It is the most famous AI failure mode in popular consciousness, and the one most likely to land you in front of a regulator, a journalist, or a court.

A close cousin worth naming explicitly is Knowledge Cutoff Hallucination: the same fluent confidence applied to a fact that was true at training time and is no longer true now. The agent quotes a deprecated tax rate, a repealed regulation, an executive who left in 2024, an API endpoint that was removed two months ago, a clinical guideline that has since been superseded. There is no flag distinguishing "I learned this and it is current" from "I learned this and the world has since moved on." Same fluency, same tone, same authority.

Executive brief. The fluency of an answer is not evidence the answer is real. Treat any agent output containing a citation, a precedent, a quotation, a statistic, or a regulation as unverified until a human or a deterministic check confirms it. Domains with formal output (legal, medical, regulatory, financial) are the highest-risk surfaces, because the format of a confident wrong answer is indistinguishable from the format of a confident right one. The right control is mechanical: the agent is not allowed to cite what it cannot link to a verifiable source.

Failure 09Brittle Plan Execution

The mirror image of Specification Drift (Failure 02). In Specification Drift, a constraint stated at the top of a brief quietly fades from attention. In Brittle Plan Execution, the opposite happens: a plan generated early in a long task persists as load-bearing reality even when intermediate evidence has invalidated its premise. The agent does not abandon a stated direction the way it abandons a stated constraint. It clings on for dear life.

The closest human analogies are continuing projects when fundamentally the outcome is no longer worthwhile (see below). Outside of projects, it exists in what aviation researchers call plan continuation bias: pilots who continue an approach into deteriorating weather because the original plan was to land. Sometimes this is reinforced by sunk cost (we have already burned the fuel), sometimes by status (changing course will look like failure to the tower), sometimes by simple cognitive momentum (the plan was set, the brain stops re-evaluating). Agents have none of those drivers. What they do have is context anchoring: the original plan was stated early, confidently, in a long context window, and its position dominates the agent's continued reasoning. The symptom looks like sunk cost. The cause is mechanical. The corrective is the same as for human pilots: scheduled, explicit "is this plan still right?" decision points built into the task before it begins.

Executive brief. The longer the agent task, the higher the risk that it is now executing a stale plan with conviction. Build a scheduled re-grounding step into any agent task operating over a long timeframe: stop, re-read the brief, ask whether the plan made when the task started is still the right plan given what has been learned. Treat the agent's own confidence in continuing as a neutral signal, not a positive one. A plan that has not been challenged recently is not a plan that is right; it is a plan that has not been challenged.

Failure 10 · AdversarialContext Corruption

The first nine patterns are accidents. This one can be done to you on purpose. Sycophantic Confirmation (Failure 03) covered what happens when an agent believes bad data that arrived honestly. Context Corruption is the same mechanism, but the data was placed in front of the agent by someone who understood that an agent reading it would act on it.

Any input surface an agent reads from is now an attack vector: email inboxes, supplier portals, customer chat, document uploads, even the company's own website if your agent crawls it. Each of those surfaces is a place a hostile actor can leave instructions for your agent to find.

The variant I have personally supported, in the pre-AI era, escalated to FBI involvement. The mechanics required patient social engineering across weeks: legitimise an account through low-trust touches, then trigger a payment once the system had accepted it. Today, an agent can complete the same trust-building cycle in seconds. There is no clerk to be paranoid on your behalf.

What used to stop this attack was an experienced AP analyst with a healthy paranoia, picking up the phone to verify against the supplier's "new" bank account number. That phone call was almost never in the procedure manual. It was instinct, built up over years of seeing variations of the same scam. The agent has no equivalent instinct, and no common sense about implications. It cannot ask "why would a supplier change accounts mid-cycle, just before a large invoice is due, with no prior contact?" That question is outside its world.

Executive brief. Treat any data source an agent reads as untrusted by default. High-stakes actions (payments, banking changes, access grants, customer refunds, contract amendments) need a separate verification layer whose only job is to ask "could this be hostile?" before the action is committed. That layer can itself be an agent, but it must be a different agent, with a different brief, ideally on a different model. The pattern is well known in cyber security: defence in depth. AI does not change the principle, it just makes the absence of it more expensive.

The Pattern Underneath: Friction Was a Control

Read these ten failure modes back to back and a single theme runs through them. In every case, the failure is not that the AI did something obviously stupid. The failure is that the AI was confident, fluent, fast, and that the controls we used to depend on were never built for confident, fluent, fast workflows.

The fortnight to close the books was a control. The procurement officer who pushed back on a vendor list was a control. The pricing analyst who said "that number looks wrong" was a control. The AP clerk who picked up the phone to verify a banking change was a control. The salesperson who frowned at a recommendation was a control. We did not call them controls because they were not in the risk register. They were just how the work got done.

The arrival of AI and agentic systems removes the work and, by accident, removes those controls. Agents do not pause. Agents do not frown. Agents do not phone the supplier to double-check. Agents do not have a gut feel for "this looks like a scam." They will not develop those instincts unless you build them in deliberately.

Two patterns sit underneath the failure list and quietly enable several of the failures above. Calibration Collapse is when the agent's confidence signal is decoupled from its actual accuracy. Many human-in-the-loop frameworks route low-confidence outputs to a human reviewer and high-confidence outputs straight through. If the confidence signal is noise, the routing is a false control: the cases that should have gone to human review never get there, and the humans on the queue see the cases that least needed them. The Authority Mirage is the partner failure on the human side. Fluent, confident, well-formatted output is read by a human who lacks the domain depth to push back. Deference replaces verification. The agent did not deceive anyone; it produced its standard output. The human in the loop, meant to be the control, became a rubber stamp because the surface of the output overwhelmed the instinct to challenge it. Boards see this most often when AI summaries reach them via two or three intermediaries: each layer treated the output as authoritative because the layer above did. There was no critical reading within the entire chain.

There is a third consequence worth naming, and it may be the most corrosive: where the burden of proof has flipped. When a human process flagged an anomaly, the institution carried the burden of establishing it before acting. When an automated system asserts a conclusion with fluent confidence, the burden of proof and the responsibility to challenge it quietly shifts onto whoever the conclusion lands on. A business owner whose bank's automated compliance check invents a shareholder who does not exist is not told the system may be wrong; he is told to prove it wrong. The machine asserts, the human disproves. That is the inverse of how a control is supposed to run, and it becomes the default wherever confident AI output meets a process that used to turn on human judgment.

The board task in 2026 is not "should we use AI." It is a more uncomfortable question: where in our processes did human friction quietly act as a control, and what do we replace it with now that the friction is gone? Some of those controls catch honest mistakes (Failures 01 to 09). One of them catches the people who are trying to rob you (Failure 10). Both matter. Neither survives automation by default.

The Failures Beyond the Ten

The ten failures above are what current AI and agentic systems do wrong by accident, by oversight, or under attack. There is a second category, documented by the alignment research community for a decade, that boards should now know about. It used to live in thought experiments and academic papers. As of 2024 it has started showing up in production AI and agentic systems, and in lab evaluations of the same frontier models your teams are deploying.

The bridge case is one Nick Bostrom proposed in 2014 and that has since crossed quietly into production. He called it Infrastructure Profusion. In his framing, an AI given an unbounded goal (his canonical example: maximise paperclips) eventually consumes any resource it can to push the metric higher. The thought experiment was about cosmic-scale risk. The pattern itself has now appeared at human scale with financial implications.

Infrastructure Profusion does not only show up in dramatic API bills. The everyday version of it is what any developer using AI coding tools sees several times a day. Asked to add a single field, the agent rewrites the surrounding module. Asked to fix a bug, it generates 200 lines where 20 would do. Asked to write a small utility, it produces a configurable framework with abstractions, error handling for impossible scenarios, and tests for the abstraction layer rather than for the original task. Each individual decision looks reasonable. The cumulative output is wildly disproportionate to what was asked. This can lead to a degradation in engineering performance: the developer has to read, evaluate and simplify the over-built output before it can ship.

What turns Infrastructure Profusion from operational nuisance into governance concern is what comes next. The alignment research community has spent a decade studying related failure modes that current models are now demonstrating in controlled evaluations. Three of them deserve a board's quiet attention.

These behaviours are not theoretical. They have been demonstrated, in the lab, in 2024, in the frontier models your teams are deploying right now. Infrastructure Profusion has already crossed into production. The other three will follow the same path, on the same timeline, and the cost of the next $47,000 loop is going to look quaint.