Classical software fails honestly. An exception has a stack trace, a crashed process trips an alert, a 500 turns the dashboard red. The entire operational practice of IT (monitoring, alerting, on-call) rests on a quiet assumption: when something breaks, you notice.
AI agents break that assumption. A language model in a loop always produces something, even when its tools come back empty, its context has been clipped, or its stop signal got lost. The failure arrives as plausible-looking output. At scale.
The five failure patterns below come from agentic systems we build and run on our own hardware with local models. Every one of these stories happened to us. Not one threw an exception.
An agent's most dangerous failure is the output that looks like work.
The update that removed the stop
One of our pipelines has local language models generate scripts overnight, score them automatically, and refine them over successive rounds. Four parallel inference slots, each with a large context window. One evening, the second round seemed stuck. No error, no crash, the system simply kept running.
The cause was three days old. A chat-template update had accidentally declared a tool token to be an end-of-generation signal. The inference server, llama.cpp in our case, thereby lost the real stop token from its list. After that, generations of over 90,000 tokens ran through that nobody had asked for. All four slots occupied. By then the gateway had long since answered the requests with a timeout. The orphaned generations kept decoding in the background, invisible to anyone watching only the API.
Model updates are a supply chain for us now: weights, chat template and tokenizer metadata all change behavior, even when 'only a small thing' was patched. After every update we check the stop-token configuration in the inference server's log [1]. And a truncated generation (finish_reason: length) is a first-class alarm in monitoring, not a footnote.
# after every model update: is the stop-token list still intact?
podman logs llama-server 2>&1 | grep special_eog_ids
# in the harness: a truncated generation is an alarm, not a footnote
if response.finish_reason == "length":
alert("runaway_generation", model=model, tokens=usage.completion_tokens)The pile called 'Other'
One of our workflows triages an inbox, a local model classifies every e-mail into defined categories. Twice this workflow failed quietly, in two different ways and with the same symptom.
The first time, a well-intentioned output cap was to blame. A reasoning model spent its token budget mid-thought, the actual answer came back empty, and the pipeline filed every single mail under 'Other'. That looked like a model quality problem and was a budget problem. Removing the max_tokens limit fixed it outright.
The second time, the model server was unreachable. Every call failed within milliseconds, and the workflow dutifully recorded every mail as 'Other (error)', while the progress display looked frozen, because everything 'finished' so fast. Today the workflow aborts hard when the first wave of calls fails across the board, with a red banner instead of a fully 'processed' inbox.
# fail fast: if the first wave fails across the board, stop the job
first = [classify(mail) for mail in inbox[:3]]
if all(r.status >= 500 for r in first):
raise PipelineDown("model unreachable, refusing to file everything as 'Other'")Nobody double-checks the 'Other' pile. A loud context-length error gets noticed immediately. A plausible wrong answer caused by a hidden cap just sits there. And a pipeline whose model is down must stop, instead of uniformly 'working' wrong.
Check the tool before you blame the model
An autonomous agent was set up to drive a third-party system through its scripting API. The local 12B model did poorly, and switching to a large cloud model brought some improvement. So the obvious suspicion was that the small model is too weak.
The real cause sat in the logs. The tool that prepared the system's API reference for the agent was silently returning zero functions and zero namespaces: a parser tripped up by Windows line endings and a typed stub format. Both models had been working without a real API reference the entire time, reconstructing function signatures from the memory of their training data. The small model had even reported twice that the function list looked empty. It was right, and nobody listened. The fix was a few lines in the parser: from zero to 49 functions.
'The model is too dumb' is the most expensive misdiagnosis in agent operations, because its therapy is a bigger model and more hardware instead of a bugfix. Since then, tool outputs get hard invariants here. A reference with zero entries is a reason to halt the pipeline. And when a model reports that its input looks empty, it is worth a look.
# a tool output with zero entries is a reason to stop, not an input
reference = parse_api_stubs(raw_docs)
if not reference.functions:
raise ToolOutputError("API reference is empty, halting pipeline")Progress that isn't
An extraction pipeline was meant to move facts from a large document set into a knowledge base, section by section, in a loop with the termination condition: stop when no new facts are added. A single section yielded 155 'new' facts. On closer inspection, most were rewordings: ten to thirteen variants of the same underlying fact per entity. Each new phrasing was technically a new entry and reset the progress counter. The loop ran in circles, looking productive.
A stricter prompt ('no rewordings!') changed nothing. Lexical similarity measures could not reliably separate a paraphrase from a genuinely new, related fact either. What finally worked was semantic deduplication by embedding instead of string comparison.
A model in a loop satisfies the letter of its termination condition, not its intent, and that holds well beyond this example. Progress measures for agents have to measure meaning, not strings. Otherwise the system 'makes progress' for as long as the power stays on.
Green dashboards, invented answers
Two shorter finds from the same family. In the Model Context Protocol, the open standard for connecting tools to agents, a failed tool call comes back as an error flag inside a successful HTTP response [2]. A logger that only looks at the status code shows an error rate of zero while every single call is failing. The transport reports success, and the error sits one protocol layer deeper.
# MCP: the transport says 200, the truth sits in the payload
result = await session.call_tool(name, args)
if result.isError:
metrics.increment("tool_error", tool=name) # this call was NOT a successIn an assistant system we had switched off web search, cleanly removed from the tool list. The system instruction still said: 'use web search for current information.' The model resolved the contradiction its own way. It invented search results, including a freely hallucinated weather report, in a tone of complete confidence [3]. What helped was telling the model explicitly that the tool is gone, and forbidding fabrication. Afterwards, the same question got the honest answer: no current data available.
Hallucination lives in the gap between what an agent is allowed to do and what it has been told to do. Anyone switching tools off per tenant, per role or for compliance reasons (the normal case in enterprise use) has to test exactly this case explicitly.
Failing loudly is an architecture decision
Five stories, one pattern. Nowhere did an exception fly, every system kept running and produced something that looked like work: endless generations, a sorted inbox, API calls from memory, a growing knowledge base, a green dashboard. Reliable agents come from failure becoming visible before it has a real cost. Before the GPU spends nights computing scrap, and before people manually correct the very work the AI was meant to take off their hands [4]:
- Runaway detection as infrastructure: count truncated generations, check the stop-token configuration after every model update.
- Circuit breakers: if the first calls fail across the board, the pipeline halts. Loudly.
- Invariants on tool outputs: an empty reference or an empty answer halts the pipeline.
- Measure progress semantically: termination conditions via embeddings, not strings.
- Evaluate error semantics at every protocol layer, not just the HTTP status.
- Capability changes reach the agent's instructions, not just its tool list.
Not one of these failures was found by looking at 'the model'. Every single one sat in the logs of a layer we operate ourselves: the inference server, the gateway, the orchestration's own trace. On-premise does not prevent these failures, but it makes every layer they hide in inspectable. On a managed agent platform, several of those layers are a third party's black box, and a quiet failure stays quiet. What an appliance looks like where hardware, model, and agent orchestration run under your control: see Product, and what agents concretely take over in mid-sized companies today: see Use cases.