Agent Failure Modes: 5 Production Breaks Nobody Demos

Most multi-agent demos work perfectly. The orchestrator calls a tool, the sub-agent responds, the output lands clean. Then you hit production and the whole thing unravels — not from some exotic edge case, but from five failure modes that are completely predictable if you've shipped these systems before.

I've been building multi-agent LLM systems at Etera AI — a multi-agent platform for travel — and the patterns of failure are consistent enough that I can now name them before a team hits them. This article is that taxonomy, with enough engineering detail to actually do something about it.

Why the Demo Always Works (And Production Doesn't)

Demos are optimized for the happy path: the right tool is called with clean inputs, the model reasons correctly, and there's no latency pressure, no concurrent load, and no ambiguous state. Production is the opposite of that. You have:

• Inputs that are partially malformed, ambiguous, or in the wrong language
• Tools that fail, timeout, or return partial results
• Multiple agents writing to shared state simultaneously
• Token budgets that blow out when reasoning chains go deep
• Users who don't follow the expected interaction flow

None of these are surprises. They're structural properties of production systems. The mistake teams make is architecting for the demo and then bolting on resilience after the first outage.

Failure Mode 1: Context Rot Across Long Agent Chains

When you chain three or more agents, each one receives a compressed or summarized version of what the previous one produced. The problem isn't summarization per se — it's that each compression step introduces subtle drift. By the time you're at agent five in a chain, the task description has morphed enough that the agent is solving a slightly different problem than the one the user actually submitted.

This is context rot, and it's insidious because individual agents look correct when you inspect them in isolation. The system is wrong but no single component is obviously broken.

How to design around it:

• Maintain a canonical task object (a structured dict or JSON schema) that passes through the entire chain without summarization. Each agent reads from it and writes deltas back to it, rather than receiving prose summaries from the upstream agent.
• If you must use prose handoffs (for natural-language reasoning steps), include a verbatim copy of the original user intent in every agent's system prompt. It's extra tokens, but it's an anchor.
• Run periodic consistency checks: after every N agents, have a lightweight validator agent compare the current working state against the original task spec and flag divergence before it compounds.

Failure Mode 2: Tool Call Ambiguity and Silent Wrong-Tool Selection

LLMs select tools based on the names and descriptions you give them. When two tools have overlapping functionality — say, search_flights and search_availability in a travel system — the model will guess, and it will sometimes guess wrong. The dangerous part is that it won't tell you it's guessing. It will call the wrong tool with full confidence, get a result, and continue reasoning as if everything is fine.

This is a specification problem, not a model problem. No amount of prompting fixes it reliably. You have to design the tool surface.

What actually works:

• Write tool descriptions as decision rules, not feature lists. Instead of "searches for available flights," write "use this when you have an origin, destination, and date range and need to return a list of flight options. Do NOT use this to check seat availability on an already-selected flight."
• Group tools by domain and present only the relevant tool subset to each sub-agent. An agent responsible for itinerary planning doesn't need access to payment tools. Narrow the choice space and wrong selections drop dramatically.
• Log every tool call with the agent's reasoning trace (the chain-of-thought before the tool call). When you see a wrong tool selected, you'll see exactly why — and you can fix the description or the routing logic.

Failure Mode 3: State Collision in Concurrent Agent Execution

This is the distributed systems problem that teams who come from sequential LLM pipelines aren't expecting. When you fan out to multiple agents in parallel — which you should, because serial execution is slow — they all need somewhere to write results. If that's a shared dict in memory, or a shared database row, or even a shared Markdown document in a scratch pad, you now have a race condition.

Imagine a team building a research agent that fans out to five sub-agents, each retrieving a different data source, all writing their findings to the same "research summary" object. With async execution, you get overwrites, partial merges, and summary sections that are internally inconsistent because they were written by different agents with different local context.

The fix is to treat agent state like you'd treat a distributed database:

• Give each agent a scoped, namespaced write area. They write to agent_id/output, never to a shared root.
• Use a merge step — a dedicated aggregator agent or a deterministic merge function — that explicitly combines scoped outputs. Don't let agents merge each other's work.
• If you're using something like Redis or a document store for scratch state, use optimistic locking or versioned writes. The LLM world has been slow to adopt these patterns from distributed systems. That's a gap you can close in a day of engineering.

Failure Mode 4: Retry Loops That Compound Token Cost Without Converging

Agents retry. That's a feature. When a tool call fails, when an output doesn't pass validation, when a sub-agent returns an unexpected schema — retrying is the right behavior. The problem is that naive retry logic doesn't change the conditions that caused the failure. You get an exponential blowup in token usage, your costs spike, and you often still don't get a correct result.

In systems like this, a single ambiguous user query can trigger a retry cascade that consumes many times the token budget of a successful path, all while the agent is confidently reporting progress. The cost invisibility makes it worse — you don't see it until the billing cycle.

(If you're not already watching token cost at the agent-operation level, read Stop Paying for Tokens You Don't Need — the cost math at scale is genuinely different from what most teams assume.)

Designing retry logic that actually converges:

• Apply a maximum retry budget per task, not just per tool call. If the total retry count across all tools in a task exceeds N, escalate or fail gracefully rather than continuing to burn tokens.
• On retry, modify the prompt or tool parameters — not just call the same thing again. The simplest version: append the previous failure reason to the tool call context. The model can often self-correct if it knows what went wrong.
• Classify failures before retrying. A 429 rate limit is retriable with backoff. A schema validation error is not retriable without prompt intervention. A tool that returned null because the resource doesn't exist is not retriable at all. Treat these differently in your retry logic — not as a single catch-all.

python
import time
from enum import Enum

class RetryClass(Enum):
    RETRIABLE = "retriable"          # backoff and retry as-is
    PROMPT_FIXABLE = "prompt_fixable" # modify context, then retry
    TERMINAL = "terminal"            # do not retry

def classify_error(error: dict) -> RetryClass:
    error_type = error.get("type")
    if error_type == "rate_limit":
        return RetryClass.RETRIABLE
    if error_type == "validation_error":
        return RetryClass.PROMPT_FIXABLE
    if error_type == "not_found":
        return RetryClass.TERMINAL
    return RetryClass.TERMINAL  # fail closed on unknown errors

def retry_tool_call(tool_fn, args, context, max_retries=3):
    attempt = 0
    while attempt < max_retries:
        try:
            return tool_fn(**args)
        except Exception as e:
            error = {"type": getattr(e, "error_type", "unknown"), "message": str(e)}
            classification = classify_error(error)

            if classification == RetryClass.TERMINAL:
                raise  # no point retrying

            if classification == RetryClass.PROMPT_FIXABLE:
                # Inject the failure reason so the model can self-correct
                context["last_error"] = error["message"]
                args = rebuild_args_from_context(context)  # re-derive call args

            if classification == RetryClass.RETRIABLE:
                time.sleep(2 ** attempt)  # exponential backoff

            attempt += 1

    raise RuntimeError(f"Tool call failed after {max_retries} attempts")

Fail closed on ambiguous errors. Always.

Failure Mode 5: Guardrail Bypass Through Agent-to-Agent Trust

This is the one that keeps me up at night, and it's underappreciated because it requires thinking about the threat model of your own system. When Agent A calls Agent B, Agent B typically doesn't verify that the instruction is legitimate. It trusts the orchestrator. That trust chain is exploitable.

In practice, this surfaces in two ways. The first is indirect prompt injection: a tool returns content from an external source (a website, a database record, a user-submitted document) that contains embedded instructions, and a downstream agent follows them because it can't distinguish between system instructions and data. The second is privilege escalation within your own system: an agent granted read-only access asks a higher-privilege agent to perform a write, and the higher-privilege agent complies because the request comes from within the trusted agent network.

This isn't theoretical. The security community has documented prompt injection against multi-agent systems in public research, and write-access agents carry a security tax that most teams underprice.

What production-grade agent security looks like:

• Every agent that executes actions (writes, API calls with side effects, external communications) should validate that the original human user authorized the action — not just that an upstream agent requested it. Build an authorization check into the tool wrapper, not just the orchestrator.
• Treat all content retrieved from external sources as untrusted data. Sanitize it before it enters an agent's reasoning context. The simplest heuristic: if a retrieved document contains text that looks like a system prompt (You are a..., Ignore previous instructions...), strip or flag it before passing it downstream.
• Implement a least-privilege principle at the agent level: define a permission set per agent role and enforce it in your tool registry, not in the prompt. Prompts are persuadable. Code is not.

The Common Thread: These Are Architecture Problems, Not Prompt Problems

Every one of these failure modes has a version of the same misdiagnosis: teams try to fix them with better prompts. More detailed instructions, more examples, more explicit chain-of-thought guidance. Sometimes that helps at the margins. It never fixes the root cause.

Context rot is a data architecture problem. Tool ambiguity is a specification design problem. State collision is a distributed systems problem. Retry loops are a control flow problem. Guardrail bypass is a security architecture problem. None of them live in the prompt.

The engineering discipline of multi-agent systems is still being written in real time. But the failure modes are converging fast, and the teams that name them clearly are the ones building systems that survive contact with real users.

The prompt is the last place to look. If your multi-agent system is fragile, the architecture is fragile.

What to Actually Do

1. Audit your handoffs. Map every agent-to-agent data transfer in your current system. For each one, ask: is the original user intent still verbatim-accessible at this point? If not, add an anchor.
2. Rewrite your tool descriptions as decision rules. Every tool description should tell the model when NOT to use the tool. Do this in a single afternoon — it's the highest ROI prompt engineering you'll do.
3. Add namespaced state from the start. If you're not in production yet, this costs you almost nothing to add now. If you are in production, schedule it as a refactor — the first time you hit a state collision in production you'll wish you had.
4. Build a failure classifier before you build retry logic. One enum, three categories (retriable, prompt-fixable, terminal), applied before every retry decision. Write it in a day.
5. Add an authorization check to every action-taking tool wrapper. Not in the orchestrator, not in the prompt — in the tool. Treat it like you'd treat auth middleware in an API: it belongs at the boundary.

The demo ships in a day. The production system ships in a week. The failure modes hit in month two. Design for month two from day one.