Agent Idempotency: The Production Gap That Breaks Multi-Agent Pipelines

Retries are not optional in distributed systems — they are how distributed systems survive. The moment you wire multiple agents together in production, you have a distributed system, and every agent call can fail. What most teams don't plan for is what happens when that failed call already did half its work.

Idempotency is the property that says: run this operation twice, get the same result as running it once. In CRUD APIs, engineers think about this on day one. In multi-agent LLM pipelines, I watch teams skip it completely — and then spend weeks debugging why bookings are duplicated, why ledger entries don't balance, or why a downstream agent received the same instruction four times and acted on all four.

This is the gap. It's not glamorous, but it's load-bearing.

Why Agents Break the Assumptions Classic Retry Logic Was Built On

HTTP retry logic assumes the call is either pure (read-only) or idempotent by construction (PUT, DELETE with a known resource ID). LLM agent steps are neither.

An agent step typically does some combination of: generating text with non-deterministic sampling, calling one or more tools (write to a database, send a message, book a slot), updating shared or persistent memory, and publishing a result to the next agent in the pipeline. Each of those sub-operations can succeed or fail independently. If the LLM call completes and the tool call crashes on write, you retry the whole step. Now the LLM may produce different output (temperature > 0), and the tool may execute a second time.

The blast radius depends entirely on what that tool does. A read is fine. An append to a ledger is not. A webhook to an external payment processor is definitely not.

In a single-agent loop this is already a problem. Across a multi-agent graph — where Agent A's output is Agent B's input, and Agent B is running concurrently with Agent C — the non-idempotent failure propagates and fans out.

The Four Failure Patterns You Will Actually See

1. Duplicate tool execution. The most obvious failure. An agent calls book_flight(), the write succeeds, the acknowledgment times out, and the orchestrator retries. Two bookings. In a travel platform context — where I work on multi-agent systems at Etera AI — this class of bug translates directly to real-world consequences, not just test failures.

2. Inconsistent state across agents. Agent A updates the session state and crashes before publishing to the queue. Agent B reads stale state and proceeds on the old version. When A retries, B has already branched on bad data. You now have two divergent execution paths that need to be reconciled manually.

3. Non-deterministic LLM output on retry. Even at temperature 0, model providers don't guarantee byte-identical outputs across calls with the same prompt — caching aside. At temperature 0.3–0.7 (where most production agents run, because you want some flexibility), the retry produces a materially different decision. If you treat the retry as equivalent to the first attempt, you've silently swapped decisions mid-pipeline.

4. Cascading acknowledgment failures. In async pipelines (Kafka, SQS, Redis Streams), an agent that successfully processes a message but fails to commit the offset will receive that message again. If every step downstream is not idempotent, you get a cascade of duplicate work — each agent executing once more for every retry in the chain above it.

Building the Idempotency Layer: What Actually Works

The fix is an idempotency key that flows with the work unit from the moment it enters the pipeline to the moment it exits.

Here's the minimal pattern:

python
import uuid
import hashlib
from datetime import datetime, timedelta

def create_idempotency_key(agent_id: str, task_payload: dict, run_id: str) -> str:
    """Stable key: same agent + same task + same pipeline run = same key."""
    payload_hash = hashlib.sha256(
        str(sorted(task_payload.items())).encode()
    ).hexdigest()[:16]
    return f"{run_id}:{agent_id}:{payload_hash}"

async def execute_agent_step(
    agent_id: str,
    task_payload: dict,
    run_id: str,
    store,  # Redis, DynamoDB, Postgres — any atomic store
) -> dict:
    key = create_idempotency_key(agent_id, task_payload, run_id)
    ttl_hours = 24  # tune to your pipeline's retry window

    # Atomic check-and-set: only one winner
    existing = await store.get(key)
    if existing:
        return existing  # return cached result, skip execution

    # Lock the key before executing (prevents concurrent duplicates)
    acquired = await store.set(key, {"status": "in_progress"}, nx=True, ex=ttl_hours * 3600)
    if not acquired:
        raise RetryableConflict(f"Key {key} already in progress")

    try:
        result = await run_agent_logic(agent_id, task_payload)
        await store.set(key, {"status": "complete", "result": result}, ex=ttl_hours * 3600)
        return result
    except Exception as e:
        await store.delete(key)  # release lock on failure so retry can proceed
        raise

A few decisions buried in this pattern that matter:

• nx=True (set if not exists) is the atomic operation that prevents race conditions when two retries fire simultaneously. Without atomicity, two workers can both read "no key exists" and both proceed.
• Delete on failure lets the next retry attempt the operation clean. If you leave an in_progress key on failure, your pipeline deadlocks until TTL expiry.
• TTL tied to your retry window: if your orchestrator gives up after 6 hours, a 24-hour TTL is safe. Match these.
• The key includes run_id: this scopes idempotency to a single pipeline execution. If the same task legitimately re-runs in a new pipeline run, it should execute fresh.

Tool Calls Are the Hard Part

LLM generation you can cache. Tool calls that write external state you cannot un-execute. This is where you need a different strategy per tool type.

Tool Type	Idempotency Strategy
Database insert	Use upsert with idempotency key as unique constraint
External API (payment, booking)	Pass idempotency key in request header — most payment APIs (Stripe, Adyen) support this natively
Message queue publish	Use message deduplication ID (SQS FIFO, Kafka exactly-once semantics)
Email / notification send	Track sent events by idempotency key before calling provider
File write / S3 put	Object key = idempotency key; overwrite is safe if content is deterministic
LLM generation	Cache against (model, prompt hash, temperature=0) — skip cache at temperature > 0 unless you pin seed

The pattern for external APIs is: push your idempotency key into their system. Stripe's idempotency key header has been a standard since 2015. If your payment vendor doesn't support idempotency keys, that's a vendor selection problem, not something to paper over in your retry logic.

The LLM Output Problem: When Retries Change Decisions

This is the subtler issue that no checklist covers cleanly. If you retry an agent step at temperature > 0, the LLM may genuinely produce a different answer. Which answer is correct? The first one, by definition — because the first execution set downstream state.

The practical rules I apply:

1. Cache the first successful LLM output as part of the idempotency record. On retry, return the cached output. Do not re-invoke the LLM. This is the only way to guarantee downstream consistency.
2. For decisions with external consequences (booking, payment, send), set temperature to 0 for that specific step. You lose some quality, but you gain determinism. The creative exploration can happen earlier in the pipeline, before the write boundary.
3. Mark your "write boundary" explicitly in your agent graph. Everything before it is exploratory and retryable freely. Everything at or after it uses cached LLM output + idempotent tool calls.

This ties directly to thinking about your agents as state machines — once an agent has crossed the write boundary, it should not revisit that transition on retry, it should resume from it.

Cross-Agent Idempotency: The Protocol Layer

When Agent A's result is Agent B's input, you need idempotency at the handoff, not just within each agent. I covered the contract structure for agent handoffs in Agent Handoff Contracts: The Protocol Gap Costing You Production Reliability — the short version is that every message crossing an agent boundary should carry the originating run_id and a step_id that Agent B uses as its own idempotency key seed.

Without this, Agent B has no way to know if it's processing a legitimate new instruction or a retry of an instruction it already processed and the acknowledgment was lost.

python
# Message envelope for inter-agent communication
{
  "run_id": "run_abc123",
  "step_id": "step_002",        # Agent A's step that produced this message
  "source_agent": "research_agent",
  "target_agent": "booking_agent",
  "payload": { ... },
  "produced_at": "2025-07-31T10:00:00Z",
  "idempotency_key": "run_abc123:booking_agent:step_002"  # pre-computed for target
}

Agent B checks idempotency_key before processing. If it's already in the store with status complete, it returns the cached result immediately and does not re-execute. This is the difference between a pipeline that recovers gracefully and one that requires manual intervention every time a worker restarts.

What to Actually Do

1. Audit every tool call in your agent graph today. Classify each as read, idempotent-write, or non-idempotent-write. If you have any non-idempotent writes without an idempotency key strategy, you have a live bug waiting for the next retry event.

2. Add an idempotency key store (Redis is fine) with TTL matching your retry window. Wire it as middleware, not per-agent logic. Every agent step goes through the same check-and-set pattern before executing.

3. Cache the first successful LLM output in your idempotency record. Never re-invoke the LLM on retry for a step that already produced output. Return the cached result.

4. Push idempotency keys to external vendors. Check your payment, notification, and booking providers for native idempotency key support. Use it. This offloads the deduplication burden to the vendor's systems.

5. Add a write-boundary marker to your agent graph spec. Steps before it are freely retryable. Steps at or after it must use full idempotency discipline. Make this explicit in documentation — not assumed.

Idempotency is not a feature you add when things break. By the time the duplicates show up in production, they're already in your customer's account history. Wire this before you ship agentic workflows with write access to anything that matters.