Building agents that survive production reality

Every agent demo looks the same: a clean prompt, a cooperative API, and a happy-path response that earns applause at the conference. What you never see is the agent three weeks later, wedged between a legacy ERP that returns XML, a rate-limited third-party service, and an on-call engineer who just got paged at 2 a.m.

The gap between demo and production is not a technical gap — it is a systems-thinking gap. Agents fail in production for the same reasons distributed systems fail: partial failures are not handled, retries are not idempotent, and state is assumed to be reliable when it is not. The LLM at the center of your agent is the least of your problems; the scaffolding around it is everything.

Durable execution is the first thing to get right. Whether you reach for a workflow engine like Temporal or build your own checkpoint layer, every action your agent takes should be replayable. If the process crashes mid-task, the agent must be able to resume from the last known good state — not restart from scratch and bill the user twice for a flight they already booked.

The second thing is observability. Trace every tool call, every LLM invocation, every decision branch. Structured logs and spans are not optional extras; they are the only way you will diagnose the class of bugs that only surface when real users interact with your agent in ways you never imagined. Production agents that cannot be observed cannot be improved, and cannot be trusted.