LLM agents in production: what actually breaks, and how to keep them reliable

An agent demo is one of the most persuasive things in software. You type a request in plain English, the agent reasons, calls a few tools, and hands back something that looks like magic. Then you put it in front of real users, real data, and real volume, and you discover that the demo was the easy 20%.

Agents are worth building, we build them for a living. But the gap between a compelling prototype and a system a business can depend on is real, and it's almost entirely about engineering discipline, not model choice. Here's what actually breaks, and what keeps agents reliable.

The failure modes we see most

• Confident wrong answers — the model hallucinates a fact or a value and states it with total confidence.
• Bad tool calls — it invokes the wrong tool, passes malformed arguments, or misreads the result.
• Brittle chains — a multi-step plan works until one step returns something unexpected, and the whole run derails.
• Latency and cost blow-ups — long reasoning loops and large contexts quietly turn a cheap feature into an expensive one.
• Prompt and version drift — a model or prompt update subtly changes behaviour, and nobody notices until output quality slips.
• No observability — when something goes wrong in production, there's no trace of what the agent actually did.
• Security and data leakage — prompt injection, over-broad tool permissions, or sensitive data landing where it shouldn't.

What reliable agents need

None of these are reasons not to ship. They're a checklist. The agents we put into production share the same backbone:

• Scoped tools and guardrails — least-privilege access, validated inputs and outputs, and hard limits on what the agent can touch.
• Evaluations, not vibes — a test set of real tasks with expected outcomes, run on every change, so quality is measured, not guessed.
• Retries, fallbacks, and budgets — timeouts, step limits, and a graceful fallback path when a tool or model fails.
• Observability and tracing — every run logged end to end, so you can see the reasoning, the tool calls, and where it went wrong.
• Human-in-the-loop where it matters — for high-stakes actions, the agent proposes and a person approves.
• Grounding over guessing — retrieval against your own governed data so answers cite real sources instead of inventing them.

Compliance is part of reliability

For EU teams especially, reliability and governance go together. The EU AI Act pushes toward transparency, human oversight, and risk management, exactly the practices that also make agents dependable. Building logging, oversight, and data-protection in from day one isn't overhead; it's the same discipline that keeps the system trustworthy.

Start small, then widen

The teams that succeed with agents don't launch an all-powerful autonomous system on day one. They pick one narrow, valuable task, wrap it in guardrails and evals, ship it, watch it closely, and expand scope only once it's proven. An agent that reliably does one thing well beats an ambitious one that occasionally embarrasses you.

The takeaway

The model is rarely the hard part anymore. Making an agent reliable, observable, safe, and cost-controlled, in the messy conditions of real operations, is the engineering that separates a demo from a product. That's the part worth investing in.