Agent Reliability Is an Evaluation Problem, Not a Prompting Problem

Agents are getting better at taking actions, but I think we are still diagnosing many of their failures at the wrong layer.

When an agent misses a step, developers often return to the prompt. Add another instruction. Make the wording stricter. Tell the model to double-check its work. This can help, especially when the task is poorly specified.

But once an LLM is operating across tools and changing external state over many steps, reliability is no longer mainly a writing problem. It becomes a systems problem: how do we know the task is complete, how do we know each important state transition actually happened, and how do we detect failure before the agent builds on top of it?

My view is that the next improvement in agents will come as much from evaluation and verification as from better prompting.

A good next action is not the same as a successful task

Single-turn LLM evaluation is relatively simple. Give the model an input, inspect the response, score the answer.

An agent operates differently. It reads an environment, chooses an action, receives feedback, updates its plan and acts again. A failure early in that sequence changes the state seen by every later step.

That means local correctness is not enough.

An agent can make ten reasonable decisions and still fail the task because it edited the wrong file, misunderstood a tool result or stopped one step too early. Conversely, its internal reasoning may look messy while the final environment is exactly correct.

This is why I care more about task-level success than whether an agent produced a convincing trace.

WebArena makes the distinction concrete. Its tasks are long-horizon interactions with realistic websites, and success is checked against the resulting environment rather than the elegance of the generated text. The best GPT-4-based baseline reported 14.41% end-to-end success, while humans reached 78.24%. That gap is difficult to explain as a missing prompt trick. It reflects navigation, state tracking, tool use and error recovery across the whole trajectory.

The environment needs to grade the agent

SWE-bench uses a similar idea for software engineering.

The model receives a real GitHub issue and repository, then has to produce a patch. The important part is how the patch is judged. Tests that previously failed must pass, while tests that were already passing must remain passing.

The agent does not get credit because its patch looks plausible. The repository decides whether the work succeeded.

OpenAI's SWE-bench Verified release in August makes another useful point: even the evaluator needs evaluation. OpenAI and the SWE-bench authors found tasks in the original benchmark that were ambiguous, broken or impossible to solve as specified. They created a human-validated subset of 500 tasks to make the measurement more reliable.

That is a pattern I expect production teams to encounter repeatedly. Before improving the agent, define what success actually means and verify that the test can distinguish success from failure.

For a coding agent, that can be unit tests. For a data workflow, it can be schema checks and reconciled totals. For an asset pipeline, it can be file existence, naming rules, polygon limits, render settings and export validation.

Prompt engineering stops at the system boundary

SWE-agent is one of the clearest signals here.

The project does not treat the language model as the only object worth optimising. The researchers design an Agent-Computer Interface specifically for the model: commands for viewing files, searching repositories, editing code and running programs, with feedback that helps the model understand when an operation failed.

Using GPT-4 Turbo, SWE-agent reached 12.5% on the original SWE-bench. The paper's experiments show that changing the interface between the model and the environment can materially change agent performance without changing the underlying task.

Tool shape matters. Error messages matter. State representation matters. The amount of irrelevant information returned after an action matters. Whether the model can easily verify a change matters.

You can spend hours rewriting a system prompt while leaving a poorly designed tool interface untouched. At some point that becomes optimisation at the wrong layer.

I see the same problem in production automation. If I expose Blender automation through one unrestricted Python execution tool, the model has enormous freedom but weak observability. If I expose narrower operations such as validate_scene, list_export_errors and export_asset, each operation can return structured evidence about what actually happened.

Reliability requires explicit completion criteria

One of the most dangerous agent failures is not a dramatic error. It is premature success.

The model believes the task is finished, produces a confident final response, and stops. The external state says otherwise.

A production agent therefore needs completion criteria that exist outside the model's own judgment.

If the instruction is "prepare every product asset for export," the completion condition should not be "the model says it finished." The pipeline can count the target assets, verify that each required export exists, run validation rules and report which items failed.

This is close to how I already think about render and build pipelines. A Blender script does not become reliable because it prints done. A job is complete when the expected artifacts exist and pass the checks that matter downstream.

Agents should be held to the same standard.

The model can decide what to try next. Deterministic code should decide whether important invariants still hold.

Long tasks expose reliability mathematically

There is a simple reason agent reliability becomes harder as tasks get longer.

Imagine a workflow where every necessary step has a high probability of being executed correctly. If twenty dependent actions all have to succeed, even a small per-step failure rate can produce a much lower end-to-end success rate.

Real agent failures are not independent, so this is only an intuition, not a model of actual performance. In practice it can be worse because one mistaken state transition can contaminate several later decisions.

This is why measuring individual tool-call accuracy is not enough. We need both local metrics and end-to-end evaluation.

OSWorld does this for computer-use agents by creating real desktop tasks with reproducible initial states and execution-based grading. Its 2024 paper reported 12.24% success for the best evaluated model against 72.36% for humans at the time of publication.

What I would build into an agent system now

For production work, I would treat evaluation as part of the architecture rather than a final QA stage.

Every important tool call should produce observable state. Actions that change files, databases or project settings should have a verification path. Long workflows should have checkpoints where the system can test whether assumptions still hold before continuing.

I would keep a collection of real failed tasks and replay them whenever the model, prompt, tool definitions or orchestration logic changes. That test set should include normal cases, edge cases and cases where the agent previously stopped too early or modified the wrong state.

I would track more than final success. Number of tool calls, retries, invalid actions, recovery attempts, latency and cost can reveal a system that technically succeeds but is becoming unstable.

Security belongs in the evaluation set too. AgentDojo shows why: once agents consume data from external tools, that data can contain instructions designed to redirect the agent. A system can be capable on normal tasks and still be unreliable in an adversarial environment.

A clear prompt remains useful, but it stops being the final control mechanism once the model can affect real systems.

My prediction

I think agent development will start to resemble software testing more than prompt crafting.

Teams will maintain task suites with known initial states and machine-checkable outcomes. Failed production runs will become regression tests. Tool interfaces will be changed when repeated errors reveal that the model receives poor feedback. Agents will be evaluated over complete trajectories rather than screenshots of impressive individual steps.

The strongest systems may be the ones that make failure visible early and success objectively measurable.

For me, that is the dividing line between an agent demo and production automation.

A prompt tells the model what we want.

An evaluation tells us whether we actually got it.

Sources

1. OpenAI, Introducing SWE-bench Verified, 13 August 2024.
2. Yang et al., SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering, 2024.
3. Zhou et al., WebArena: A Realistic Web Environment for Building Autonomous Agents, ICLR 2024.
4. Xie et al., OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments, 2024.
5. Debenedetti et al., AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents, 2024.