Testing AI agents: tool calls, trajectories and permissions

An agent does not just answer; it acts on other systems, and its actions change state that later steps depend on. This article sets out how to test that behaviour, drawing on public agent benchmarks and the OWASP entry on excessive agency. Research figures carry a numbered source; directory counts are computed from published records at build time.

What makes an agent different to test

A chat application maps one input to one output. An agent loops: the ReAct pattern interleaves reasoning traces, which "track, and update action plans", with actions that call external sources [2]. OWASP describes agent-based systems as making "repeated calls to an LLM using output from previous invocations" to direct the next step [1]. Each loop adds a decision that can go wrong, and an early wrong step changes every later input.

State is the other difference. A tool call can book, delete, send or pay, and the effect stays after the conversation ends. AgentBench, which tests models in 8 interactive environments, names poor long-term reasoning, decision-making and instruction following as the main obstacles to usable agents [3].

Outcome evaluation and trajectory evaluation

Outcome evaluation checks the end state against a goal. tau-bench "compares the database state at the end of a conversation with the annotated goal state" [4]. SWE-bench, built from 2,294 real issues in 12 Python repositories, checks each fix with at least one test that failed before and passes after, plus a median of 51 tests for existing behaviour [5]. WebArena scores the functional correctness of completed web tasks; its strongest baseline agent succeeded on 14.41% of tasks against 78.24% for humans [6].

Outcome checks miss how the agent got there. An agent can reach the right end state after calling a tool it should not have used, or after skipping a confirmation step a policy requires. AgentBoard's authors note that a final success rate reveals "few insights during the process" and add a progress rate that credits partial advancement [7].

Score both. Write the expected end state for each task, then list the steps a correct trajectory must contain, the steps it must not contain, and the step limit. Grade free-text reasoning between calls with an LLM-as-judge only after calibrating it against human reviewers, as set out in how to evaluate LLM applications.

LLM-as-judge: Using one language model to grade another model's answers against a rubric.

How to validate tool calls

Validate every call the agent emits before judging the task. The Gorilla paper matches each generated call against an API database by its syntax tree and separates two errors [8]. A hallucination is "an API call that is not a sub-tree of any API in the database", while a call to a real API with wrong arguments is counted as an error [8].

hallucination: Model output that is fluent but not supported by facts or sources.

AgentBench classifies how runs end in a similar way: invalid format when the agent ignores the format instruction, invalid action when the format is right but the action is not allowed, and task limit exceeded when the agent runs out of rounds or repeats itself [3]. Log each class separately, because each has a different fix.

For each call, check the tool name against the tools offered, the arguments against the schema, the values against the task (right customer, right amount), the call count against a budget, and the order of calls where order matters. Test the cases where the right action is no call at all, including requests a domain policy says the agent must refuse [4].

How to test permissions and data access

OWASP LLM06:2025 defines Excessive Agency as the vulnerability that "enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM" [1]. It names three root causes: excessive functionality, excessive permissions and excessive autonomy [1]. Its mitigations translate into test cases.

• Functionality. List every tool and function the agent can reach and compare the list with what the use case needs [1].
• Permissions. Check the identity each tool uses downstream; OWASP's example is a read extension whose database identity also holds update, insert and delete rights [1].
• User context. Run the same task as two users with different rights and confirm actions execute "in the context of that specific user" [1].
• Approval. Confirm high-impact actions wait for human approval [1].
• Complete mediation. Ask the agent to do something its user may not do and confirm the downstream system refuses, since OWASP says authorisation belongs there, "rather than relying on an LLM to decide" [1].

Tool choice needs its own test. ToolPrivBench found that agents often pick a higher-privilege tool when a lower-privilege one would do, and that transient tool failures make this more common [12].

How to test hostile tool output

Every tool result is input to the next step, so a document, email or web page the agent reads can carry instructions. AgentDojo studies prompt injection "where data returned by external tools hijacks the agent to execute malicious tasks", with 97 tasks and 629 security test cases [9]. Plant instructions in the data each tool returns and check that the agent neither follows them nor leaks data to the address they name. Prompt injection and LLM security testing covers the attack classes, and the directory lists this work as AI security penetration testing.

prompt injection: Input crafted to override a model's instructions or extract hidden data.

How to inject tool failures

Benchmarks mostly test the happy path. The ToolMaze authors describe existing tool-use benchmarks as evaluating "idealized happy paths" and classify tool faults as explicit or implicit, and transient or permanent [11]. Performance fell under nearly all perturbations, most sharply under implicit failures where a tool returns wrong content without an error, which the authors attribute to "over-trust in corrupted outputs" [11].

Build a fault matrix for each tool: timeout, error code, empty result, truncated result, stale data and plausible but wrong data. Check that the agent retries within a limit, reports the failure to the user, and does not escalate to a broader tool [12]. Where real tools are too risky to call, ToolEmu emulates them with a language model; in its authors' human evaluation, 68.8% of the failures it found would be valid real-world failures [10].

Why one run per task is not enough

An agent that succeeds once may fail on the next attempt with the same input. tau-bench defines pass^k as "the chance that all k i.i.d. task trials are successful, averaged across tasks" [4]. Function-calling agents tested in the paper succeeded on fewer than 50% of tasks, and pass^8 in the retail domain fell below 25% [4].

Report the single-run success rate and pass^k side by side. Choose k from how often the same kind of task recurs in production, and require pass^k for every task that changes money, records or permissions. Agents that hold multi-turn conversations with users, as in tau-bench, also need the checks listed under conversational flow testing.

An agent test plan

1. 01

   Define end states

   For each task, write the expected state of every system the agent can change.

2. 02

   Define trajectories

   List required steps, forbidden steps, confirmation points and the step limit.

3. 03

   Validate every call

   Check tool name, schema, argument values, call count and call order, and classify each failure.

4. 04

   Test permissions

   Run tasks as users with different rights and confirm downstream systems enforce them.

5. 05

   Plant hostile data

   Insert instructions into tool results and confirm the agent ignores them.

6. 06

   Inject faults

   Return timeouts, errors, empty, stale and wrong data from each tool and check recovery.

7. 07

   Repeat and gate

   Run each task k times, report pass^k, and block release when a state-changing task falls below its threshold.

Rerun the plan on every change to the model, prompt, tool set or tool permissions; prompt and model regression testing is the directory service for that loop.

Which listed vendors state agent testing

43 published entries state AI agents as a product type they test, and 25 of them also state AI security penetration testing. The table counts entries for each related service.

Each count reports what a vendor's own AI page states, not a finding about delivery. The guide on which of the two security services to scope first separates attacks on the model from attacks on the tools around it.

Sources

[1] OWASP Gen AI Security Project. "LLM06:2025 Excessive Agency." OWASP Top 10 for LLM Applications 2025, OWASP Foundation, 2025. https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

[2] Yao, S. et al. "ReAct: Synergizing Reasoning and Acting in Language Models." arXiv (ICLR 2023), 2022. https://arxiv.org/abs/2210.03629

[3] Liu, X. et al. "AgentBench: Evaluating LLMs as Agents." arXiv (ICLR 2024), 2023. https://arxiv.org/abs/2308.03688

[4] Yao, S., Shinn, N., Razavi, P., Narasimhan, K. "tau-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains." arXiv, 2024. https://arxiv.org/abs/2406.12045

[5] Jimenez, C. E. et al. "SWE-bench: Can Language Models Resolve Real-World GitHub Issues?" arXiv (ICLR 2024), 2023. https://arxiv.org/abs/2310.06770

[6] Zhou, S. et al. "WebArena: A Realistic Web Environment for Building Autonomous Agents." arXiv (ICLR 2024), 2023. https://arxiv.org/abs/2307.13854

[7] Ma, C. et al. "AgentBoard: An Analytical Evaluation Board of Multi-turn LLM Agents." arXiv (NeurIPS 2024 Datasets and Benchmarks), 2024. https://arxiv.org/abs/2401.13178

[8] Patil, S. G. et al. "Gorilla: Large Language Model Connected with Massive APIs." arXiv, 2023. https://arxiv.org/abs/2305.15334

[9] Debenedetti, E. et al. "AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents." arXiv (NeurIPS 2024 Datasets and Benchmarks), 2024. https://arxiv.org/abs/2406.13352

[10] Ruan, Y. et al. "Identifying the Risks of LM Agents with an LM-Emulated Sandbox." arXiv (ICLR 2024), 2023. https://arxiv.org/abs/2309.15817

[11] Zhu, D. et al. "When Tools Fail: Benchmarking Dynamic Replanning and Anomaly Recovery in LLM Agents." arXiv preprint, 2026. https://arxiv.org/abs/2606.05806

[12] Yang, K. et al. "When Lower Privileges Suffice: Investigating Over-Privileged Tool Selection in LLM Agents." arXiv preprint, 2026. https://arxiv.org/abs/2606.20023