How security testing of an LLM application works

What is prompt injection?

prompt injection is the entry LLM01:2025 in the OWASP Top 10 for LLM Applications 2025 [1]. OWASP describes it as user prompts that alter the model's behaviour or output in unintended ways [2]. The model reads instructions and data in the same text stream, so any text it reads can try to act as an instruction. OWASP also states that "it is unclear if there are fool-proof methods of prevention for prompt injection" [2].

prompt injection: Input crafted to override a model's instructions or extract hidden data.

OWASP treats jailbreaking as one form of prompt injection, where the input makes the model "disregard its safety protocols entirely" [2]. A jailbreak attacks the model's safety training. A wider prompt injection test also attacks what the application lets the model see and do.

How do direct and indirect prompt injection differ?

Direct injection comes from the person typing into the application. OWASP defines it as a user's prompt input that "directly alters the behavior of the model in unintended or unexpected ways" [2]. The attacker needs an account or a public chat box.

Indirect injection arrives through content the model reads on the user's behalf. OWASP defines it as the case where "an LLM accepts input from external sources, such as websites or files" [2]. Greshake et al. showed in 2023 that attackers can exploit LLM-integrated applications "remotely (without a direct interface)" by placing prompts in data likely to be retrieved [3]. Their paper argues that these applications "blur the line between data and instructions" [3].

The paper's taxonomy of impacts includes data theft, worming and information ecosystem contamination, and the authors demonstrated the attacks on real-world and synthetic applications [3]. For a test plan, this means every source the model reads is an attack surface: web pages, uploaded files, emails, tickets and documents in a RAG index. OWASP adds a multimodal case in which instructions are hidden in images that accompany benign text [2].

RAG: Retrieval-augmented generation: the model answers from documents fetched at query time.

Which OWASP 2025 items does a security test cover?

The OWASP Top 10 for LLM Applications 2025 was published by the OWASP Gen AI Security Project in November 2024 [1]. Its ten official item names are listed below [1].

ID	Official name
LLM01:2025	Prompt Injection
LLM02:2025	Sensitive Information Disclosure
LLM03:2025	Supply Chain
LLM04:2025	Data and Model Poisoning
LLM05:2025	Improper Output Handling
LLM06:2025	Excessive Agency
LLM07:2025	System Prompt Leakage
LLM08:2025	Vector and Embedding Weaknesses
LLM09:2025	Misinformation
LLM10:2025	Unbounded Consumption

LLM01, LLM02, LLM05, LLM06, LLM07 and LLM08 can be tested from outside through the running application. LLM03 and LLM04 concern the model and data supply chain, so testing them needs access to build pipelines and training data. LLM09 overlaps with output quality work such as LLM output evaluation.

How are data leakage and system prompt leakage tested?

Sensitive Information Disclosure covers data such as "personal identifiable information (PII), financial details, health records, confidential business data, security credentials, and legal documents" [4]. Testers seed the environment with marked canary records and try to make the model return records that belong to another user or tenant. A finding records which prompt or document caused the leak and which access control failed.

System Prompt Leakage concerns instructions that "contain sensitive information that was not intended to be discovered" [5]. OWASP states that the system prompt "should not be considered a secret, nor should it be used as a security control" [5]. A test therefore does two things: it tries to extract the system prompt, and it checks whether anything in the prompt (keys, internal URLs, role rules) would cause harm once extracted. MITRE ATLAS lists system prompt discovery as technique AML.T0069.002, System Prompt [9].

What is improper output handling?

Improper Output Handling is "insufficient validation, sanitization, and handling of the outputs generated by large language models before they are passed downstream" [6]. In the 2023/24 edition the same risk was named Insecure Output Handling, so older reports use that term [7]. The test treats model output as untrusted user input to every component that consumes it.

OWASP gives concrete cases: generated JavaScript or Markdown interpreted by the browser leads to XSS, and generated SQL run without parameterisation leads to SQL injection [6]. Output passed to a shell or to functions such as exec or eval can lead to remote code execution [6]. One OWASP scenario combines indirect injection with rendering: hidden instructions make the model insert an image link that exfiltrates the private conversation [2]. MITRE ATLAS lists this pattern as AML.T0077, LLM Response Rendering [9].

How is tool abuse by an agent tested?

When the model can call functions, OWASP classifies the risk as Excessive Agency [8]. It is defined as the vulnerability "that enables damaging actions to be performed in response to unexpected, ambiguous or manipulated outputs from an LLM" [8]. OWASP names three root causes: "excessive functionality; excessive permissions; excessive autonomy" [8].

A tool abuse test lists every tool, the permissions each one runs with and which actions need human confirmation. Testers then use direct and indirect injection to make the agent call tools outside the user's intent, such as sending email, changing records or reading another tenant's data. MITRE ATLAS covers this as AML.T0053, AI Agent Tool Invocation, and AML.T0086, Exfiltration via AI Agent Tool Invocation [9]. Functional agent testing is the subject of the post testing AI agents, tools and trajectories; the AI agents page lists entries that state agent testing.

Which MITRE ATLAS techniques map to these tests?

MITRE ATLAS is a public knowledge base of adversary tactics and techniques against AI systems; release 2026.10 is dated 28 October 2026 [9]. The technique ids below come from that release [9].

Test area	ATLAS id and name
Direct injection	AML.T0051.000 Direct (under AML.T0051 LLM Prompt Injection)
Indirect injection	AML.T0051.001 Indirect
Safety bypass	AML.T0054 LLM Jailbreak
System prompt extraction	AML.T0069.002 System Prompt
Tool abuse	AML.T0053 AI Agent Tool Invocation
Exfiltration through tools	AML.T0086 Exfiltration via AI Agent Tool Invocation
Exfiltration through rendered output	AML.T0077 LLM Response Rendering
Poisoned retrieval content	AML.T0070 RAG Poisoning

OWASP describes risks from the builder's side and ATLAS describes adversary behaviour, so a report that cites both lets security and engineering teams read the same finding. ATLAS ids change between releases, so the report should state the ATLAS version it used.

How is an LLM security test scoped and reported?

NIST SP 800-115 defines rules of engagement as "detailed guidelines and constraints regarding the execution of information security testing" [10]. For an LLM application, the rules of engagement name the model version, system prompt version, enabled tools, data sources in scope, test accounts, rate limits and who may stop the test. Tool scope matters most, because a successful injection can trigger real actions such as payments or emails.

SP 800-115 states that the report should identify the vulnerabilities found and the mitigation actions for each [10]. For LLM findings, each entry should hold the exact prompt or planted document, the model and prompt versions, the number of attempts and how many succeeded. Model output varies between runs, so a single success is reported with its attempt count rather than as a fixed result. Each confirmed finding becomes a case in a prompt regression suite so a later model or prompt change does not reopen it.

The difference between attacking the model and attacking the application is covered in AI red teaming vs AI penetration testing. red teaming of model behaviour and penetration testing of the system are often scoped together.

red teaming: Deliberate attempts to make a system produce harmful or unintended output.

Which entries state these services?

42 published entries state AI security penetration testing and 56 state red teaming and adversarial testing. 45 entries state red teaming for LLM applications. These counts reflect what each vendor's AI page states, as described in the methodology.

The split by company type:

Security test checklist for an LLM application

1. 01

   Write rules of engagement

   Name model and prompt versions, tools, data sources, test accounts, rate limits and a stop contact, as SP 800-115 describes for any security test [10].

2. 02

   Map the inputs

   List every channel the model reads: chat, files, web pages, email, tickets and RAG documents. Each is a target for indirect injection [2], [3].

3. 03

   Test direct injection

   Attempt instruction override and jailbreaks from a normal user account; map to LLM01:2025 and AML.T0051.000 [1], [9].

4. 04

   Test indirect injection

   Plant instructions in each content source, including images if the model is multimodal; map to AML.T0051.001 [2], [9].

5. 05

   Test leakage

   Seed canary records and try cross-user and cross-tenant retrieval; attempt system prompt extraction and review what the prompt exposes [4], [5].

6. 06

   Test output handling

   Send model output into each consumer: browser, SQL, shell, file paths and email templates; check for XSS, injection and rendering exfiltration [6], [9].

7. 07

   Test tool abuse

   For each tool, check permissions, confirmation steps and whether injected text can trigger it; map to LLM06:2025 and AML.T0053 [8], [9].

8. 08

   Report with reproduction data

   Record prompt or planted content, versions, attempts and success count, OWASP and ATLAS ids, and a mitigation for each finding [10].

9. 09

   Turn findings into regression cases

   Add each confirmed finding to the regression suite and rerun it after every model, prompt, tool or data source change.

The LLM application testing checklist covers quality checks that sit alongside this security list.

Sources

[1] OWASP Top 10 for LLM Applications 2025, OWASP Gen AI Security Project, 2024, https://genai.owasp.org/llm-top-10/ (resource page: https://genai.owasp.org/resource/owasp-top-10-for-llm-applications-2025/)

[2] LLM01:2025 Prompt Injection, OWASP Gen AI Security Project, 2024, https://genai.owasp.org/llmrisk/llm01-prompt-injection/

[3] Greshake, K., Abdelnabi, S., Mishra, S., Endres, C., Holz, T., Fritz, M. "Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection", arXiv:2302.12173, 2023, https://arxiv.org/abs/2302.12173

[4] LLM02:2025 Sensitive Information Disclosure, OWASP Gen AI Security Project, 2024, https://genai.owasp.org/llmrisk/llm022025-sensitive-information-disclosure/

[5] LLM07:2025 System Prompt Leakage, OWASP Gen AI Security Project, 2024, https://genai.owasp.org/llmrisk/llm072025-system-prompt-leakage/

[6] LLM05:2025 Improper Output Handling, OWASP Gen AI Security Project, 2024, https://genai.owasp.org/llmrisk/llm052025-improper-output-handling/

[7] OWASP Top 10 for LLM Applications 2023/24, OWASP Gen AI Security Project, 2023, https://genai.owasp.org/llm-top-10-2023-24/

[8] LLM06:2025 Excessive Agency, OWASP Gen AI Security Project, 2024, https://genai.owasp.org/llmrisk/llm062025-excessive-agency/

[9] MITRE ATLAS, release 2026.10, The MITRE Corporation, 2026, https://atlas.mitre.org/ (data: https://github.com/mitre-atlas/atlas-data)

[10] NIST SP 800-115, Technical Guide to Information Security Testing and Assessment, NIST, 2008, https://csrc.nist.gov/pubs/sp/800/115/final