How Do You Pen Test an AI Application? Understanding LLM Security Testing

October 15, 2026
Digital verification interface with checkmark, analytics, settings, and compliance icons for cybersecurity management.

An LLM application can appear secure during normal use while responding very differently when someone deliberately tries to manipulate it. A user may ask an AI assistant to summarize a document, retrieve information, answer questions, or interact with an internal system. During LLM security testing, those same functions become targets for controlled adversarial testing. The objective is to determine how the model and the application respond when an attacker attempts to override instructions, expose protected information, manipulate retrieved content, bypass safeguards, or misuse connected tools. Effective LLM security testing examines the model's behavior alongside the application controls that determine what an attacker can ultimately access or execute.

Testing How Attackers Can Manipulate LLM Behavior

The first step in testing an LLM application is understanding how the system is built. A tester needs to know which model the application uses, how prompts are constructed, what information enters the model's context, and how the application processes the model's response. The assessment should also identify external data sources, retrieval systems, APIs, databases, plugins, and tools connected to the model. This architecture determines the application's attack surface. An AI chatbot that only generates text has a different security profile from an AI agent that can retrieve customer records, modify data, send messages, or call external services. The more capabilities connected to the model, the more important it becomes to test the boundaries between model behavior and application permissions.

Prompt injection is one of the primary areas examined during LLM security testing. The attack involves supplying instructions intended to influence the model in a way that conflicts with the application's intended behavior. A tester may attempt to make the model disregard its original instructions, follow attacker-controlled commands, disclose information, or produce output that triggers unintended behavior. Testing should use different forms of input instead of relying on a single known prompt. Attackers can change wording, introduce instructions gradually, combine malicious instructions with legitimate requests, or place them inside content the application retrieves. OWASP identifies prompt injection as one of the leading security risks for LLM applications.

Direct prompt injection originates from information supplied by the user. Indirect prompt injection creates another attack path because the attacker may not need direct control over the application's primary input. An LLM can process emails, documents, websites, knowledge-base articles, or other external content while completing a task. If an attacker places malicious instructions within one of those sources, the model may process them as part of its context. During an LLM security assessment, testers can introduce controlled instructions into content the application is expected to retrieve and observe how that content affects the model. This helps determine if untrusted information can influence trusted application behavior. NIST's adversarial machine learning taxonomy recognizes indirect prompt injection as an attack technique relevant to generative AI systems.

Testing prompt injection also requires looking beyond the model's immediate response. A model refusing an attack does not automatically establish that the application is secure. The tester needs to examine the complete request and response chain. If the model connects to a retrieval system, the assessment can determine if an injection changes which documents are returned. If the model can call tools, testing can determine if manipulated instructions cause functions to execute. If generated content is passed to another application component, the tester can examine whether that output affects downstream behavior. The security impact depends on what an attacker can accomplish through the weakness.

Jailbreak testing is another part of LLM security testing. A jailbreak attempts to bypass restrictions placed on the model or application. Testers can evaluate how consistently the application's safeguards respond to adversarial requests and changes in conversation context. The objective is not to compile a collection of unusual prompts. The assessment should determine if an attacker can bypass a specific control and what that bypass makes possible. This distinction helps security teams understand the difference between an unexpected model response and a vulnerability that creates a meaningful security risk.

Evaluating Data Exposure and LLM Application Controls

Sensitive information disclosure requires particular attention when an LLM has access to private data. An application may retrieve internal documents, customer information, employee records, proprietary business information, or other protected material to generate responses. LLM security testing can determine if a user can manipulate the model into returning information outside their authorization. The assessment can also examine if information belonging to one user or tenant becomes available to another. The test should follow the application's actual access model because the security impact depends on the permissions associated with the requesting user. OWASP identifies sensitive information disclosure as a major risk for LLM applications.

System prompt leakage should also be evaluated. Developers commonly use system instructions to define the model's role, behavior, limitations, and interaction with application functions. Testers can attempt to extract those instructions through conversation and other input techniques. Disclosure of a system prompt does not automatically constitute a serious vulnerability. The assessment needs to determine what the prompt contains and how its exposure affects the application's security. If it reveals sensitive information, internal implementation details, or instructions that help an attacker bypass another control, the finding can have greater significance.

LLM security testing becomes particularly important when an application uses retrieval-augmented generation. In a RAG architecture, the model receives information retrieved from a separate knowledge source and uses that information to generate a response. The security assessment should examine how documents are indexed, retrieved, filtered, and presented to the model. Testers can determine if users can retrieve documents outside their authorized scope or if information from different users or security groups becomes mixed. The assessment should also consider if malicious instructions can be introduced into documents that the model later retrieves. OWASP's LLM security guidance identifies vector and embedding weaknesses as a specific risk category associated with these systems.

Authorization becomes another critical testing area when an LLM can access tools. An AI assistant may be able to search a database, retrieve files, create records, send emails, call APIs, or perform other operations. Each capability expands the application's attack surface. Testers can determine if the model can invoke a function that the user should not be allowed to access. They can also examine if the model has broader permissions than necessary or if an attacker can manipulate the model into using a legitimate capability for an unintended purpose. Authorization should be enforced by the underlying application and APIs rather than relying on the model to make the security decision.

This is particularly relevant to excessive agency. An LLM may have the technical ability to perform an action, but that does not mean the action should be available in every situation. OWASP describes excessive agency through excessive functionality, excessive permissions, and excessive autonomy. LLM security testing can evaluate these areas by attempting to move from a conversational request to an action that affects an external system. Testers can then determine which controls prevent unauthorized activity. Limiting permissions, restricting available functions, requiring confirmation for sensitive actions, and enforcing authorization at the application layer can reduce the consequences of model manipulation.

Testing LLM Outputs, Connected Systems, and Attack Paths

Generated output is another part of the LLM attack surface. An application may display a model's response to a user, store it in a database, pass it to another service, or use it as input for another operation. If the application treats generated content as trusted without appropriate validation, manipulated output can create downstream security problems. LLM security testing should therefore follow the model's response beyond the model itself. Testers can determine how generated content is processed and identify situations where untrusted output reaches sensitive functions without sufficient safeguards. OWASP identifies improper output handling as a distinct risk for LLM applications.

Testing should also consider data and knowledge-base poisoning. An LLM application that relies on external datasets, documents, or retrieval systems can be affected by malicious content introduced into those sources. Testers can examine who can add, modify, or remove information and how changes become available to the model. In a RAG system, the assessment can determine if an attacker can introduce content that changes future responses or influences model behavior. NIST's adversarial machine learning taxonomy includes data and knowledge-base poisoning among the techniques relevant to AI systems.

Resource consumption is another area of LLM security testing. Generating responses requires computational resources, and some applications also incur costs based on model usage. An attacker who repeatedly submits large or resource-intensive requests can create operational or financial consequences when appropriate controls are absent. OWASP identifies unbounded consumption as an LLM application risk. Testing can evaluate request limits, input restrictions, rate controls, and other safeguards designed to prevent excessive use. The objective is to determine how the application responds when usage moves beyond normal patterns and if an attacker can cause disproportionate resource consumption.

A strong LLM security test also examines how multiple weaknesses can be combined. A prompt injection may have limited impact when the model only generates text. The same weakness can become substantially more serious when the model has access to confidential information or privileged tools. An attacker could potentially manipulate retrieved content, influence the model's behavior, and then attempt to make the model access a protected resource or perform an unauthorized action. Testing these sequences helps establish the actual attack path instead of evaluating each vulnerability in isolation.

The assessment should also distinguish model behavior from application security failures. Large language models can produce different responses to similar inputs, so variation alone does not establish a vulnerability. The key question is whether an attacker can use that behavior to bypass a security control, access information they should not have, influence a protected process, or cause an unauthorized action. Answering that question requires testing the LLM within the context of the complete application.

LLM security testing gives organizations a practical way to identify weaknesses in AI applications before those systems become part of critical business workflows. Prompt injection, indirect prompt injection, jailbreaks, sensitive information disclosure, system prompt leakage, retrieval weaknesses, improper output handling, excessive agency, poisoning, and unbounded consumption can create different security risks depending on the application's architecture. A useful assessment connects each finding to a concrete attack path and its potential impact rather than treating unusual model behavior as a vulnerability on its own. If your organization is developing or deploying an LLM application, Contact Alacrinet to discuss LLM security testing and evaluate your model, prompts, retrieval workflows, connected tools, and application controls before attackers can exploit them.

‍

Explore More Articles & Insights