Protect an AI Application From Prompt Injection

Prompt injection occurs when untrusted text tries to override an AI application’s instructions or manipulate its actions. You can reduce the risk by treating user input and retrieved content as untrusted, limiting what the model can do, and checking its outputs before they reach people or systems. Start with an inventory of your application’s model calls, data sources, tools, and sensitive actions, plus a test environment where you can safely try attacks.

What You’ll Need

• A map of the application’s prompts, models, retrieval sources, tools, and data flows.
• A development or staging environment with test accounts and non-sensitive sample data.
• Access to application configuration and logs, and a way to change permissions or disable tools.
• A small test set of ordinary requests and adversarial inputs, including indirect instructions embedded in documents.

Step-by-Step

1. Identify what an attacker could influence. Trace every place text enters the model: chat messages, uploaded files, web pages, email, database records, code comments, and search results. Mark which sources are controlled by users or third parties. Also list what the model can reveal or change, such as private records, account settings, or external services. Do not assume retrieved content is safe just because your application fetched it.
2. Separate trusted instructions from untrusted content. Keep system and developer instructions in the appropriate instruction fields supported by your model interface. Pass user messages and retrieved passages as clearly labeled data, not as text that can be mistaken for application policy. Use explicit delimiters and state that quoted or retrieved material is reference content, not instructions to follow. This helps structure the request, but delimiters alone are not a security boundary.
3. Give the model only necessary context. Retrieve the smallest relevant set of documents and remove secrets, credentials, and unrelated personal data before sending content to the model. Apply access checks before retrieval so a user cannot obtain a document merely by asking the model about it. Scope search to the signed-in user’s permissions, and avoid placing sensitive records in prompts when the task can be done without them.
4. Constrain tools and sensitive actions. Make tools unavailable unless the task requires them. Give each tool narrow permissions and validate its arguments in application code. For actions that send messages, change records, make purchases, or expose sensitive information, require an explicit user confirmation or an independent approval step. Do not let model-generated text alone authorize a high-impact action.
5. Validate model outputs before using them. Treat generated text and structured output as untrusted input. Check it against a strict schema, enforce length and type limits, escape text before displaying it, and use parameterized database queries rather than building commands from generated strings. Reject unexpected fields or values. Never execute generated code, shell commands, queries, or URLs without a separate, purpose-built validation and permission layer.
6. Enforce security in ordinary application code. Authentication, authorization, data filtering, rate limits, and business rules should be checked outside the model. Verify access for every requested object and every tool call, even if the model claims the user is authorized. Keep secrets in a secret manager or protected server configuration, not in prompts or client-side code. The model should assist the workflow, not act as its security authority.
7. Test direct and indirect injection attempts. Try prompts that ask the model to ignore prior instructions, reveal hidden prompts, or misuse tools. Also place such instructions inside a test document, web page, or email that the application retrieves. Check whether the system protects data, refuses unauthorized actions, and continues to handle legitimate requests. Include variations in wording and language; a single successful test does not establish safety.
8. Monitor, respond, and retest. Log relevant request identifiers, tool decisions, validation failures, and security events while minimizing or redacting sensitive content. Set alerts for unusual tool use, repeated refusals, or unexpected access patterns. Define how to disable a risky integration, revoke credentials, and investigate an incident. Retest after changing prompts, models, tools, retrieval logic, or permissions.

Common Mistakes

• Relying on a “do not obey malicious instructions” prompt. This can improve behavior but cannot guarantee it. Enforce permissions and safety checks in code.
• Assuming retrieved documents are trusted. A search result, shared file, or web page can contain an indirect injection. Label and constrain all retrieved content.
• Giving a general-purpose agent broad access. A model with more tools and permissions has a larger possible impact. Start with the minimum capabilities needed.
• Checking only for suspicious phrases. Attack wording changes, and benign text may contain those phrases. Use layered controls rather than keyword filters alone.
• Putting secrets in hidden prompts. Prompt text is not a vault. Keep credentials out of model context and restrict access to sensitive data independently.
• Testing only the happy path. Include indirect injection, malformed outputs, unauthorized records, and tool failures in security tests.

Tips

• Use separate model workflows for low-risk responses and actions that can change data or contact other systems.
• Keep tool descriptions precise, arguments narrow, and outputs limited to what the next step needs.
• Prefer allowlists for permitted operations and destinations; reject unknown tools, fields, and action types.
• Design graceful failure behavior. If validation or authorization fails, stop the action and offer a safe explanation rather than trying an unreviewed fallback.
• Track whether protections work with repeatable tests, but do not treat a passing test suite as proof that injection is impossible.

Troubleshooting

The model follows instructions found in a document: Check whether retrieved text is clearly separated from trusted instructions and whether the model is asked to treat it only as data. Reduce the retrieved content, review the source, and add a test case. Most importantly, ensure that the resulting tool calls and data access remain blocked unless separately authorized.

A tool runs an unauthorized action: Disable or restrict the tool while investigating. Review application-side authorization, argument validation, and confirmation requirements; do not rely on a prompt refusal. Revoke exposed credentials if needed, inspect logs, and test the corrected control before restoring access.

Valid requests are rejected: Inspect which validation rule or permission check failed. Adjust the narrowest relevant rule, not the entire security boundary, then rerun both legitimate and adversarial tests. Keep a safe refusal path when the request remains ambiguous.

Logs do not explain what happened: Add trace identifiers and record which sources were retrieved, which tool was considered, and why validation or authorization passed or failed. Avoid logging full sensitive prompts by default; use redaction and restricted access for any diagnostic capture.

Frequently Asked Questions

Can prompt injection be eliminated completely?
No. It is safer to assume attempts will occur and limit the data and actions they can affect through layered controls.

Are system prompts enough to protect an application?
No. Instructions can guide model behavior, but authorization, validation, and access control must be enforced outside the model.

Does retrieval-augmented generation prevent prompt injection?
No. Retrieved documents can carry indirect instructions. Filter access, treat passages as untrusted data, and constrain any actions based on them.

Should I block phrases such as “ignore previous instructions”?
Phrase filters can be one signal, but attackers can reword requests and legitimate content may include those words. Do not use phrase matching as the main defense.

How often should I test for prompt injection?
Test before release and whenever prompts, models, tools, data sources, or permissions change. Also rerun tests after incidents and significant security updates.

Can an AI agent use tools safely?
It can be made safer with narrow permissions, validated arguments, independent authorization, and confirmation for consequential actions. There is no tool setup that makes broad access risk-free.

Related

• Evaluate an ML Trading Strategy from Data to Execution
• How to Choose an Office Headset Mic for Calls and Comfort
• Best European VPS Hosting: 5 Providers Compared