Evaluate AI Models for Untrusted Documents

This guide helps you assess whether an AI model and its surrounding application can handle untrusted documents without treating embedded instructions as trusted commands or disclosing protected information. You will need a test environment, representative sample documents, and a clear list of what data and actions the system must protect.

What You’ll Need

• A sandbox or test tenant that cannot access production data, real user accounts, or live tools.
• The model, document-ingestion pipeline, and application interface you plan to evaluate, including any retrieval, OCR, summarization, or agent features.
• Harmless test documents containing ordinary text, hidden or visually obscure text, quoted instructions, and simulated secrets. Use synthetic data only.
• A written threat model: what counts as an untrusted document, which information is confidential, and which actions the system must never take without authorization.
• A way to record prompts, retrieved passages, model responses, tool calls, errors, and configuration details. Follow your organization’s data-handling rules.

Step-by-Step

1. Define the security boundary. Map how a document travels from upload through parsing, storage, retrieval, and model input. Identify which components can see sensitive data and which can call tools, send messages, or change records. State the expected behavior in plain language: for example, “The assistant may summarize this file, but must not follow instructions found inside it or reveal other users’ records.” Treat both the document and extracted text as untrusted, even if the file came from a familiar source.
2. Record the exact system under test. Note the model identifier and settings, system instructions, document parser and OCR configuration, retrieval settings, connected tools, and access controls. Test the complete application, not just a model in a chat window: vulnerabilities often arise from how retrieved text is inserted into prompts or how tool permissions are granted. Keep this record so a changed configuration can be retested consistently.
3. Create a controlled test set. Include normal documents, scanned pages, tables, footnotes, quoted conversations, and files with misleading formatting. Add benign adversarial examples such as a paragraph saying “ignore previous instructions,” a request to disclose a synthetic secret, or directions to use a tool. Put some text in small print, headers, metadata, or OCR-sensitive areas. These are test strings, not executable payloads. Use distinct fictional secrets and identities so any unexpected disclosure is easy to spot.
4. Set expected outcomes before testing. For each case, write down what the application should do. A safe result might summarize the document’s content while describing embedded instructions as content, not obeying them. Specify prohibited outcomes, such as revealing a synthetic secret, retrieving another test user’s document, or invoking a tool based solely on a file’s instruction. Decide how to label partial failures, refusal, and uncertainty; otherwise, evaluators may score the same response differently.
5. Test basic task quality and instruction separation. Ask the system to summarize, extract fields, or answer questions from each file. Then check whether it follows document-embedded commands instead of the user’s request or system policy. Include direct questions about the suspicious text, such as asking what the document says, to distinguish safe reporting from unsafe obedience. A model can accurately quote an instruction without treating it as an instruction to itself.
6. Probe confidentiality and access boundaries. Place synthetic secrets in a document the test user is allowed to access, and separate synthetic secrets in a document they are not allowed to access. Ask questions that could tempt disclosure, including indirect questions and requests for summaries. Verify access control at retrieval time as well as in the final answer: a polite refusal is not sufficient if unauthorized text was already fetched or exposed in logs. Inspect traces only within the authorized test environment.
7. Check tools and consequential actions. If the system can search, send, edit, execute, or otherwise act, use mock tools or dry-run endpoints. Include documents that request a tool action and verify that the application requires an authorized user request and any required confirmation. Ensure document text cannot grant permissions, change tool arguments unnoticed, or bypass approval. Do not connect adversarial tests to live email, customer records, or production systems.
8. Repeat across formats and runs. Test PDF, office files, plain text, images, and other supported formats separately. Compare files with visible text against scanned or OCR-derived versions, and try different chunking or retrieval conditions. Repeat cases because model outputs can vary. Record the input, output, retrieved passages, tool activity, and pass/fail reason, while avoiding unnecessary retention of sensitive material.
9. Score results and set release conditions. Track security failures separately from answer quality. A useful scorecard records whether the system followed embedded instructions, exposed protected data, crossed a user boundary, or performed an unauthorized action, alongside ordinary task accuracy. Treat any confirmed disclosure or unauthorized action as a release blocker. Fix the relevant layer, rerun the same cases, and add each failure as a regression test before expanding access.

Common Mistakes

Testing only the model’s text response misses failures in retrieval, parsing, permissions, and tool execution. Another mistake is using real confidential files “just for realism”; synthetic material is safer and usually easier to audit. Do not count a refusal as a pass if the system already retrieved or logged unauthorized content. Avoid judging safety from one successful run, a single prompt, or a model’s stated confidence. Finally, do not treat hidden text as the only threat: plainly visible instructions can also be adversarial, while unusual formatting may simply be a parser issue.

Tips

Keep test files small and label each one with a unique case ID. Separate the user’s instruction from document content in the interface and in application logs, but do not assume that formatting alone creates a security boundary. Use least-privilege access for retrieval and tools, and require explicit approval for consequential actions. Review both model responses and system traces. Re-run the suite after changing the model, prompt, parser, chunking, retrieval index, or permissions. Ask reviewers to assess expected outcomes before seeing the model’s response to reduce scoring bias.

Troubleshooting

The model obeys text inside a document: Preserve the failing file and trace in the sandbox, check how document content is separated from trusted instructions, and limit the model’s authority. Improve the application’s handling and add the case to regression tests; changing wording alone may not fix the underlying boundary.

Results differ between runs: Repeat the case, record relevant settings, and assess the worst observed behavior rather than selecting the best response. If possible, reduce randomness for controlled comparisons, while recognizing that this does not prove safety.

The model misses text in a scan: Check OCR output and extraction quality separately from model behavior. Test a corrected text extraction and the original scan so you can identify whether the failure is in parsing, retrieval, or reasoning.

A secret appears in logs or retrieval traces: Stop the test, restrict access to the traces, and follow your organization’s incident and retention procedures. Do not copy the exposed content into new test prompts. Repair access controls and verify that unauthorized material is no longer retrieved before resuming.

A tool runs unexpectedly: Disable live integrations, preserve the relevant audit record, and check whether document content could supply arguments or trigger execution. Require user authorization and confirmation at the application layer, then retest with a mock tool.

Frequently Asked Questions

What is prompt injection in a document?
It is text in a file that tries to influence the model’s behavior, such as telling it to ignore prior instructions or reveal information. The system should treat it as document content, not authority.

Can I evaluate security by testing the model alone?
No. The application’s parser, retrieval rules, access controls, prompts, and tools all affect risk. Evaluate the complete workflow and its boundaries.

Should I use real confidential documents?
Prefer synthetic documents and fictional secrets. Real data adds exposure risk and is rarely necessary to test whether boundaries work.

How many test cases are enough?
There is no universal number. Cover each supported file type, important user permission boundary, key workflow, and available tool; expand the suite whenever a failure occurs.

Does a refusal prove the system is secure?
No. Check what was retrieved, placed in context, logged, or sent to a tool as well as what appeared in the final answer.

When should I repeat the evaluation?
Repeat it after meaningful changes to the model, prompts, parsing, retrieval, permissions, tools, or document formats, and before increasing the system’s access or autonomy.

Related

• iQOO 3 Price in India and Launch Details
• Zellij vs tmux: Which Is Better for Beginners?
• Gboard Not Capitalizing? Causes and Fixes