This guide shows you how to keep retrieved RAG (Retrieval-Augmented Generation) chunks in the conversation context when using Anthropic’s Claude models with Spring AI. You’ll learn how to structure your prompts and manage conversation history so the model can reference the source material across multiple turns, leading to more accurate and consistent answers.
What You’ll Need
- Java 17 or later
- Spring Boot 3.2+ project with Spring AI dependency
- An Anthropic API key
- Basic familiarity with Spring AI’s
ChatClientand vector stores
Step-by-Step
- Add Spring AI dependencies – Include
spring-ai-starter-model-anthropicandspring-ai-starter-vector-store(e.g., for PGVector or Redis) in yourpom.xml. Configure your Anthropic API key inapplication.ymlunderspring.ai.anthropic.api-key. - Set up a vector store – Instantiate a
VectorStorebean (e.g.,PgVectorStore) and populate it with your documents, chunked appropriately. Use a chunk size that fits your use case (e.g., 500-1000 tokens). - Create a retrieval service – Build a component that, given a user query, calls
vectorStore.similaritySearch(query)to get the top-k chunks. Return these chunks as a list ofDocumentobjects. - Build a system prompt with placeholders – In your prompt template, include a section for the retrieved context. For example:
"Context:\n{context}". This will hold the chunks. - Maintain conversation history – Use Spring AI’s
ChatMemory(e.g.,InMemoryChatMemory) to store previous turns. Retrieve the conversation history and pass it to theChatClientalong with the current user message. - Inject chunks into the prompt each turn – For every user request, re-run the retrieval with the current query, format the chunks into the
contextplaceholder, and combine it with the conversation history. This ensures the model always has the source material in its context window. - Use a dedicated method to build the final prompt – Write a helper that takes the user message, retrieves chunks, and constructs a
Promptobject with the system message (including context) and the user message, plus the history. Then callchatClient.call(prompt). - Test with multi-turn queries – Ask a follow-up question that references earlier information, e.g., “What did you just say about X?” Verify the model uses the chunks from the first turn to answer correctly.
Common Mistakes
- Not updating the context on each turn – If you only include chunks from the first query, follow-up questions may lack relevant information. Always re-run retrieval with the latest user message.
- Ignoring token limits – Anthropic models have a maximum context window. If you keep appending chunks and history, you may exceed it. Trim old messages or reduce chunk count.
- Putting chunks in the user message – This can confuse the model. Keep the context in the system prompt for better adherence.
- Forgetting to clear chat memory – In long sessions, memory can grow indefinitely. Implement a strategy to clear or summarize old turns.
Tips
- Use a sliding window for history – Keep only the last N turns (e.g., 5-10) to avoid bloating the context.
- Summarize chunks – If chunks are long, consider pre-summarizing them before injecting into the prompt to save tokens.
- Add a delimiter – Wrap each chunk in XML-like tags (e.g.,
<chunk>...</chunk>) to help the model distinguish between sources. - Set a low temperature – For RAG tasks, a temperature of 0.2-0.3 reduces hallucination.
Troubleshooting
- Model ignores context – Check that the context is actually in the system prompt. Log the final prompt to verify. Also, ensure the chunks are relevant; if not, adjust your retrieval (e.g., increase top-k).
- Context window exceeded – Reduce the number of chunks (e.g., top-k from 5 to 3), shorten chunk size, or trim conversation history. Monitor token usage via Anthropic’s response metadata.
- Follow-up questions fail – If the model doesn’t reference earlier chunks, ensure the conversation history is passed correctly. Test with a simple echo to confirm history is included.
- API errors – If you get 400 errors, check that your prompt doesn’t contain invalid characters or that the context isn’t empty. Also verify your API key has sufficient quota.
Frequently Asked Questions
Do I need to re-retrieve chunks for every user message?
Yes, because the user’s latest query may require different information. Re-running retrieval ensures the most relevant chunks are in context.
Can I keep chunks in context without using ChatMemory?
Yes, you can manually append previous user and assistant messages to the prompt, but ChatMemory simplifies the process.
How many chunks should I include?
Typically 3-5 chunks, depending on chunk size and model context limit. Start with 4 and adjust based on answer quality.
What if my chunks are too large for the context?
Reduce chunk size at indexing time, or summarize the chunks before injecting them into the prompt.
Does this work with other Anthropic models?
Yes, the technique is model-agnostic within Claude family. Just ensure the model supports the context length you need.
How do I clear the chat memory?
Call chatMemory.clear() or use a method to remove specific messages. You can also schedule a clear after a certain number of turns.
Related
Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.
More cards in this drawer
- Arctic Sea Ice Forecasts Disagree: What to Do
- MySQL Workbench to VPS: SSH Tunnel Setup
- Bluesky Beginners Guide: How to Get Started5.00 min/km
- LG V20 Won’t Turn On? Fixed
- Snagit vs Camtasia: Which One Do You Need?
- Scale a Small Business Without Losing Culture