keepcalculator

06 Numbers and Powers

AI Agent Memory, RAG & Tools: What Goes Where?

Card No. 06·70343 · Updated · Standard formula, shown below

Memory share 25.0 %
Docs that fit in window
7 count
Result
Context window (tokens)
8000 tokens
You entered
Max memory tokens
2000 tokens
You entered
You enter
Context window (tokens), Max memory tokens, Documents to retrieve
You get
Memory share, Docs that fit in window
Rounding
1 dp · 0 dp

AI agents combine three distinct capabilities: memory (what the agent retains), retrieval-augmented generation or RAG (what the agent looks up), and tools (what the agent can do). Memory belongs in the agent’s state, RAG belongs in the retrieval pipeline, and tools belong in the execution layer. This article explains the difference, why it matters, and how to choose the right approach for your use case.

Formula

The calculator estimates how much of your context window is consumed by memory versus how many documents (each ~800 tokens) can fit alongside it. The formula is:

Memory share (%) = (Memory tokens / Context window) × 100

Docs that fit = floor((Context window − Memory tokens) / 800)

Variables:

  • Context window (tokens): total tokens the model can process at once.
  • Max memory tokens (tokens): tokens reserved for persistent memory (summaries, conversation history).
  • Docs to retrieve (count): number of documents you want to inject via RAG.

This helps you see the trade-off: memory and RAG both consume the same limited context window.

Worked Example

Using the defaults: context window = 8000 tokens, max memory = 2000 tokens, docs to retrieve = 5.

  1. Memory share = (2000 / 8000) × 100 = 25%
  2. Remaining tokens = 8000 − 2000 = 6000
  3. Docs that fit = floor(6000 / 800) = 7

So with 5 documents requested, you have room for all 5 plus 2 extra slots. The calculator shows that memory consumes a quarter of your context, leaving 6000 tokens for RAG and generation.

Background: Why the Confusion?

Modern AI agents are built on large language models (LLMs) that have a fixed context window—the number of tokens (words/subwords) they can process in a single request. When developers talk about “memory,” “RAG,” and “tools,” they are often describing different ways to work around this limit. Memory stores information across turns, RAG fetches relevant external data on demand, and tools let the agent perform actions (like searching a database or calling an API). The confusion arises because all three can appear as “context” to the model, but they serve fundamentally different roles.

How Memory Works in Agents

Memory in an AI agent is any information that persists across interactions. It can be short-term (the current conversation) or long-term (user preferences, past decisions, learned facts). Implementations include:

  • Conversation history: storing previous turns in the context window.
  • Summaries: compressing older conversation into a shorter form.
  • Vector memory: embedding past interactions and retrieving them when relevant.
  • External databases: storing structured facts (e.g., user’s name, settings) in a key-value store.

Memory is the agent’s own state. It is not fetched from an external source at query time (unless you use a vector store, but that’s a hybrid). The key point: memory is owned by the agent and updated as it interacts.

How RAG Works and When to Use It

Retrieval-augmented generation (RAG) is a technique where the agent retrieves relevant chunks of text from a knowledge base (documents, wikis, PDFs) and injects them into the prompt. The steps are:

  1. Chunk your documents into small pieces (e.g., 500-1000 tokens).
  2. Embed each chunk into a vector space.
  3. At query time, embed the user’s question and find the most similar chunks.
  4. Add those chunks to the prompt as context.

RAG is ideal when the answer is not in the model’s training data—e.g., internal company policies, recent news, or private documents. It is also useful for reducing hallucinations because the model can ground its answer in retrieved text. However, RAG does not update the agent’s memory; it is a per-query retrieval.

Tools: The Execution Layer

Tools are functions the agent can call to perform actions or get real-time data. Examples include:

  • Web search APIs
  • Database queries (SQL)
  • File system operations
  • Calculators
  • Email sending
  • Calendar management

Tools are typically defined with a name, description, and input schema. The LLM decides when to call a tool based on the user’s request. The tool executes and returns a result, which is then fed back into the context. Tools are not memory; they are capabilities. They can be used to fetch data (like RAG) but they are action-oriented—they change the world or retrieve live data.

Choosing What Goes Where

The decision of whether to use memory, RAG, or a tool depends on the nature of the information:

  • Use memory for facts about the user or the conversation that persist across turns (e.g., “the user prefers metric units”).
  • Use RAG for static knowledge that is too large to fit in context (e.g., a 10,000-page manual).
  • Use tools for dynamic data or actions (e.g., checking the weather, sending an email).

A common pattern is to combine all three: the agent remembers user preferences (memory), retrieves relevant documentation (RAG), and calls a calculator tool to compute a result. The key is to avoid redundancy—don’t store in memory what you can retrieve via RAG, and don’t use RAG for data that is better fetched via an API.

Common Problems and Misconceptions

One frequent mistake is treating RAG as memory. If you retrieve the same document every time, you are not learning from past interactions. Another issue is context overflow: stuffing too much memory and RAG chunks into the prompt can exceed the window, causing errors or degraded performance. A third problem is stale memory: if the agent stores facts that change (e.g., a user’s location), it may give outdated answers. Finally, tools can be overused—calling a tool when the model already knows the answer wastes time and tokens.

Costs and Trade-offs

Each approach has costs:

  • Memory: consumes context tokens every turn; long-term memory requires storage and retrieval infrastructure.
  • RAG: requires embedding and vector search, which have compute and storage costs; retrieval latency adds to response time.
  • Tools: API calls may have per-use costs; latency increases with each tool call.

Balancing these costs is an engineering decision. For high-volume applications, you might minimize memory by summarizing aggressively, use RAG only when needed, and cache tool results.

Frequently Asked Questions

Can RAG replace memory?

No. RAG retrieves external information on demand, but it does not retain anything about the user or past interactions. Memory is essential for continuity. You can use a vector store as both memory and RAG source, but they serve different purposes: memory is about the conversation, RAG is about knowledge.

Do tools count as memory?

No. Tools are actions. However, the results of tool calls can be stored in memory (e.g., saving the result of a database query for later). The tool itself is not memory.

How do I decide between RAG and a database tool?

If you need to search unstructured text (documents, emails), RAG is better. If you need structured data with exact queries (e.g., “get user by ID”), a database tool is more precise and faster. For hybrid cases, you can use RAG to find relevant documents and then a tool to query a structured database for specifics.

What is the best way to manage memory in a long conversation?

Use a combination: keep the last few turns in full, summarize older turns, and store key facts in a structured memory (e.g., JSON). Some frameworks use a “memory bank” that is periodically updated. The goal is to keep the context window under control while retaining essential information.

Can I use RAG with tools together?

Yes. A common pattern is: RAG retrieves relevant documents, then a tool extracts specific data from those documents (e.g., a PDF parser), or the agent uses a tool to fetch live data that complements the retrieved knowledge.

What is the difference between RAG and fine-tuning?

Fine-tuning changes the model’s weights to learn new patterns, while RAG does not change the model—it just adds context to the prompt. RAG is cheaper and easier to update, but fine-tuning can make the model more specialized. They can be complementary: fine-tune for style, RAG for facts.

Related

Formula used
in t | Context window (tokens) | tokens | 8000 | 100 | 1000000
in m | Max memory tokens | tokens | 2000 | 0 | 1000000
in d | Documents to retrieve | count | 5 | 1 | 100
out ratio | Memory share | % | (m / t) * 100 | 1
out docs | Docs that fit in window | count | floor((t - m) / 800) | 0

Each line is one input (in, sel, list) or one result (out: label, unit, formula, decimals). How the calculators work

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →