Changed Embedding Models? What to Do With Old Vectors

When you switch to a new embedding model, your old vectors become incompatible with the new model’s vector space, so you must either re-embed your entire corpus or keep the old model running for queries against the old vectors. This article explains why that happens, what your options are, and how to handle the transition with minimal disruption.

Why Changing Models Breaks Your Vectors

Embedding models map text to high-dimensional vectors (e.g., 384, 768, or 1536 dimensions). Each model learns its own coordinate system, so the same sentence produces different vectors in different models. Even models with the same dimension count are not interchangeable because the axes and distances mean different things. Cosine similarity between vectors from different models is meaningless — a 0.8 similarity between a new-model vector and an old-model vector does not indicate semantic closeness.

This is why you cannot simply mix old and new vectors in the same index. If you do, retrieval quality collapses, and ranking results become random. The only reliable way to use a new model is to regenerate vectors for all your documents with that model.

Your Options: Re-embed, Keep Both, or Map

You have three main paths when you change embedding models:

• Re-embed everything — the most straightforward and recommended approach. Run your entire corpus through the new model and rebuild the index. This gives you a clean, consistent vector space.
• Keep both models running — maintain two separate indexes and route queries to the appropriate one based on which model was used to embed the documents. This works if you need to support old data for a while, but it doubles storage and maintenance costs.
• Map old vectors to the new space — train a linear or non-linear transformation that approximates the new model’s output. This is risky because it introduces error and is only feasible if you have a large amount of paired data (same text embedded with both models). It is rarely used in production.

How to Re-embed Your Corpus Safely

Re-embedding is a batch job, not a live operation. Here is a practical workflow:

1. Take a snapshot of your documents and their metadata.
2. Run the new embedding model over the entire snapshot in parallel batches. Use a GPU or a cloud service to speed this up.
3. Store the new vectors in a separate index or a new collection, keeping the old index untouched until you are ready to switch.
4. Validate the new index by running a few test queries and comparing results against the old index.
5. Switch your application to the new index, then delete the old index only after you are confident everything works.

If your corpus is huge (millions of documents), re-embedding can take hours or days. Plan for it during a maintenance window or use a blue-green deployment to avoid downtime.

Choosing a New Embedding Model

Before you commit to a new model, consider these factors:

• Dimension size — larger dimensions capture more nuance but cost more storage and compute. For example, a 1536-dimension model uses 4x the storage of a 384-dimension model.
• Language support — ensure the model handles your document languages well.
• Domain fit — a model trained on general web text may underperform on specialized domains like legal or medical. Test on your own data.
• Inference speed — measure how long it takes to embed a single document and how that scales.
• License and cost — some models are open-source, others are API-based. Check the terms for commercial use.

It is wise to run a small pilot: embed a few hundred representative documents with the candidate model and manually evaluate retrieval quality on a set of test queries.

Common Problems and How to Avoid Them

Here are the pitfalls people often hit when changing embedding models:

• Mixing vectors in one index — never add new-model vectors to an old index. Always create a new index.
• Forgetting to update the query encoder — your query must be embedded with the same model as the documents. If you update documents but not the query path, you get garbage.
• Not normalizing vectors — many models output vectors that are not unit length. For cosine similarity, you should normalize both document and query vectors, or the similarity scores will be biased.
• Ignoring version drift — even the same model name can change if the provider updates weights. Pin the exact model version and record it in your metadata.

Cost and Performance Trade-offs

Re-embedding has a real cost. If you are using a commercial API, you pay per token. For a corpus of 10 million documents averaging 1,000 tokens each, that is 10 billion tokens. At a typical rate of $0.02 per 1,000 tokens, that is $200,000 — a significant expense. Self-hosting an open-source model on a GPU can be cheaper but requires infrastructure and time.

Storage also grows. If you go from 384 to 1536 dimensions, your vector index size quadruples. For a 10-million-document corpus, that is 60 GB vs. 15 GB (assuming float32). You may need to upgrade your vector database or use dimensionality reduction techniques like PCA, but that adds complexity.

Alternatives to Full Re-embedding

If re-embedding is not feasible, consider these alternatives:

• Incremental re-embedding — process documents in batches over time, updating the index as you go. This spreads the load but requires a hybrid search that can handle both old and new vectors during the transition.
• Use a model that is a drop-in replacement — some models are trained to be compatible with older ones (e.g., a newer version of the same architecture). Check the model card for compatibility notes.
• Keep the old model for legacy queries — if only a small portion of your data is old, you can keep the old model running just for those documents and route queries based on document age. This is messy but avoids re-embedding everything.

Frequently Asked Questions

Can I use old vectors with a new model if I just re-normalize them?

No. Normalization only changes the magnitude, not the coordinate system. The semantic relationships are different, so even normalized old vectors will not align with new ones.

How long does re-embedding take?

It depends on your corpus size and hardware. On a modern GPU, you can embed roughly 1,000–10,000 documents per minute, depending on length and model complexity. For 1 million documents, expect anywhere from a few hours to a few days.

What if I only have a small dataset?

Then re-embedding is trivial — just do it. The cost is minimal, and you avoid all the complexity of maintaining two models.

Can I train a mapping from old to new vectors?

You can, but it is rarely worth it. You would need a large set of paired embeddings (same text through both models) to train a regression. The mapping will be approximate, and you will likely see a drop in retrieval quality compared to re-embedding.

Should I keep the old index for backup?

Yes, keep it until you have validated the new index thoroughly. Once you are confident, you can delete it to save storage.

What if my vector database doesn’t support multiple indexes?

Most vector databases (e.g., Pinecone, Weaviate, Qdrant) support multiple indexes or namespaces. If yours does not, you can store the model version as a metadata field and filter on it during search, but that is inefficient. Better to migrate to a database that supports separate indexes.

Related

• Water Heater Operating Cost Calculator and Formula
• Water Heater Load Calculation: Estimate Energy Use
• Water Heater ROI Calculator: Estimate Payback and Savings