Measure Prompt Drift With a Small Python CLI

A small Python command-line tool can measure how much a prompt’s output changes across repeated runs or after a prompt edit. You provide a baseline prompt, a comparison prompt, and a way to generate text; the CLI saves the outputs and reports a similarity score you can track over time. This is a practical signal, not proof that two responses mean the same thing.

What You’ll Need

Python 3.9 or later and a terminal.
A text-generation function or API client that you are permitted to use. Keep any API key in an environment variable, not in the script.
A baseline prompt and a revised prompt, or a saved baseline output.
A repeatable test set: ideally several representative inputs, including ordinary and edge cases.
Optional: the scikit-learn package for TF-IDF cosine similarity. The example below uses it.

Step-by-Step

Decide what “drift” means for your use case. Output drift is a change in generated text for the same input. Prompt drift can also mean a change in the prompt itself, or a gradual change in behavior after model, settings, or data updates. This guide measures output drift by comparing a fixed baseline response with a current response. Keep the model, generation settings, and test inputs unchanged when you want to isolate the prompt change.
Create a project and install the dependency. Make a folder, open a terminal there, and create a virtual environment: python -m venv .venv. Activate it using the command for your operating system, then run python -m pip install scikit-learn. Save the following as drift.py. It expects two text files containing outputs; separating generation from measurement keeps the tool usable with local models, APIs, or manually saved results.

Add a small comparison CLI. Use this code:

import argparse
from pathlib import Path
import re
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.metrics.pairwise import cosine_similarity


def normalize(text):
    return re.sub(r"\s+", " ", text.casefold()).strip()


def main():
    parser = argparse.ArgumentParser(
        description="Compare two generated text outputs."
    )
    parser.add_argument("baseline", type=Path)
    parser.add_argument("current", type=Path)
    args = parser.parse_args()

    baseline = normalize(args.baseline.read_text(encoding="utf-8"))
    current = normalize(args.current.read_text(encoding="utf-8"))
    if not baseline or not current:
        parser.error("Both files must contain non-whitespace text.")

    matrix = TfidfVectorizer().fit_transform([baseline, current])
    score = cosine_similarity(matrix[0:1], matrix[1:2])[0, 0]
    print(f"Similarity: {score:.3f}")
    print(f"Drift: {1 - score:.3f}")


if __name__ == "__main__":
    main()

The script normalizes capitalization and repeated whitespace, then calculates cosine similarity between TF-IDF representations. Similarity ranges from 0 to 1 for these nonempty texts; the displayed drift is one minus similarity. This is a lexical comparison, not a semantic or quality judgment.

Prepare the baseline and current outputs. Save one response per file, such as baseline.txt and current.txt, encoded as UTF-8. Generate both using the same test input. If comparing prompt versions, keep the model identifier, system instructions, temperature, token limit, tools, and other generation parameters fixed where possible. If comparing model updates, keep the prompt and settings fixed instead.
Run the CLI and inspect the result. From the project folder, run python drift.py baseline.txt current.txt. A result such as Similarity: 0.842 and Drift: 0.158 means the texts have some lexical difference under this method; it does not mean 15.8% of the response is wrong. Review the actual output, especially required facts, formatting, refusals, and edge-case behavior.
Repeat on a test set and record results. Run the comparison for multiple representative inputs and store each score alongside the date, prompt revision, model, settings, and test case. Summarize the scores consistently—for example, report the median and the lowest similarity—rather than selecting only a favorable example. Establish a baseline range before setting an alert threshold; a single arbitrary cutoff is rarely meaningful.
Compare prompts directly when needed. To measure wording changes, save the prompt texts themselves as the two input files and run the same command. Treat that as a separate measurement from output drift: a large wording difference can yield similar behavior, and a tiny edit can cause a significant output change. For output drift, compare generated responses, not just prompt files.

Common Mistakes

Comparing different inputs, model versions, or sampling settings at once makes it hard to tell what caused a score change. Fix these variables or record them explicitly. Another mistake is treating cosine similarity as a correctness score: it rewards shared vocabulary and may rate a valid paraphrase as different, while overlooking a changed number or negation in otherwise similar text. Always inspect high-impact changes manually or with domain-specific checks.

Do not compare only one response if generation is stochastic. Repeat runs when appropriate and compare distributions or ranges. Keep test cases representative, but avoid putting secrets or personal data into saved files or logs. For API use, follow the provider’s data handling terms.

Tips

Version prompts in source control and give each test case a stable ID. Save raw outputs as well as normalized inputs, so you can reproduce and audit comparisons. Keep normalization conservative: case-folding and whitespace cleanup are usually easy to explain, while removing punctuation or numbers may hide important changes.

Use a threshold only after observing normal variation across repeated baseline runs. Consider adding exact checks for required phrases, JSON validity, key fields, or numerical tolerances. Those checks answer specific questions that a general similarity score cannot. If you need stronger semantic comparison, evaluate a suitable embedding-based metric or human judgments, and document its limitations too.

Troubleshooting

“No module named sklearn”: activate the project’s virtual environment and install the package with python -m pip install scikit-learn. Ensure the python command running the script is the same environment where you installed it.

The score is always 1: the normalized texts may be identical, or differ only in capitalization and whitespace. Check that the files contain the intended outputs. The score seems unexpectedly low: short texts and paraphrases can behave poorly under word-based TF-IDF; check the text directly and test several cases. It fails on an empty vocabulary: files may contain only punctuation or tokens filtered by the vectorizer. Add meaningful text, or deliberately configure a character-based analyzer for your use case. Output changes between runs: fix generation settings, record model versions, and repeat tests to estimate ordinary sampling variation.

Frequently Asked Questions

Does a higher drift score mean the newer output is worse?
No. It means the text differs more under this metric. Judge usefulness and correctness separately.

Can I measure drift without an API?
Yes. The CLI only reads text files. You can create them from any permitted model, a local tool, or saved test outputs.

Should I compare prompts or responses?
Compare prompts to track wording changes. Compare responses from the same test input to measure behavioral output drift.

Is TF-IDF cosine similarity semantic?
No. It measures similarity in weighted word usage. It can miss meaning changes and penalize valid paraphrases.

How many test cases should I use?
Use enough to cover your normal use and important edge cases. A small, varied, repeatable set is more informative than one example.

Can I automate this in continuous integration?
Yes. Run fixed tests and save scores, but fail a build only after establishing a justified threshold and adding checks for the behaviors that matter.

Method: the standard formula for this calculation, rounded as each result states. See how the calculators work or report an error.

More cards in this drawer

The whole Numbers and Powers drawer →