Compare Token Costs Across JSON, CSV, YAML and More

Comparing token costs across JSON, CSV, YAML, XML, and plain text shows how much each representation contributes to a model’s input or output usage. You’ll need the actual content, a tokenizer that matches the model you plan to use, and a consistent way to compare equivalent data; the result is a token count, not a universal price.

What You’ll Need

• The same underlying data expressed in each format, with equivalent values and records.
• The target model name or tokenizer documentation, if available. Tokenization varies by model and model family.
• A tokenizer tool or API that supports the target model, plus a text editor or script for preparing samples.
• A note-taking sheet for recording counts, file size, and any formatting or schema choices.

Step-by-Step

1. Define what “cost” means for your comparison. Start with input tokens, output tokens, or both. If you want an estimated monetary cost, you’ll also need the applicable input and output rates for the exact model and account. Rates can change, and some services price cached input or other usage differently. Token count and price are related but are not interchangeable.
2. Choose the model before counting. Find the tokenizer associated with the model you will actually use. Different models may split the same string differently, and a general-purpose tokenizer may not match the service’s reported usage. If the model’s tokenizer is not available, treat your result as an estimate and verify it against usage reported by a real request, where possible.
3. Create one canonical dataset. Write down the records and values once, then represent those exact values in JSON, CSV, YAML, XML, and any other formats you want to test. Include the same fields, number of records, units, and text in every version. Do not compare a compact CSV containing only two columns with verbose JSON containing extra descriptions; that measures different information, not just format overhead.
4. Decide which representation the model will receive. Count the exact text that would be placed in the prompt, including labels, instructions, code fences, headings, and surrounding prose if those are part of the real request. If a file is transformed before prompting—for example, parsed and converted into a summary—measure the transformed text rather than the original file. Keep system and user prompt content consistent across trials.
5. Prepare valid, consistently formatted samples. Use the same newline convention and encoding, normally UTF-8. For JSON, decide whether to use compact output or indentation; for CSV, keep the same header and quoting rules; for YAML, keep indentation and key order consistent where practical. Avoid adding comments or explanatory text to only one format. Save a copy of each exact sample so you can reproduce the count.
6. Count tokens with the selected tokenizer. Paste or submit each sample to a tokenizer tool compatible with your target model. Record the count and the tool or tokenizer name. If using a programmatic tokenizer, make sure you count the final prompt string—not just an individual data fragment—and account for any service-specific message framing that the tool may not include. API usage reports are the best check for actual billed usage.
7. Run both a controlled test and a realistic test. First compare the formats using a small, fixed dataset to isolate representation overhead. Then test representative real data, including long text, missing values, unusual characters, and the largest records you expect. Token efficiency can change with the content: repeated keys, punctuation, numbers, and natural-language values may tokenize differently.
8. Calculate the difference and relative change. For each format, subtract the reference count from its count to get the token difference. To express the change as a percentage, use (format count − reference count) ÷ reference count × 100. A negative result means fewer tokens than the reference; a positive result means more. If the reference count is zero, a percentage comparison is undefined, so report raw counts instead.
9. Translate counts into an estimate only when useful. For an approximate input charge, multiply input tokens by the current input price per token; do the equivalent separately for output tokens. If prices are listed per million tokens, divide the token count by 1,000,000 before multiplying. Keep input and output separate, and label the result as an estimate because actual billing may include other rules or usage categories.
10. Choose based on more than the smallest count. Check that the model can reliably interpret the format, that your application can produce and validate it, and that the format preserves the needed structure. A small token saving may not justify fragile parsing or ambiguity. Store the sample, tokenizer details, counts, and date so future comparisons are meaningful.

Common Mistakes

• Assuming characters equal tokens: They do not. A token can be a word, part of a word, punctuation, or whitespace, and token boundaries depend on the tokenizer.
• Using the wrong tokenizer: A count from a different model can be useful for rough comparison, but it is not an exact count for the intended model.
• Comparing unequal content: Extra fields, omitted values, different labels, or different instructions undermine the result.
• Ignoring formatting: Pretty-printed JSON, CSV headers, YAML indentation, XML tags, and Markdown fences all add text that may be counted.
• Calling a token count a bill: Pricing may distinguish input from output and may use additional categories. Use current model-specific pricing and actual usage reports for confirmation.
• Overgeneralizing from one sample: A format that wins on a tiny dataset may not win for long text, repeated records, or different field names.

Tips

• Report both raw token counts and tokens per record. The latter can make datasets of different sizes easier to compare, provided each record contains equivalent information.
• Keep a “bytes” or character count alongside tokens. It helps diagnose changes, but it is not a substitute for tokenization.
• Test compact and readable versions separately. Compact JSON may save whitespace tokens, while indentation can make data easier to inspect.
• Use stable headers and field names. Renaming or repeating labels can affect counts, especially in repeated records.
• When comparing formats for model comprehension, run a separate quality test with the same question and data. A lower token count alone does not prove the model will interpret the content correctly.

Troubleshooting

Your counts differ from a provider’s usage report: Check that the model, complete prompt, message roles, tool definitions, and formatting match. The provider may count message framing or other request components that a standalone tokenizer omits. Compare one small request and inspect the service’s reported usage details.

The tokenizer tool does not list your model: Use the closest documented tokenizer only for a rough comparison, label the result clearly, and verify with the target service. Do not present the estimate as exact.

Two runs give different counts: Confirm that the exact input has not changed due to line endings, whitespace, escaping, Unicode normalization, or a template inserting timestamps or variable values. Save and compare the final strings character for character.

CSV or YAML appears unexpectedly expensive: Inspect repeated headers, quoting, indentation, long field names, and verbose values. The format name alone does not determine the count; the actual serialized text does.

The cheaper format causes parsing errors: Validate the data with a parser and test whether the model follows the structure. Restore explicit labels or clearer delimiters if needed, then recount the revised representation.

Frequently Asked Questions

Is JSON always more expensive than CSV?
No. JSON often repeats keys and punctuation, while CSV can be compact for regular tables. The result depends on the values, headers, formatting, and tokenizer.

Does YAML use fewer tokens than JSON?
Not consistently. YAML can omit some punctuation, but indentation, repeated keys, and tokenization of its text can offset that difference. Count the actual samples.

Can I compare file sizes instead of token counts?
File size is useful for storage and transfer comparisons, but it does not tell you how many tokens a model will process. Tokenize the text that enters the prompt.

Should I count the filename or file attachment?
Count whatever the model service actually includes in its prompt or usage accounting. Some file workflows transform or extract content before processing, so the original file’s text may not equal the billable input.

How can I estimate cost from a token count?
Use the current rate for the exact model and usage category. Multiply the relevant token count by the per-token rate, or convert from a per-million rate by dividing tokens by one million first.

How often should I repeat the comparison?
Repeat it when you change the model, tokenizer, serialization settings, schema, or representative data. Recheck pricing separately because rates and billing rules can change.

Related

• Prevent AI Coding Agents From Creating Duplicate Code
• CI Checks for AI-Generated Code: A Practical Setup
• Best Python Frameworks for Building Multi-Agent AI