What Happens When an LLM Generates a Single Token?

When a large language model generates one token, it processes the context so far, calculates a score for each possible next token, turns those scores into probabilities, and selects one token. That token is added to the context, and the process repeats; a token may be a whole word, part of a word, punctuation, or another text unit.

What a token is

A token is a unit in the model’s vocabulary, not necessarily a word or a character. Before text is processed, a tokenizer converts it into token IDs—numbers that identify entries in that vocabulary. The exact division depends on the tokenizer, so the same word can be represented differently by different models.

For example, a familiar word may be one token, while a rare or long word may be split into several pieces. Spaces and punctuation can be included in tokens too. A token ID is not itself a piece of readable text; it is a reference the model uses internally. When generating, the model eventually maps the selected ID back to a visible text fragment.

This is why “the model writes a word at a time” is an unreliable description. It predicts one vocabulary item at a time. The item may look like a word, but it could also be a suffix or a space followed by a word.

How the model uses the existing context

The model starts with a sequence of tokens representing the prompt and, if applicable, earlier generated output. Each token is converted into a learned numerical representation, with information about its position in the sequence. A transformer model uses attention and other neural-network operations to compute representations influenced by the surrounding context.

In practical terms, the model’s layers transform the context into information useful for predicting what might come next. Attention lets positions in the sequence weigh information from other positions; it does not mean the model consciously reads or understands text. The result is a final representation used to produce scores for possible next tokens.

For a prompt such as “The capital of France is,” the context makes some continuations more likely than others. The model does not simply look up a fixed answer in a table. Its output reflects patterns learned during training, the current prompt, and any generation settings. Another prompt or a different model can lead to a different distribution of likely continuations.

From scores to a selected token

The output layer produces a logit—a raw numerical score—for each token in the model’s vocabulary. These scores are not probabilities: they can be positive or negative and do not have to add up to one. A softmax transformation converts them into a probability distribution, where each possible token has a value between zero and one and all values sum to one.

Selection is a separate step. A system can choose the token with the highest probability, often called greedy decoding, or sample from the distribution so that lower-probability choices sometimes appear. Sampling can be adjusted by settings such as temperature. Lower temperature generally concentrates probability on more likely choices; higher temperature generally spreads it out. The precise effect depends on the decoding method and implementation.

Other controls can alter the selection process, including top-k or top-p sampling, which restrict the candidates considered. These methods affect which token is selected from the model’s scores; they do not change the basic fact that the model predicts a distribution over next tokens. Different services may use different settings or additional constraints.

Why one token can take computation

Generating a token involves more than choosing a word from a list. The model must run neural-network calculations over the available context and produce scores across its vocabulary. The cost depends on factors including model size, context length, hardware, and implementation. It is not accurate to assume that every token takes the same time or uses the same amount of computation.

Many systems use a key-value cache during generation. Rather than recomputing all attention information for earlier tokens from scratch at every step, the system can retain certain intermediate values and reuse them. This can make continued generation more efficient, although the cache itself requires memory and grows with the sequence.

There is also a distinction between processing the prompt and generating new tokens. Processing the supplied context is often called prefill; producing output one token after another is decoding. A service may process many prompt tokens together, but the next-token choices in ordinary autoregressive generation are sequential: the model needs the newly selected token as part of the context for the following step.

How text appears, and why output can vary

After a token is selected, its ID is converted back to its corresponding text fragment and appended to the output. The next prediction then uses the updated sequence. The process continues until a stopping condition is reached, such as an end-of-sequence token, a configured output limit, or a system-specific stop rule.

Because tokens can be partial words, a displayed stream may briefly end in what looks like an unfinished word. The next token can complete it. Some interfaces also buffer or assemble fragments before showing them, so the visible arrival of text need not line up neatly with individual model tokens.

Repeated runs can differ if token selection uses randomness, or if the prompt, settings, model version, or surrounding system instructions differ. Even when the same choice is made, one early token can change the context for every later prediction. A fluent continuation is therefore not a guarantee that each statement is correct: token prediction describes how text is generated, not how facts are verified.

Common misconceptions and practical implications

• “One token means one word.” Not necessarily. It can be a word, word fragment, punctuation mark, or other tokenizer unit.
• “The model picks the most likely next word every time.” Some decoding methods do; sampling methods can select another candidate.
• “A probability is the model’s confidence that a statement is true.” It is a score for a possible continuation under a particular context and setup, not a fact-check.
• “The model plans a whole answer before writing.” In ordinary autoregressive generation, each next-token decision is made from the context available at that step. Some systems may add planning or other processing around this process, but that is not what the basic next-token operation means.

Tokenization also matters for practical limits. Context windows and output limits are commonly counted in tokens rather than words, and token counts vary by language, text, and tokenizer. A short-looking string can use several tokens, while a longer familiar phrase may tokenize efficiently. For exact counts, use the tokenizer associated with the model rather than estimating from word count.

Frequently Asked Questions

Does an LLM generate one token at a time?
In standard autoregressive generation, it predicts and selects the next token, appends it to the sequence, and repeats. A user interface may display several tokens together.

How many characters are in one token?
There is no fixed number. A token can represent a single character, part of a word, a whole word, or multiple characters. The tokenizer and the text determine the division.

Is the highest-probability token always chosen?
No. Greedy decoding chooses the highest-scoring candidate, but sampling methods can select another token according to the probability distribution and configured controls.

Why can the same prompt produce different answers?
Sampling can introduce variation, and results can also change with the model, prompt context, system instructions, or decoding settings. A different early token changes the context for later ones.

Does a token probability show whether an answer is true?
No. It indicates how likely a token is as a continuation in that context under the model and decoding setup. It does not independently establish that the resulting claim is accurate.

Related

• Visio Alternatives for Mac: What to Use Instead
• Bluehost Privacy vs Gemini Advanced: Which Fits You?
• How to Let an AI Agent Use Websites Safely