N-Gram Language Models: How Next-Word Prediction Worked

Before neural language models, an n-gram language model predicted the next word by counting how often short sequences of words appeared in a text collection. Given the most recent words, it estimated which word was likely to follow, then selected or sampled a candidate; the method was simple, fast, and limited by how much context its counts could reliably cover.

What an n-gram model is

An n-gram is a sequence of n consecutive tokens. A unigram has one token, a bigram has two, and a trigram has three. Tokens were often words, although a system could instead use characters, punctuation-aware tokens, or other units. A language model assigns probabilities to sequences so it can compare possible continuations.

For example, in “the cat sat,” the word sequences “the cat” and “cat sat” are bigrams; “the cat sat” is a trigram. An n-gram model learns from a corpus: a large collection of example text. It does not understand that a cat is an animal or that sitting is an action. It records statistical regularities in the training text.

How next-word prediction worked

In principle, the model wants the probability of a candidate word given all the words before it. Directly estimating that probability for every possible history would require vast amounts of data. The standard n-gram approximation assumes that only the last n − 1 tokens matter. For a trigram model, the estimate of the next word depends on the previous two words.

The basic maximum-likelihood estimate is a count ratio:

P(wᵢ | wᵢ₋ₙ₊₁ … wᵢ₋₁) = C(wᵢ₋ₙ₊₁ … wᵢ) / C(wᵢ₋ₙ₊₁ … wᵢ₋₁)

Here, C is the number of times a sequence occurs in the training corpus. Suppose “in the morning” occurs 40 times, and “in the” occurs 100 times. The unsmoothed trigram estimate for “morning” after “in the” is 40/100, or 0.4. Candidates such as “house” or “room” receive probabilities from their own counts in that same context.

To produce a sentence, a system starts with a prompt and repeatedly predicts a next token. It may choose the highest-probability candidate, called greedy decoding, or randomly sample according to the probability distribution. The process continues until a stopping token or another length limit is reached.

Training data, counts, and boundaries

Training typically meant tokenizing the corpus and counting every sequence of the selected length. A trigram model, for instance, counts each adjacent three-token window. At prediction time, it looks up the context and candidate counts in those tables. These counts can be stored in databases or compact data structures, but the number of distinct sequences can still become large.

Sentence boundaries matter. Systems commonly mark the beginning and end of a sentence with special tokens, so the model can learn that certain words occur at sentence starts and when a sentence tends to end. The exact tokenization and boundary conventions affect the learned probabilities. Capitalization, punctuation, spelling variants, and whether contractions count as one token or several can all change the counts.

Corpus choice matters too. A model trained on newspaper text may predict formal wording more readily than slang; one trained on technical documents may have many specialized terms. The model reflects the frequencies and biases of its data rather than a general-purpose understanding of language.

The unseen-word problem and smoothing

A major weakness of raw counts is sparsity. Natural language can form far more word sequences than even a large corpus contains. If a context never appeared, its denominator may be zero; if a plausible continuation did not appear after a known context, the unsmoothed model assigns it probability zero. A zero probability can make a whole sentence seem impossible, even when it is perfectly grammatical.

Smoothing addresses this by reserving some probability for unseen or rare sequences. Add-one smoothing, for example, adds one to each count, though it is often too crude for realistic language. Other methods discount observed counts and distribute the leftover probability among less frequent possibilities. Interpolation combines estimates from several context lengths:

• A trigram estimate uses the previous two words.
• A bigram estimate uses only the previous word.
• A unigram estimate uses overall word frequency.

If the trigram is rare or missing, a smoothed or interpolated model can rely more on the shorter-context estimates. Backoff methods similarly fall back to shorter sequences when a longer one has insufficient evidence. These techniques make predictions less brittle, but they do not make the model understand meaning.

Strengths, limits, and comparison with neural models

N-gram models were useful because they were conceptually clear, relatively inexpensive to train, and easy to inspect: a developer could examine the counts behind a prediction. They worked well for tasks where local word patterns were informative, including older speech recognition and predictive text systems. Their probabilities also provided a practical way to rank candidate sentences.

The central limitation is short, fixed context. A trigram cannot directly use a subject mentioned ten words earlier to determine verb agreement. Increasing n can capture longer phrases, but it also creates more possible sequences, most of which are rare or absent. Smoothing helps with missing counts, yet cannot efficiently represent broad relationships or meaning.

Neural language models instead learn numerical representations of words and contexts, allowing them to generalize across some sequences that did not appear verbatim in training. Later neural architectures can use much longer contexts and capture richer patterns. This does not mean n-gram models vanished: they remain useful for teaching probability, lightweight baselines, and some specialized applications. The distinction is not “counts versus no data”; neural models also learn from data, but encode patterns in learned parameters rather than relying mainly on explicit sequence-count tables.

Frequently Asked Questions

Does an n-gram model predict a whole sentence at once?
Usually it generates one token at a time. At each step it estimates the next-token probabilities from the available recent context, chooses or samples a token, and then uses that token as part of the next context.

What does the “n” in n-gram mean?
It is the number of tokens in each counted sequence. A bigram has two tokens and conditions a next-word estimate on one prior token; a trigram has three and uses two prior tokens.

Why can’t a model just use a very large n?
Larger n captures more context in principle, but the number of possible sequences grows quickly. Many sequences will have few or no examples, making estimates unreliable and increasing storage and data requirements.

What is the difference between smoothing and backoff?
Smoothing adjusts probability estimates so rare or unseen sequences do not automatically receive zero probability. Backoff uses a shorter context when a longer one is unavailable or unreliable; systems often combine backoff with smoothing.

Were n-gram models the same as autocomplete?
No. An n-gram model is one possible method for estimating likely continuations. Autocomplete is a user-facing feature that can use language models along with interface rules, dictionaries, or other ranking methods.

Related

• Best OneDrive Alternatives for Older iPhones
• Why Is Jev So Fast Compared to Traditional LLMs?
• Apple Intelligence and Google Gemini on iPhone