Tokens and tokenization

The unit a language model actually reads, and why the count matters

A language model does not read words. It reads tokens. The text you type into a chat interface, paste into a prompt, or attach as a document gets converted into tokens before the model sees any of it. The output the model produces is also generated as tokens, which are then assembled back into the words you read on the screen. The conversion happens silently and almost no one is shown what it produced. The result is a small but significant gap between what you think you sent and what the model actually processed.

Closing this gap is worth a few minutes of attention because tokens are the unit that determines almost everything practical about working with these systems. The cost of a call is computed in tokens. The size of the context window is measured in tokens. The speed at which the model generates a response is measured in tokens per second. The reason a long document costs more than a short one is that it has more tokens. The reason that some languages cost more than others to process is that they tokenize less efficiently. Once you know what a token is, the practical behavior of these systems becomes much easier to predict.

This article explains what tokenization is, how it works conceptually, what tokens look like in practice, why the count varies across kinds of text, and what to do with this information when working with LLMs in real projects. No prior technical background is assumed.

What tokenization is

Tokenization is the step that converts a piece of text into a sequence of small units that the model can process. The units are called tokens. A token is sometimes a whole word, sometimes a part of a word, sometimes a single character or piece of punctuation. The exact rules are determined by a tokenizer, which is a small program that comes paired with each model.

The reason for breaking text into tokens rather than processing whole words is practical. Models work with a fixed vocabulary of possible tokens, typically somewhere between 50,000 and 200,000 entries. Languages have far more words than that, especially when you count inflected forms, technical terms, names, typos, and made-up words. Trying to give every possible word its own slot in the vocabulary would be unworkable. Tokenizers solve this problem by giving common words a single token each, while rarer words get split into smaller pieces that combine to spell them out. This way the vocabulary stays small, but every possible string of text can still be represented.

The practical effect is that common, ordinary text tokenizes very efficiently. Rare or unusual text tokenizes less efficiently. The same idea, expressed in different words, can have meaningfully different token counts.

The diagram above shows what happens when a typical English sentence is tokenized. The four-word sentence “Understanding tokenization is useful.” becomes six tokens. The word “Understanding” is common enough that the tokenizer keeps it as a single token. The word “tokenization” is less common and gets split in two: “ token” and “ization”. The word “ is” is its own token (with the leading space attached). The word “ useful” is its own token. The period is its own token. Six tokens for four words.

This pattern, where common words stay whole and rarer words split into pieces, is the heart of how tokenization works in modern LLMs. There is nothing magical about it. The tokenizer was trained on a lot of text, learned which sequences of characters appear most often, and assigned single tokens to those frequent sequences. Rarer sequences have to be assembled from smaller pieces.

Why token counts vary

The bottom of the diagram shows four short examples that all illustrate how the same kind of pattern produces different counts.

A plain English sentence like “The cat sat on the mat.” takes seven tokens. Each word is common and tokenizes cleanly. The period is one token. This is the efficient case.

A long technical word like “Antidisestablishmentarianism” takes around ten tokens by itself, because the tokenizer has to assemble it from smaller pieces. The word fits in your head as one concept, but the model is reading ten chunks. This is why technical writing in specialized fields tends to use more tokens than everyday prose: there are simply more uncommon words.

A short snippet of structured data like the JSON object {”name”: “Smith”, “year”: 2023} takes around fourteen tokens, even though it has only a small number of words. Punctuation, brackets, quotation marks, and digits each consume tokens. Code and structured data are often more token-expensive than equivalent prose, which is one of the reasons that JSON-heavy workflows can rack up token counts faster than people expect.

A short word in another language, like the Japanese greeting こんにちは, takes around five tokens. The same idea in English (”hello”) would be one or two. This is because most modern tokenizers were trained primarily on English-heavy data, and they encode English very efficiently while encoding other writing systems less efficiently. The practical consequence is that the same task in non-English languages often costs more and consumes more of the context window.

The general rule of thumb is that one token is roughly three quarters of a word in English, or about four characters of text. A 1,000-word document is approximately 1,300 tokens. A 100-page book is approximately 50,000 tokens. These rough conversions are useful for sizing a job before running it.

Why this is worth knowing

Three practical consequences follow from understanding tokens.

The first is cost. Almost every commercial LLM API charges per token, with separate rates for the input you send and the output you receive. A long input is more expensive than a short one. A model that produces a long answer costs more than one that produces a short one. The total cost of a job is the cost per token multiplied by the number of tokens, summed across every call. For a single chat session this is usually trivial. For a pipeline that processes thousands of documents, the math matters, and being able to estimate token counts in advance is the difference between a budget that holds and a budget that does not.

The second is the context window. As discussed in the post on context windows, the window is a fixed budget that has to hold the input, the model’s thinking, and the output. The size of that budget is measured in tokens. A 128,000-token window holds approximately 95,000 words, which sounds like a lot until you start filling it with documents. A research article might be six to eight thousand tokens. Twenty articles is most of the budget. Thirty articles is the whole budget, with no room for the question or the answer. Estimating tokens is how you predict, before running anything, whether your input will actually fit.

The third is consistency. Because tokenization happens silently, two superficially similar inputs can have very different token counts. Switching from one document format to another can change token counts substantially even when the content is the same. Adding markup, adding headers, adding line numbers, all of this affects the count. Pipelines that depend on token-bound processing should be tested with realistic samples, not estimated from word counts alone.

What different tokenizers do differently

There is no single tokenizer used by all models. Each model family ships with its own. OpenAI uses one family of tokenizers, Anthropic uses another, Google uses yet another, and open-source models often use their own. The differences are not large, but they exist. The same text can take 1,200 tokens in one model and 1,250 in another. For most purposes this does not matter. For tight budget calculations, the relevant tokenizer is the one belonging to the specific model being used. Most providers offer a tokenizer tool on their developer site that lets you paste in text and see the exact count.

The differences across tokenizers also help explain why some models handle certain languages better than others. A tokenizer trained on a corpus that included substantial non-English text will tokenize that language more efficiently than one trained almost entirely on English. This can have downstream effects on how well the model performs in those languages, because efficient tokenization means more of the meaningful content fits in the window and more of the model’s attention budget can be spent on the substance rather than on assembling fragmented words.

Practical guidance

A few practical conclusions follow from all of this.

When estimating the size of a job, count tokens, not words. Most provider websites have a free tokenizer tool. Paste in a typical document, get the count, and multiply by the number of documents. This gives you a real number to size against.

When the input feels large, check the count before running anything. If the input plus the expected output exceeds the window, the model will either truncate or fail. Knowing this in advance is much cheaper than discovering it on a long batch run.

When working with structured data or code, expect higher token counts than the equivalent prose. Plan accordingly. If you are processing JSON-heavy material, the cost per item will be higher than the page count suggests.

When working in languages other than English, expect higher token counts and proportionally higher cost. The same task in a non-English language is often two to three times more expensive, simply because the tokenizer is less efficient on that script.

When optimizing a prompt for cost, words matter, but so does formatting. Excess whitespace, repeated headers, and verbose instructions all consume tokens. A concise prompt that gives the model the same information for fewer tokens is a better prompt for production use, holding accuracy constant.

Where this leaves the practitioner

Tokens are the underlying unit of measurement for everything that happens with a language model. They determine cost, they determine what fits in the window, and they determine how the model reads what you sent. None of this is technical in any specialized sense. It is just the layer beneath the words. Once you know that the model is reading tokens, the practical behavior of these systems becomes legible: you can estimate sizes before running jobs, predict where costs will land, anticipate where pipelines will run into limits, and design prompts that respect the format the model is actually receiving.

The text you type and the tokens the model reads are not exactly the same thing. The gap is small but consequential. Knowing that the gap exists is the first step toward working with these systems on the terms they actually operate on.