AI glossary
What is a token in AI?
A token is the small chunk of text a language model works with: a word, part of a word or a symbol. Context length and cost are both measured in tokens.
A language model does not read letters or whole words. Before it processes any text, a tokenizer splits it into tokens: a short, common word is usually a single token, a long or rare one is split into several, and punctuation and spaces count too.
Tokens matter for three reasons. The model writes its answer one token at a time, picking one of the most likely options. The context window, the most it can take into account at once, is measured in tokens. And AI APIs charge per token, with input tokens (what you send) priced apart from output tokens (what it answers), which usually cost more.
As a rough rule, one token is about four characters of English text. Other languages, like Spanish, usually need a few more tokens for the same text, because tokenizers were built mostly from English.
An example
"the" is a single token, while "tokenization" may be split into two or three. The exact count depends on each model's tokenizer.