A language model uses a tokenizer to split input text into units called tokens. Depending on the tokenizer, a token may be an entire short word, part of a longer word, punctuation or a character sequence. The model predicts or processes token identifiers rather than treating a paragraph as one indivisible unit.
Tokens are important for both model limits and cost. Context windows are usually measured in tokens, and many hosted APIs charge separately for input and generated output tokens. Two passages with the same number of words may occupy different numbers of tokens, especially across languages or with code and unusual symbols.
Tokenization is not the same as dividing text at spaces. The same expression may be split differently by different models, and a token's numeric identifier has meaning only within the tokenizer used to create it. Changing tokenizers can therefore affect preprocessing, token counts and compatibility.
When estimating inference cost or designing prompts, use the tokenizer or usage meter for the chosen model. Keep a margin for system instructions, retrieved context and output so that long requests do not exceed the available context window.