Wooden compartmentalized box filled with metal type pieces arranged in divided sections, with tools blurred in background

ArticleaiDeep read

Tokens, Explained: How Language Models Read Your Text and How You're Billed

Ahmad JAug 7, 20268 minUpdated Sep 14, 2026

Tokens are the unit of currency in every LLM interaction. Understand how text becomes tokens, why the count never matches your word count, and which levers actually control cost at scale.

A deep read: the full picture, with the receipts.

Signalstrong3independent sources

Tokens are the atomic unit every large language model uses to read your input and generate its output. If you use any LLM API (OpenAI, Anthropic, Google, Meta's Llama, DeepSeek, Mistral, or otherwise), you are billed in tokens, and the way those tokens are counted is not the same as counting words or characters.

Understanding this isn't an academic exercise. It changes how you write prompts, how you architect systems, and how much your application costs at scale.

What a Token Actually Is#

A token is a chunk of text produced by a process called tokenization: splitting raw text into pieces that the model can map to a numerical ID. Tokenizers don't split on word boundaries the way a human would. They use algorithms (Byte Pair Encoding and its variants are the most common) that learn frequent character sequences from a large training corpus and merge them into single tokens. The more often a sequence appears in training, the more likely it earns its own token.

In practice this means:

What you feed it	How it tokenizes
Common short words: "the", "is", "it"	Typically one token each
A longer word like "tokenization"	Two or three tokens, something like `token`, `ization`
Rare words, proper nouns, heavily technical terms	Fragment more aggressively
Whitespace and punctuation	Sometimes absorbed into an adjacent token, sometimes a token of their own. The leading space in " token" often makes it a different token than "token"
Long numbers	Fragment badly. Modern tokenizers break long digit strings into chunks of a few digits at a time, so "1000000" can become several tokens rather than one
Non-English text	Almost always more tokens per meaningful unit than the equivalent English, because most tokenizer vocabularies were trained on English-heavy corpora

A rough heuristic that holds across most English-language models: one token ≈ four characters ≈ 0.75 words.

But it is only a heuristic. Paste code, an account number, or a paragraph in Korean, and the ratio shifts, sometimes sharply.

The Tokenization Pipeline, Step by Step#

When you submit a prompt, the following happens before any computation on meaning occurs:

Stage	What happens to your text
Normalization	Cleaned and standardized: Unicode encoding, whitespace handling, and in some tokenizers case folding
Byte-pair encoding, or a similar algorithm	The raw text is split into the longest matching token sequences the vocabulary knows. GPT-family models use a BPE implementation called tiktoken; Llama 3 and several newer open models adopted a tiktoken-style BPE, while older Llama and many other open models use SentencePiece or a Unigram language model
Vocabulary lookup	Each chunk maps to an integer ID. Older models sat around 32,000 IDs; current ones commonly run from roughly 100,000 to 200,000 or more (GPT-4o's tokenizer has about 200,000 entries, Llama 3's about 128,000). A larger vocabulary packs more meaning into each token, which helps non-English and technical text in particular
Embedding	Each integer becomes a high-dimensional vector the transformer can process. From here on the model never sees your original text, only these vectors

This is why models can stumble on character-level tasks ("how many r's in 'strawberry'?"). They don't see letters; they see token IDs that represent chunks. The word may arrive as a single token or two, with the individual letters never exposed to the model at all.

Context Windows and Why Token Limits Bite#

Every model has a context window: the maximum number of tokens it can hold in a single interaction, covering both the input you send and the output it generates. If you exceed the window, the request is rejected, or older content is truncated, depending on the implementation.

Context windows have expanded dramatically, from a few thousand tokens in early GPT-3 variants to a million tokens or more in several current frontier models, with a handful of open-weight models advertising multi-million-token windows. But a bigger number on the spec sheet doesn't solve everything:

What the spec sheet doesn't say	What it means in practice
Advertised is not effective	A model rated for a million tokens does not reason equally well across all of them. Long-context benchmarks, RULER and NoLiMa among them, consistently show retrieval and multi-fact reasoning accuracy dropping well before the advertised limit, often somewhere past a couple hundred thousand tokens. Treat the headline number as a ceiling, not a working budget
Attention is not uniform	LLMs attend more reliably to text at the very beginning and the very end of a long context. Content buried in the middle of a massive prompt gets attended to less reliably: the well-documented "lost in the middle" problem
Latency scales with context length	A context near the limit is slower and more expensive to process than a short one
Cost scales roughly linearly with token count	Sending a 50-page document with every query is not free
Output has its own, much smaller cap	Even models with million-token input windows typically limit a single response to tens of thousands of tokens. A huge context window does not mean a huge answer

The architecture implication: for production systems, retrieval-augmented generation (RAG) exists precisely to avoid stuffing entire knowledge bases into the context. Retrieve only the relevant chunks, keep the context lean, and you win on cost, latency, and answer quality at once.

How You're Actually Billed#

API providers bill separately on input tokens and output tokens, and output tokens typically cost several times more than input tokens. The reason: generating tokens (autoregressive decoding, one token at a time) is computationally heavier than encoding the prompt in a single forward pass.

Key billing mechanics to understand:

What gets billed	The part teams miss
System prompts	Every token in your system message is an input token, billed on every single API call. A bloated system prompt on a high-volume product is a steady, invisible cost leak
Chat history	Most chat APIs are stateless and resend the full conversation history with every turn. Turn ten of a ten-turn conversation includes nine prior turns of tokens as input, every time
Tool definitions	With function calling or tool use, the JSON schema describing your tools is tokenized and sent as input on every call. Complex schemas can add hundreds or thousands of tokens silently
Reasoning tokens, billed at the output rate	Extended-thinking and chain-of-thought modes, now standard across the major providers, generate a large volume of intermediate "thinking" tokens before the visible answer. On a hard problem they can dwarf the final response, and you often can't see them. This is the single most common surprise on a modern LLM bill
Output length, partially in your control	Instructing the model to be concise, setting a `max_tokens` parameter, and choosing a lower reasoning effort where the provider exposes that knob all cap output cost
Cached input, at a reduced rate	Most major providers now offer prompt caching: resend the same prefix (system prompt, a long document, tool definitions) and the provider reuses the cached state. For an application with a large, static context this is one of the biggest available savings

Practical Token Arithmetic for Builders#

A few concrete implications for anyone building on top of an LLM API:

The lever	What to actually do
Profile before you optimize	Every major provider returns usage metadata in the API response: input, output, and where applicable reasoning and cached-token counts. Log these per request. You cannot optimize what you haven't measured
Trim the system prompt ruthlessly	Every sentence that doesn't change model behaviour is dead cost, paid on every call. Write system prompts iteratively, and remove instructions to verify they are actually doing work
Choose the right model for the task	Smaller, cheaper models handle a surprising range of work. Model routing, sending simple queries to a lighter model and reserving a frontier model for hard ones, can cut costs substantially without measurable quality loss on most traffic
Compress retrieved context	With RAG, don't paste full documents. Extract the relevant passages, truncate aggressively, and consider a summarization step for long sources
Watch structured output	Asking for JSON tends to be token-heavy: every key, bracket and quote is tokenized on every response. Deeply nested keys and verbose field names cost more than a flat structure with short names
Don't trust character length as a proxy	If your app enforces a character limit on user input but never checks tokens, users can submit text that is short in characters yet expensive in tokens, particularly Unicode-heavy, code, or numeric content

Why Non-English Text Costs More (and What to Do About It)#

Because tokenizer training corpora skew heavily toward English, other languages are underrepresented in the vocabulary. A sentence in Turkish, Thai, or Arabic typically requires more tokens than the same meaning in English. This is not a quality problem (the model handles the language fine), but it is a cost and context-window problem: the same conversation eats more of your budget and fills the window faster.

Newer, larger-vocabulary tokenizers have narrowed this gap (GPT-4o's 200k-token vocabulary, for instance, compresses many non-Latin scripts noticeably better than its predecessor), but they have not erased it. For multilingual applications at scale:

The move	The trade-off
Translate user queries to English, run the model, translate the response back	Adds latency and a translation dependency. Works for some use cases, not all
Prefer models with modern multilingual tokenizers	They represent non-Latin scripts more efficiently
Monitor per-language token costs separately	Your median cost estimate for English users won't hold for users in other locales

The Meta-Point: Tokens Are a Design Constraint#

Most developers treat token limits as a nuisance to route around. A sharper view: tokens are the fundamental design constraint of LLM-based systems, the same way memory and bandwidth are constraints in systems programming. Every architectural decision in an LLM application (what goes in the context, how conversation history is managed, which model handles which request, whether reasoning mode is worth its output cost) is really a decision about tokens.

Getting fluent with tokenization doesn't require deep ML knowledge. It requires measuring, experimenting with your specific tokenizer (OpenAI, Meta, and others publish theirs as open-source libraries), and treating token counts as a first-class engineering metric alongside latency and error rate.

Key Takeaways

Topic	What to remember
What a token is	A sub-word chunk, not a word. Roughly four characters or 0.75 words in English, worse for code and non-English text
The vocabulary	Tokenizers map text to IDs from a fixed vocabulary, now commonly 100k–200k entries, so the model never sees your raw characters
The bill	Input and output are billed separately. Output costs several times more, and hidden reasoning tokens bill at the output rate
What inflates input	System prompts, chat history and tool schemas all add to your input token count on every single call
The context window	Effective context is smaller than the advertised number: attention degrades for content buried mid-context
The levers	Prompt caching, model routing and RAG are the main architectural levers for controlling token costs at scale
How to measure	Count with your actual tokenizer library, not character length. They diverge in ways that hit your bill