Tokens, Explained: How Language Models Read Your Text and How You're Billed

Tokens are the atomic unit every large language model uses to read your input and generate its output. If you use any LLM API — OpenAI, Anthropic, Google, Meta's Llama, DeepSeek, Mistral, or otherwise — you are billed in tokens, and the way those tokens are counted is not the same as counting words or characters.

Understanding this isn't an academic exercise. It changes how you write prompts, how you architect systems, and how much your application costs at scale.

What a Token Actually Is#

A token is a chunk of text produced by a process called tokenization — splitting raw text into pieces that the model can map to a numerical ID. Tokenizers don't split on word boundaries the way a human would. They use algorithms (Byte Pair Encoding and its variants are the most common) that learn frequent character sequences from a large training corpus and merge them into single tokens. The more often a sequence appears in training, the more likely it earns its own token.

In practice this means:

• Common short words like "the", "is", and "it" are typically one token each.
• A word like "tokenization" might split into two or three tokens — something like token, ization.
• Rare words, proper nouns, or heavily technical terms often fragment more aggressively.
• Whitespace and punctuation are sometimes absorbed into an adjacent token, sometimes treated as their own token. The leading space in " token" often makes it a different token than "token".
• Numbers fragment badly. Modern tokenizers usually break long digit strings into chunks of a few digits at a time, so "1000000" can become several tokens rather than one.
• Non-English text almost always costs more tokens per meaningful unit than equivalent English text, because most tokenizer vocabularies were trained on English-heavy corpora.

A rough heuristic that holds across most English-language models: one token ≈ four characters ≈ 0.75 words. But it is only a heuristic. Paste code, a phone number, or a paragraph in Korean, and the ratio shifts — sometimes sharply.

The Tokenization Pipeline, Step by Step#

When you submit a prompt, the following happens before any computation on meaning occurs:

1. Normalization — the text is cleaned and standardized (Unicode encoding, whitespace handling, and in some tokenizers, case folding).
2. Byte-pair encoding (BPE) or a similar algorithm — the raw text is split into the longest matching token sequences the vocabulary knows. GPT-family models use a BPE implementation called tiktoken; Llama 3 and several newer open models adopted a tiktoken-style BPE, while older Llama and many other open models use SentencePiece or a Unigram language model.
3. Vocabulary lookup — each text chunk maps to an integer ID. Vocabulary sizes have grown over time: older models sat around 32,000 IDs, while current ones commonly run from roughly 100,000 to 200,000 or more (GPT-4o's tokenizer has about 200,000 entries; Llama 3's has about 128,000). A larger vocabulary packs more meaning into each token, which helps non-English and technical text in particular.
4. Embedding — each integer is converted to a high-dimensional vector the transformer can process. From this point on, the model never sees your original text — only these vectors.

This is why models can stumble on character-level tasks ("how many r's in 'strawberry'?"). They don't see letters; they see token IDs that represent chunks. The word may arrive as a single token or two, with the individual letters never exposed to the model at all.

Context Windows and Why Token Limits Bite#

Every model has a context window — the maximum number of tokens it can hold in a single interaction, covering both the input you send and the output it generates. If you exceed the window, the request is rejected, or older content is truncated, depending on the implementation.

Context windows have expanded dramatically, from a few thousand tokens in early GPT-3 variants to a million tokens or more in several current frontier models, with a handful of open-weight models advertising multi-million-token windows. But a bigger number on the spec sheet doesn't solve everything:

• Advertised is not effective. A model rated for a million tokens does not reason equally well across all of them. Long-context benchmarks (RULER and NoLiMa among them) consistently show retrieval and multi-fact reasoning accuracy dropping well before the advertised limit — often somewhere past a couple hundred thousand tokens. Treat the headline number as a ceiling, not a working budget.
• Attention is not uniform. LLMs attend more reliably to text at the very beginning and very end of a long context. Content buried in the middle of a massive prompt gets attended to less reliably — the well-documented "lost in the middle" problem.
• Latency scales with context length. A context near the limit is slower and more expensive to process than a short one.
• Cost scales roughly linearly with token count. Sending a 50-page document with every query is not free.
• Output has its own, much smaller cap. Even models with million-token input windows typically limit a single response to tens of thousands of tokens. Don't assume a huge context window means a huge answer.

The architecture implication: for production systems, retrieval-augmented generation (RAG) exists precisely to avoid stuffing entire knowledge bases into the context. Retrieve only the relevant chunks, keep the context lean, and you win on cost, latency, and answer quality at once.

How You're Actually Billed#

API providers bill separately on input tokens and output tokens — and output tokens typically cost several times more than input tokens. The reason: generating tokens (autoregressive decoding, one token at a time) is computationally heavier than encoding the prompt in a single forward pass.

Key billing mechanics to understand:

• System prompts count. Every token in your system message is an input token, billed on every single API call. A bloated system prompt on a high-volume product is a steady, invisible cost leak.
• Chat history counts. Most chat APIs are stateless and resend the full conversation history with every turn. A ten-turn conversation means turn ten includes nine prior turns of tokens as input, every time.
• Tool definitions count. If you're using function calling or tool use, the JSON schema describing your tools is tokenized and sent as input on every call. Complex tool schemas can add hundreds or thousands of tokens silently.
• Reasoning tokens count as output — and you often can't see them. Reasoning models (extended-thinking or chain-of-thought modes, now standard across the major providers) generate a large volume of intermediate "thinking" tokens before the visible answer. These are billed at the output rate, and on a hard problem they can dwarf the final response. This is the single most common surprise on a modern LLM bill.
• Output length is partially in your control. Instructing the model to be concise, setting a max_tokens parameter, and choosing a lower reasoning effort where the provider exposes that knob all cap output cost.
• Cached input tokens are cheaper. Most major providers now offer prompt caching — if you resend the same prefix (system prompt, a long document, tool definitions) across calls, the provider can reuse the cached state and charge a steeply reduced rate for the cache hit. For applications with a large, static context, this is one of the biggest available savings.

Practical Token Arithmetic for Builders#

A few concrete implications for anyone building on top of an LLM API:

Profile before you optimize. Every major provider returns usage metadata (input, output, and — where applicable — reasoning and cached-token counts) in the API response. Log these per request. You cannot optimize what you haven't measured.

Trim your system prompt ruthlessly. Every sentence that doesn't change model behavior is dead cost, paid on every call. Write system prompts iteratively and remove instructions to verify they're actually doing work.

Choose the right model for the task. Smaller, cheaper models handle a surprising range of work. Routing simple queries to a lighter model and reserving a frontier model for hard ones — model routing — can cut costs substantially without measurable quality loss on most traffic.

Compress retrieved context. When doing RAG, don't paste full documents. Extract the relevant passages, truncate aggressively, and consider a summarization step for long sources.

Watch structured output. Asking the model to return JSON tends to be token-heavy: every key, bracket, and quote is tokenized on every response. Deeply nested keys and verbose field names cost more than a flat structure with short names. Design your schemas with tokenization in mind.

Don't trust character length as a proxy. If your app enforces a "character limit" on user input but never checks tokens, users can submit text that is short in characters yet expensive in tokens — particularly with Unicode-heavy, code, or numeric content.

Why Non-English Text Costs More (and What to Do About It)#

Because tokenizer training corpora skew heavily toward English, other languages are underrepresented in the vocabulary. A sentence in Turkish, Thai, or Arabic typically requires more tokens than the same meaning in English. This is not a quality problem — the model handles the language fine — but it is a cost and context-window problem: the same conversation eats more of your budget and fills the window faster.

Newer, larger-vocabulary tokenizers have narrowed this gap — GPT-4o's 200k-token vocabulary, for instance, compresses many non-Latin scripts noticeably better than its predecessor — but they have not erased it. For multilingual applications at scale:

• Translate user queries to English, run the model, and translate responses back (adds latency and a translation dependency; works for some use cases).
• Prefer models with modern multilingual tokenizers, which represent non-Latin scripts more efficiently.
• Monitor per-language token costs separately. Your median cost estimate for English users won't hold for users in other locales.

The Meta-Point: Tokens Are a Design Constraint#

Most developers treat token limits as a nuisance to route around. A sharper view: tokens are the fundamental design constraint of LLM-based systems, the same way memory and bandwidth are constraints in systems programming. Every architectural decision in an LLM application — what goes in the context, how conversation history is managed, which model handles which request, whether reasoning mode is worth its output cost — is really a decision about tokens.

Getting fluent with tokenization doesn't require deep ML knowledge. It requires measuring, experimenting with your specific tokenizer (OpenAI, Meta, and others publish theirs as open-source libraries), and treating token counts as a first-class engineering metric alongside latency and error rate.

---

Key Takeaways

• A token is a sub-word chunk, not a word. Roughly four characters or 0.75 words in English, worse for code and non-English text.
• Tokenizers map text to IDs from a fixed vocabulary — now commonly 100k–200k entries in current models — so the model never sees your raw characters.
• You're billed separately on input and output tokens; output costs several times more, and hidden reasoning tokens bill at the output rate.
• System prompts, chat history, and tool schemas all add to your input token count on every single call.
• Context-window limits are real, and effective context is smaller than the advertised number — attention degrades for content buried mid-context.
• Prompt caching, model routing, and RAG are the main architectural levers for controlling token costs at scale.
• Count with your actual tokenizer library, not character length — they diverge in ways that hit your bill.