不同模型Tokenizer成本差异的技术解析

If you build with LLMs, you pay by the token. Not by the word, not by the character — the token. And yet most of us treat the tokenizer as a black box: text goes in, a number comes out, the bill arrives.

That black box is worth opening. Once you understand how tokenization works, a lot of otherwise-mysterious LLM behavior starts to make sense: why the same sentence costs 3 tokens on Claude and 4 on GPT, why your Spanish chatbot costs more than the English one, why models are weirdly bad at arithmetic, why some prompt styles quietly burn your budget — and, crucially, how to actually calculate what a feature will cost before you ship it.

What a Token Actually Is

A token is not a word and not a character. It's a chunk of text — usually a subword — that the model treats as a single unit. Before a model reasons about anything, your text is split into these chunks, and each chunk is mapped to an integer ID. The model only ever sees those integers.

A rough rule of thumb for English:

1 token ≈ 4 characters ≈ 0.75 words

So ~100 tokens is about 75 words, and a page of English prose (~500 words) is roughly 650-750 tokens.

But that's just an average for English. The real count depends entirely on how the text gets chunked — and that's decided by an algorithm called BPE.

How Tokenizers Are Built: BPE in Plain Terms

Nearly every major model today — GPT, Claude, Gemini, Llama, Mistral — uses some flavor of Byte-Pair Encoding (BPE) or a close relative.

BPE started life as a data-compression trick in 1994 and was adapted for language models in 2016. The idea is genuinely simple:

Start with text broken into the smallest units (bytes/characters).

Count every adjacent pair of units.

Merge the single most frequent pair into a new combined unit.

Repeat until you hit a target vocabulary size.

Each merge gets recorded, in order, into a permanent list. At inference time, the tokenizer just replays those merge rules deterministically on your text.

The key consequence: frequency during training decides everything. Common words become single tokens; rare words get split into pieces. This is why the is one token but tokenization might be two or three (token + ization), and why the splits don't follow English grammar — they follow whatever was statistically common in the training text.

Modern tokenizers also start from raw bytes (the 256 possible byte values) rather than characters. That's what lets them handle anything — emoji, Chinese, symbols, typos — without ever hitting an "unknown word." Worst case, a weird character just falls back to several byte-level tokens.

One design tension worth knowing: a bigger vocabulary means fewer tokens per sentence (cheaper, shorter sequences) but a larger embedding table and more memory. GPT-2 learned ~50,000 merges; models like GPT-4o's o200k_base use roughly 200,000. That jump is a big part of why newer models are more token-efficient per word.

Count It Yourself: tiktoken

For OpenAI models, the tokenizer is open source, so you can get exact counts locally:

# pip install tiktoken
import tiktoken

enc = tiktoken.get_encoding("o200k_base")  # GPT-4o / newer

samples = [
    "Hello, world!",
    "tokenization",
    "The quick brown fox jumps over the lazy dog.",
    "12345678901234567890",
    '{"user_id": 42, "name": "Alice", "active": true}',
]

for s in samples:
    ids = enc.encode(s)
    print(f"{len(ids):>3} tokens | {s}")

  4 tokens | Hello, world!
  2 tokens | tokenization
 10 tokens | The quick brown fox jumps over the lazy dog.
  ...       (long digit runs fragment into several tokens)
 ...        (the JSON spends tokens on braces, quotes, and keys)

For Claude, the tokenizer isn't published — use Anthropic's token-counting endpoint (POST /v1/messages/count_tokens), which accepts the same shape as a real request and returns the input token total. It's free to call. For Gemini, use Google's countTokens API. Don't cross-apply one model's count to another — they diverge.

Why the Same Text Costs Different Amounts on Different Models

Send Hello, world! to GPT and you might pay 4 tokens; send it to Claude and you might pay 3. Same text, different integers out. Why?

Because each provider trained its own tokenizer on its own data, with its own vocabulary size, merge tables, and rules for whitespace and non-Latin scripts. They share a family resemblance but diverge in the details:

• GPT (OpenAI) — tiktoken, open source. cl100k_base (~100k vocab) on older models; o200k_base (~200k vocab) on newer ones. Exact local counts.
• Claude (Anthropic) — proprietary BPE, not published. Use the count_tokens endpoint. Note: newer Claude models use a tokenizer that can produce ~30% more tokens for the same text than older ones — which matters when you compare sticker prices.
• Gemini (Google) — SentencePiece-based, not published. Use countTokens.

Takeaway: a token count measured on one model does not transfer to another — not even between generations of the same model. Budget with the exact model you'll deploy.

What Makes Token Usage Worse (and Better)

Token count isn't just "length of text." Several factors push it up or down — and these are things you can actually control.

Inflates your token count 📈

• Numbers. Digits tokenize unpredictably. 127 might be one token; 677 can split into two; long numbers fragment into several. (Also a big reason LLMs are shaky at arithmetic.)
• Whitespace & indentation. Spaces, tabs, and newlines are tokens too. Deeply indented code spends real tokens on whitespace.
• Rare / made-up words. Jargon, UUIDs, hashes, and base64 fragment into many small pieces.
• Heavy formatting. JSON — with its braces, quotes, and repeated keys — is heavier than leaner formats. This is why YAML often tokenizes cheaper than JSON for the same data.
• Template overhead. Every request is wrapped in role markers and template tokens. Same overhead in every language, so on short messages it's a proportionally bigger tax.

Reduces your token count 📉

• Concise, plain prose over verbose phrasing.
• Common words over rare synonyms.
• Removing repeated context — cache it or reference it instead of re-sending.
• Leaner data formats where structure allows.
• Capping output length (see cost math below — output is where the money goes).

This doesn't mean mangle your prompts into unreadable shorthand. It means the obvious wins — don't re-send a giant system prompt on every call, don't pad with filler, cap max_tokens — are real money.

The Language Tax: Why Non-English Costs 2-15x More

The biggest and least-known factor. Tokenizers are trained on English-heavy text, so they learn big efficient tokens for English and few for everything else. This is measured as fertility — tokens per word. English sits ~1.2-1.4; other languages run far higher.

(Figures are approximate and vary by tokenizer — measure your own.)

Two drivers: script/encoding (English is 1 UTF-8 byte per char thanks to ASCII; other scripts need 2-4 bytes) and word frequency (underrepresented languages never earned efficient tokens).

The consequences are structural — you can't prompt them away:

• Cost: 3x the tokens ≈ 3x the bill for the same meaning.
• Context window: the same 200k window holds far less non-English content.
• Latency: more tokens = slower responses.

If your usage is global, estimate cost and context per language, not once in English. The English number is the best case, not the average.

Now the Money: How to Actually Calculate Cost

APIs price input and output tokens separately, and output is almost always far more expensive. The core formula:

cost = (input_tokens  / 1,000,000 × input_price)
     + (output_tokens / 1,000,000 × output_price)

Representative flagship prices (per 1M tokens)

⚠️ Prices change constantly and vary by exact model/tier — treat these as an illustrative August 2026 snapshot, and always confirm on the provider's current pricing page before budgeting. Some models also double rates past ~200k-token context, and newer Claude tokenizers emit ~30% more tokens for the same text.

Worked example: one chatbot request

Say a single request has 1,500 input tokens (system prompt + history + user message) and 500 output tokens, on a mid-tier model at $3 / $15 per 1M:

input  = 1,500  / 1,000,000 × $3   = $0.0045
output =   500  / 1,000,000 × $15  = $0.0075
------------------------------------------------
total per request                  ≈ $0.012

Just over one cent per request. Feels trivial — until you scale.

Scaling it out (same request, 100,000/day)

per day   = 100,000 × $0.012        = $1,200
per month = $1,200 × 30             ≈ $36,000

That "one cent" is now a $36k/month line item. This is why the math matters before launch, not after the invoice.

The output tax, made concrete

Notice that in the example above, output cost more than input despite being one-third the tokens ($0.0075 vs $0.0045). If you let max_tokens default to a huge buffer and the model rambles, output balloons. Capping output length is often the single highest-leverage cost lever you have.

The language tax, in dollars

Take that same request, but the user writes in a language with 3x fertility. Input and output token counts roughly triple:

input  = 4,500  / 1,000,000 × $3   = $0.0135
output = 1,500  / 1,000,000 × $15  = $0.0225
------------------------------------------------
total per request                  ≈ $0.036   (3× the English cost)

Same feature, same user intent, triple the bill — purely from tokenization.

Blended rate: comparing providers honestly

Input and output prices differ, so don't compare "input price" in your head. Compute a blended rate using your real traffic mix. For an 80% input / 20% output workload:

blended = 0.8 × input_price + 0.2 × output_price

On a $3/$15 model: 0.8×3 + 0.2×15 = 2.4 + 3.0 = $5.40 per 1M blended. Run that for each candidate model with your ratio — it often reorders the "cheapest" ranking versus headline input prices.

Cost Levers That Actually Work

In rough order of impact:

1. Prompt caching. Cached input tokens often cost ~10% of the normal rate. If you re-send a big fixed system prompt every call, this can cut costs up to ~90% on that portion.
2. Route by difficulty. Send trivial requests (greetings, formatting, lookups) to a small/cheap model; reserve flagships for hard reasoning. A tiny classifier in front of your router commonly cuts mixed-workload cost 60-80%.
3. Cap max_tokens aggressively. The default output buffer is usually far bigger than you need, and output is the pricey side.
4. Batch API for non-realtime work. Typically ~50% off for jobs that don't need instant responses.
5. Trim and cache context. Don't re-send history or documents you can reference or summarize.
6. Budget per language. Weight your cost model by the token cost of each language you serve.

---

• You pay per token, not per word — and the exchange rate shifts with language, format, and content.
• Count with the real tokenizer: tiktoken (OpenAI, exact/local/free), count_tokens (Claude), countTokens (Gemini). Never cross-apply.
• Count the final request body you actually send — system prompt, tool schemas, history, all of it.
• Output tokens usually dominate cost. Cap them.
• Non-English can cost 2-15x more. Budget per language.
• Compute a blended rate with your real input/output mix before picking a model.
• Re-verify prices — they change often and can double past long-context thresholds.

The tokenizer isn't an implementation footnote. It's the layer where the economics of your app are quietly set — the interface between human language and the model's math, and the exact place your bill is decided. Understanding it turns a mysterious invoice into something you can reason about, forecast, and control.

Have you hit a surprising token bill or a weird tokenization bug in production? Drop the story in the comments — I collect these.