English vs Chinese vs Japanese Tokens: A Reproducible Experiment

The question this experiment answers

TokenCostAI currently estimates tokens with one browser-side character rule: divide Chinese ideographs by 1.5, divide every other UTF-16 character by 4, add the two values, and round up. That rule is deliberately described as a budgeting estimate, but a single heuristic can behave very differently across writing systems. Japanese is an especially important test because kanji match the rule's Chinese range while hiragana and katakana fall into the broad “other character” bucket.

This experiment measures the resulting error instead of repeating a generic “one token is about four characters” guideline. We use three short business tasks, each written in natural English, Simplified Chinese, and Japanese. For each text we record character and word counts, the current TokenCostAI estimate, and token counts returned by provider-supported counting endpoints.

The experiment is intentionally narrow. It does not measure output length, answer quality, translation quality, latency, or which model is “best.” It does not assume that a count from one provider applies to another provider. It asks one practical question: if a user pastes multilingual text into the current calculator, how far can the displayed estimate be from the tokenizer attached to the model they intend to use?

Test material

The fixture contains three scenarios:

1. A customer-support instruction about an app crash after receipt upload.
2. A monthly API-budget calculation with fixed request and token quantities.
3. An invoice-field extraction instruction that requests valid JSON.

Each scenario has English, Chinese, and Japanese versions. The versions preserve the requested task and required fields, but they are not literal word-for-word translations. Literal translation would make the language unnatural and would not represent normal production prompts. This means the experiment compares meaning-equivalent task instructions, not identical byte sequences.

The fixture is stored in multilingual-token-fixtures.json. The site owner approved the meaning-equivalent text set on 2026-08-23, and the three scenarios were frozen with SHA-256 ba840c8cc94575ab3bcb3fff4dd420cf70f7ad930ab02b3dbe0c72943e876292. The collector refuses to run if those texts change. This meaning review is separate from reviewing the numeric results.

What the local estimate actually counts

The current implementation uses JavaScript text.length, so its total is measured in UTF-16 code units. It then identifies characters in the CJK Unified Ideographs and Extension A ranges. Everything else—including spaces, punctuation, Latin letters, digits, Japanese kana, and any emoji surrogate pairs—is grouped together.

The formula is:

estimated tokens = ceil(CJK ideographs / 1.5 + other UTF-16 code units / 4)

This formula has two consequences that deserve testing. First, Japanese sentences mix character classes that receive substantially different weights. Second, provider tokenizers can include tokens associated with the request structure, not only visible characters. A character heuristic cannot reproduce model-specific vocabularies or message formatting.

Official counting methods

OpenAI documents a POST /responses/input_tokens endpoint that returns input_tokens for the supplied Responses API input: https://developers.openai.com/api/reference/cli/resources/responses/subresources/input_tokens

Anthropic documents POST /v1/messages/count_tokens. Its documentation says the result should be treated as an estimate and may differ slightly from the input count in a subsequent Message request: https://platform.claude.com/docs/en/api/messages/count_tokens

Google documents models.countTokens, which runs the selected Gemini model's tokenizer and returns totalTokens: https://ai.google.dev/api/tokens

The collection script sends each text as a plain user input with no system instruction, tools, images, or conversation history. It calls only token-count endpoints; it does not ask a model to generate a response. API credentials are read from environment variables and are never written to the result files.

DeepSeek is not included in the numeric table unless a current official model-specific counting method is confirmed. Token usage returned after a generated response would mix this measurement with a billable generation call and would violate the project's ¥0 research budget. Missing cross-provider data is therefore recorded as unavailable rather than inferred from another tokenizer.

Why counts should be compared within a provider

The three official endpoints do not necessarily count the same envelope. OpenAI receives Responses API input, Anthropic receives a Messages API user message, and Gemini receives a contents object. Provider tokenizers and automatically inserted formatting can differ. A raw cross-provider ranking could therefore attribute message-wrapper differences to language.

The defensible comparison is within each provider and model:

• Compare English, Chinese, and Japanese counts for the same scenario.
• Compare TokenCostAI's estimate with that provider's returned count.
• Calculate signed error to show whether the site overestimates or underestimates.
• Calculate absolute percentage error to express the size of the miss.
• Aggregate only after all nine texts have a count from the same provider.

The signed error formula is site estimate - official count. A negative value means the site underestimates. Absolute percentage error is abs(site estimate - official count) / official count × 100.

Gemini 3.7 Flash results

The frozen fixture was counted with Gemini 3.7 Flash on 2026-08-23. The candidate contains all nine requested Gemini counts. OpenAI and Anthropic were deliberately not selected because this experiment has a ¥0 budget and no existing authorized keys were available. Their empty rows are not_selected, not evidence that the providers lack token-count APIs. This is Gemini-only evidence and must not be generalized to another tokenizer.

Download the approved machine-readable multilingual-token-results.json or spreadsheet-friendly multilingual-token-results.csv. Both files preserve the frozen scenario IDs, language, site estimate, official count, signed error, percentage error, model ID, and collection status.

Scenario	Language	Characters	Site estimate	Gemini count	Signed error	Absolute error
Support triage	English	168	42	33	+9	27.3%
Support triage	Chinese	54	35	32	+3	9.4%
Support triage	Japanese	86	32	43	-11	25.6%
API budget	English	183	46	47	-1	2.1%
API budget	Chinese	77	43	47	-4	8.5%
API budget	Japanese	106	41	54	-13	24.1%
Invoice extraction	English	151	38	34	+4	11.8%
Invoice extraction	Chinese	55	30	36	-6	16.7%
Invoice extraction	Japanese	77	32	37	-5	13.5%

The English mean absolute percentage error was 13.7%. Its direction changed by scenario: the heuristic overestimated support triage and invoice extraction but slightly underestimated the numeric API-budget prompt. This variation shows why “four characters per token” is not a fixed conversion rate even within one language.

The Chinese mean absolute percentage error was 11.5%. The heuristic was close on support triage, then underestimated the other two scenarios. Across the three Chinese texts it estimated 108 tokens versus 115 returned by Gemini, a net underestimate of 7 tokens or 6.1%. Three short prompts are not enough to claim a general Chinese correction factor.

The Japanese mean absolute percentage error was 21.1%, the largest of the three language groups. More importantly, all three Japanese rows were underestimated. The heuristic estimated 105 tokens in total while Gemini returned 134, a 29-token shortfall or 21.6% of the official total. This direction is consistent with the implementation: kana fall into the generic “other characters divided by four” branch, even though Gemini's tokenizer does not follow that character rule.

Across all nine texts, the heuristic estimated 339 tokens while Gemini returned 363. That net difference is smaller than several row-level errors because English overestimates partially cancel Japanese and Chinese underestimates. A portfolio total therefore hides the risk faced by an individual prompt. For budgeting, the row-level direction and range are more useful than the aggregate net error.

These results do not prove that Japanese always consumes more tokens than English or Chinese. In these three meaning-equivalent scenarios, Japanese had the highest Gemini count each time, but sentence construction, terminology, digits, punctuation, model version, and message envelope all affect tokenization. The experiment supports a warning about this calculator and this model—not a universal language price ranking.

What this result can and cannot establish

The strongest evidence is procedural: the inputs were approved before collection, frozen by hash, sent to an official tokenizer endpoint, and preserved with the local estimates and error formulas. A reader can download the same texts and repeat the request. That is materially stronger than an unattributed token rule.

The limitations are equally important. Nine prompts are a diagnostic sample, not a representative language corpus. Gemini's countTokens result includes the request structure used by its contents API. OpenAI Responses and Anthropic Messages can apply different wrappers and tokenizers. Counts can also change when a provider updates a model alias. For those reasons, the collected date, exact model ID, and fixture hash must remain attached to the table.

The experiment contains synthetic business instructions only. It does not expose customer data, production prompts, API keys, or invoices. Google’s free API tier may use submitted content to improve its products, so synthetic inputs are the appropriate choice for this zero-budget collection.

Calculator changes that may follow

The nine reviewed candidates justify improving disclosure, but they do not justify fitting a new universal formula. The recommended first release is therefore limited:

• Keep the current formula so an isolated nine-row sample does not overfit production behavior.
• Replace the generic accuracy note with a clear statement that the result is a character heuristic, not a provider tokenizer.
• Add a Japanese-specific warning: in this Gemini 3.7 Flash sample, all three prompts were underestimated by 13.5% to 25.6%.
• Link to official provider token-count methods for billing-sensitive work.
• Publish the frozen fixture and result table so users can inspect the evidence.
• Reconsider a language-specific estimation profile only after testing a larger corpus and at least one additional provider.

An exact model tokenizer should not be simulated with unexplained correction factors. If the observed errors vary substantially by scenario or provider, the honest product change may be a wider warning rather than a more complicated formula.

Reproduce the experiment

From the project root, set a temporary Gemini API key and run:

npm run content:collect:hv04:gemini
npm run content:verify:hv04

The npm command enables Node's environment-proxy support, sends only the nine approved texts to Gemini's token-count endpoint, and marks OpenAI and Anthropic as not selected. The candidate JSON and CSV remain unpublished until the owner reviews them. Re-running against a changed tokenizer count fails the evidence lock and requires a new review rather than silently changing the article.

The review must confirm the translations, model IDs, collection statuses, all non-null counts, and the resulting error calculations. Only then can the article be completed, approved separately, connected to the blog and Sitemap, and used to modify calculator messaging.