Researcher Aran Komatsuzaki published two sets of visualization data, using OpenAI’s English token consumption as a baseline (1.00×), and conducted a cross-comparison of non-English efficiency for six mainstream models’ tokenizers: OpenAI, Anthropic, Gemini 3.1, Qwen3.6, DeepSeek V4, and Kimi K2.6. The results show that Anthropic has the highest token cost across all major languages: Hindi 3.11×, Arabic 2.74×, Korean 2.59×, Russian 1.96×, with a nine-language average of 2.07×, far exceeding the other five models. Kimi K2.6 ranks second with an average of 1.76×, while Gemini 3.1 (1.22×) and Qwen3.6 (1.23×) are tied for the lowest, making them the most friendly to non-English users.
One notable reversal in the data: the three Chinese-model tokenizers — Qwen3.6 (0.85×), Kimi K2.6 (0.81×), and DeepSeek V4 (0.87×) — all require fewer tokens to process Chinese content than the English baseline. That is, for semantically equivalent content, Chinese actually consumes fewer tokens than English on these models. For API callers, tokenization efficiency directly impacts cost and context utilization. Taking Claude as an example, Hindi users sending an equivalent amount of information would consume over three times as many tokens as English users, essentially shrinking the available context window significantly. Komatsuzaki points out that Hindi has one of the largest native speaker populations in the world, yet it is the least well-covered among current mainstream tokenizers, reflecting a significant gap in language fairness within AI infrastructure.