Before an AI model reads a single word, a quieter piece of technology has already made an important decision: how to cut the text into pieces. That piece of technology is the tokenizer, and it shapes how fairly a model treats every language it works with.
This article explains what a tokenizer does, why it can make some languages slower and more expensive to process than others, and why Ravira chose to build its own.
What a tokenizer does
AI models do not read letters or whole words. They read tokens: small pieces of text, sometimes a full word, sometimes a fragment of one. The tokenizer is the tool that splits text into these pieces, using a vocabulary it learned from the text it was trained on.
A word the tokenizer has seen often becomes a single token. A word it rarely saw is broken into several smaller pieces. So the same sentence can take very different numbers of tokens depending on the language it is written in.
The hidden cost
Researchers at the University of Oxford measured this effect across 17 tokenizers in a study presented at NeurIPS 2023. They found that the same text, translated into different languages, can need up to 15 times more tokens in some languages than in English. Even the tokenizer used by a widely known AI assistant needed about 1.6 times more tokens for Italian and 2.6 times more for Bulgarian than for the same content in English.
More tokens mean three practical things for the people who speak those languages:
- Higher cost, because many AI services charge per token.
- Slower answers, because every extra token takes time to process.
- Less room, because a model can only hold a limited number of tokens at once, so less of a long document fits.
The study concluded that future language models should be built with tokenizers that treat languages fairly. That conclusion sits at the heart of how Ravira approaches its own foundations.
How Ravira built its own tokenizer
Ravira trained its own tokenizer for close to 100 languages, with fairness as a design goal from the first step:
- A fair share for every language. The training text was balanced so that smaller languages are not drowned out by the largest ones, and each language gets enough room in the vocabulary to be read in sensible pieces.
- Text as people really write it. Many people type without diacritics on phones and keyboards that make them awkward. A tokenizer that only knows the correct forms splits that everyday text into far more pieces. Ravira's tokenizer learned both the correct forms and the everyday ones.
- Every script respected. Languages written in more than one script, such as Serbian in Latin and Cyrillic, keep each script as its own form instead of being folded into one.
The tokenizer is the first foundation of Ravira's own translation model, and it was built to last.
Built without your texts
There is a second choice behind Ravira's foundations, and it matters just as much: they are built without the texts our users translate.
Many AI products improve themselves by learning from what their users type. Ravira is privacy-first by design, so the texts you translate are there to give you your translation, not to train our models. The tokenizer was trained on publicly available multilingual text, and the same principle guides the rest of the work.
This takes more effort than learning from user traffic, and we think it is the right way to build. A contract, a medical letter or an unpublished manuscript should be translated, not turned into training material.
What this makes possible
A fair tokenizer is invisible when it works. A speaker of Estonian, Welsh or Romanian simply gets the same attention from the model as a speaker of English, at a cost and speed that do not depend on how large their language is.
That is the kind of foundation Ravira wants under every translation: fair to every language, and respectful of every text. You can read more about why owning these foundations matters in Owning the foundation early.
Research figures come from Petrov, La Malfa, Torr and Bibi, "Language Model Tokenizers Introduce Unfairness Between Languages", NeurIPS 2023.