Skip to content

Tokenizer

Short answer

A tokenizer splits text into the units a model actually reads — usually word fragments rather than whole words or characters.

"Unbelievable" might be three tokens; a rare surname might be six. This is why token counts never match word counts and why English costs fewer tokens than languages the tokenizer saw less of during training.

It matters for context budgeting. A 4,000-token window holds noticeably less Hungarian than English, and a spreadsheet full of long numeric IDs eats tokens at a startling rate.

Related terms

Other ways people ask this

  • what is tokenizer

Related