Side project · Python · applied AI
Tokenizer playground
How an AI model cuts your text into pieces before it reads a single word.
What is a tokenizer?
Language models like ChatGPT don’t read letters or words. They read tokens: chunks of text, each with a number. A tokenizer decides where those cuts go. Byte pair encoding (BPE), the method behind GPT, starts with single bytes and keeps merging the pair that appears most often in its training text, until its vocabulary is full. Frequent words end up as one token; rare words get split into pieces.
Why I built it
I wanted to see what actually happens before an AI model reads anything, and to learn Python properly along the way. So I built a BPE tokenizer from scratch, without any tokenizer library, in small tested steps, with Claude Code as my coding partner. The original is a Python app; this page runs the same logic in your browser, so you can try it here. It is made to visualise, not to compete with real tokenizers.
Try it
Type something, move the slider and watch the colours. Each colour is one token.
- Bytes
- –
- Tokens
- –
- Merges learned
- –
- Bytes per token
- –
Tokens
Hover a token to see its number; the same token lights up everywhere. Dashed boxes such as \xc3 hold only part of a character (“ä” is two bytes in UTF-8).
Token numbers
What it learned
Every merge, in the order it was learned.
| Token | Part 1 | Part 2 | Result |
|---|
Built in Python; this is a faithful JavaScript port that learns exactly the same merges.