What You Learn Rebuilding BPE From Scratch (That tiktoken Hides)

I rebuilt the core of tiktoken in pure Python: MinBPE (source). It replicates tiktoken's core, uses the GPT-2/GPT-4 regex pre-tokenizer, byte-level encoding with byte fallbacks, and a custom vocabulary trained on technical corpora. Measured results: about 15% token reduction on technical text, roughly 50K merges learned into the vocabulary, and about 2x slower than tiktoken, which is the expected price of pure Python. This essay is about what building one teaches you that importing one hides.

Why build a tokenizer to use LLMs

Most "the model is stupid" reports are tokenization artifacts: split identifiers, inconsistent casing, invisible characters, numbers chunked unhelpfully, a stray non-breaking space that changes the entire token stream. Once you have written a tokenizer, you can spot these because you have had to decide where every byte goes. Building one is the cheapest debugging education available in the LLM stack, and it pays for itself the first time a weird output turns out to be an encoding problem rather than a reasoning problem.

There is a second reason, less discussed: cost. You pay per token, and tokens are produced by a deterministic algorithm you can inspect. Knowing the algorithm means knowing why two prompts with identical word counts differ in price, why a pasted stack trace balloons, and why minified JSON is cheaper than pretty-printed JSON. tiktoken gives you the count; writing BPE gives you the count and the reasons.

The algorithm in plain terms

Byte Pair Encoding starts from bytes. Every byte value is a token, so your initial vocabulary is 256 entries and nothing is ever unknown. Count adjacent token pairs across the corpus, merge the most frequent pair into a new token, append it to the vocabulary, and repeat. After roughly 50K merges, frequent byte sequences (whole common words, ubiquitous subwords, common operator pairs in code) are single tokens, and the vocabulary is a compressed record of what the corpus says most often.

The vocab IS the algorithm. There is no separate model underneath; the merge list is the compression. Encoding a fresh string replays the merge list in the order the merges were learned; decoding reverses the expansion by substituting each merged token with its two children recursively. If you understand counting pairs and merging the winner, you understand the entire machine. Everything else is engineering around that core: efficient pair counting, caching, memory layout, and the pre-tokenizer in front.

One property follows directly and is worth internalizing: the vocabulary is greedy and local. Each merge is the best available compression step at the time it was made, not a global optimum. That is why the merge order matters and why replaying merges in the wrong order produces garbage. The tokenizer is an ordered procedure, not a lookup table, and treating it as a lookup table is the first misconception people bring to the exercise.

The GPT-2/4 regex pre-tokenization

Before merging begins, the text is split with the GPT-2/GPT-4 regex into chunks: roughly, contractions, runs of letters, numbers, punctuation with optional leading space, and runs of whitespace. This matters because merging across chunk boundaries would fuse halves of unrelated words into single tokens, which is compression in name but garbage in effect. The regex enforces that merges happen inside linguistically sane units.

The regex does more work than people expect. It is why "hello world" and "helloworld" tokenize differently, why a leading space changes the token stream, and why the same word can cost a different number of tokens depending on what precedes it. Contractions like "don't" get split into a letter-run and an apostrophe-run rather than one blob, which keeps apostrophe handling uniform across thousands of words. If a prompt's price or behavior changed when you rephrased one sentence, the pre-tokenizer is usually the reason, and now you know where to look.

Byte fallbacks

Because the base vocabulary is bytes, any byte sequence encodes. There is no unrepresentable input, ever: an emoji, a rare script, a corrupted download; all of it round-trips through the vocabulary. The cost is ugly splits on characters the corpus never showed. A rare glyph can shatter into several byte tokens, inflating sequence length and occasionally garbling generation on low-resource scripts. Byte fallback is a correctness guarantee, not an aesthetic one: nothing is ever impossible to encode, but nothing is promised to be cheap, either.

This is also why tokenizers trained predominantly on English-heavy corpora are measurably more expensive per sentence in other languages: bytes for unfamiliar scripts merge poorly, and you pay in length what the vocabulary never learned to compress. Script diversity in training data is not an ethics add-on; it is compression economics.

What a technical-corpus vocab buys

A vocabulary trained on technical corpora learns merges for the strings that dominate that text: identifiers, imports, common API names, ML and systems terminology. Domain terms become single tokens, so the same technical content fits into fewer tokens; I measured about 15% token reduction on technical text. That is directly cheaper calls and denser effective context, bought by no longer re-spelling the vocabulary of your domain one fragment at a time.

The catch is obvious and honest: a specialist vocabulary is worse on general text. It will mishandle content outside its corpus, and you will not notice until the bill or the quality regression shows up. Specialization is a bet on the workload, and it pays exactly when the workload is real. If your traffic is 80% code and ML prose, the bet is rational; if it is half customer support, keep the general tokenizer.

What it taught about LLM behavior

Token boundaries explain more mysterious model behavior than any other single concept I know:

• Long identifiers. A 40-character snake_case name fragments into chunks the model sees unevenly. The model does not see the identifier; it sees shards, which is why exact recall of long identifiers, hashes and UUIDs is unreliable no matter how good the underlying model is. If you need exact reproduction of opaque strings, put them in the prompt verbatim and keep them short, or do not expect verbatim recall.
• "Strawberry" counts fail. Character-level questions ask the model to count letters inside tokens it never encountered as letters. The r-r-r in "strawberry" may live inside one or two tokens, invisible to the model as a sequence of characters. Asking about letters is asking about an abstraction the model was never given, and no amount of reasoning fixes a missing layer.
• Leading spaces change meaning. " world" and "world" are different tokens with different embeddings. Prompts, code blocks and chat templates are sensitive to exact spacing for this reason, and template designers know it; inconsistent indentation in few-shot examples is not cosmetic.
• Numbers chunk strangely. Digits group into tokens of varying widths depending on position and locale conventions, which is one reason arithmetic on multi-digit numbers is shakier than arithmetic on single digits, and why digit-heavy tasks reward careful prompting.

The build itself, honestly scoped

The project is deliberately small enough to hold in your head: a pure Python trainer that counts adjacent pairs over a corpus, a merge loop that emits an ordered merge list, and encoders and decoders that replay it. The pre-tokenizer is the GPT-2/GPT-4 regex, not a hand-rolled approximation, because approximating it teaches you nothing and breaks parity with the reference implementation. The vocabulary was trained on technical corpora, which is where the 15% token reduction on technical text comes from, and the merge list landed at roughly 50K entries. Everything in the repo is inspectable; nothing is hidden behind a compiled extension or a training-framework dependency.

Honest trade-offs

This implementation is about 2x slower than tiktoken. It is pure Python and it shows; production tokenizers are Rust or Go for a reason, because tokenization sits on the critical path of every request in a high-throughput system. The exercise pays for itself in debugging time and in cost intuition, not in serving speed, and nothing here should be deployed where tokenizer latency matters. Build it to understand it; ship the fast one. The case study with the full write-up is at MinBPE on this site; the code is at TOKENIZER-FROM-SCRATCH.