Building a GPT From Scratch: What Tokenization Teaches You

Tokenization looks like a preprocessing detail. It quietly explains a surprising number of the strange things language models do.

Hafeez Baig•Jun 16, 2026•10 min read

Introduction

A language model cannot read. It has no notion of letters, words, or spaces. What it has is a large table of numbers and a stack of matrix multiplications. Text has to become integers before any of that machinery can run.

That conversion step is tokenization, the first thing you have to build if you set out to write a small GPT from first principles as a learning exercise. It is also the part most people skim. You import a tokenizer, you get numbers back, you move on to attention and training loops.

Skipping it is a mistake. Tokenization is a design decision with consequences that reach all the way to the behaviour users complain about. Why a model miscounts the letters in a word. Why it fumbles arithmetic it clearly "knows". Why a leading space changes an answer. Why the same prompt costs more in one language than another. All of that is downstream of how you turned text into integers.

1. Why Integers Are Non Negotiable

Start from the mechanics. A transformer's first layer is an embedding table: a big matrix where row i holds a vector of numbers representing item i. To fetch a vector, you need a row index. An index is an integer.

So the pipeline is fixed:

Split text into units called tokens.
Map each token to an integer ID via a lookup table called the vocabulary.
Use that integer to pull a row from the embedding table.
Do everything else.

The model never sees your string. It sees a list like [1212, 318, 257, 1332]. Every question about what the model can perceive reduces to a question about what survived step 1.

That is the framing worth holding onto. Tokenization decides what the model is capable of seeing. Anything below the token boundary is not hidden from the model, it is absent.

2. The Naive Options, And Why Both Fail

There are two obvious ways to split text. Both are instructive, and both are unusable at scale.

Character level. One token per character. The vocabulary is tiny, maybe a hundred or so entries for English text plus punctuation. Nothing is ever unknown, because every string is a sequence of characters you already have. It is elegant, and it is what most teaching implementations start with.

The problem is sequence length, and it matters more than it sounds, because self attention compares every token to every other token. The cost of that comparison grows with the square of the sequence length. Double the sequence, roughly quadruple the attention cost. Character level tokenization makes your sequences as long as they can possibly be, which puts your compute budget in the worst place it can be. It also forces the model to spend capacity learning that c, a, t in that order means something, before it can learn anything about cats.

Word level. One token per word. Sequences get short, and each token carries real meaning out of the box. This looks like the obvious fix. It breaks in two ways:

Vocabulary explosion. Real text has an enormous number of distinct words once you count inflections, names, typos, code identifiers, and URLs. Every entry needs a row in the embedding table and a slot in the output layer, so vocabulary size is a direct multiplier on model parameters at both ends.
Out of vocabulary. This is the fatal one. Any word not in your list has no ID. Not a bad ID: none. A new product name, a rare surname, a misspelling, and your tokenizer has nothing to return. The usual patch is a single <UNK> token, which throws the information away and tells the model only that "something was here".

Neither extreme works. One gives you a sequence problem, the other a vocabulary problem.

Approach	Vocabulary size	Sequence length	Unknown words
Character	Very small	Very long	Impossible
Word	Very large	Short	Constant problem
Subword	Tunable	Moderate	Avoidable

3. Subwords, And The Idea Behind BPE

The compromise is to let token boundaries be learned from the data rather than fixed by a rule.

The intuition is simple. Frequency should buy compression. A word that appears constantly, like the, deserves its own token: one entry is cheap and it saves you tokens on every sentence. A word that appears once in ten million, like antidisestablishmentarianism, does not deserve a slot. It can be spelled out of smaller pieces you already have.

Byte Pair Encoding, or BPE, is the algorithm that finds that split automatically. It began as a data compression technique. The training procedure is short enough to state in full:

Start with a base vocabulary of the smallest units you allow: individual characters, or raw bytes.
Represent your training corpus as sequences of those units.
Count every adjacent pair of units across the corpus.
Find the most frequent pair. Merge it into a single new unit and add it to the vocabulary.
Repeat from step 3 until the vocabulary reaches your target size.

That is the whole thing. Every merge is one new vocabulary entry, so you control the final size by controlling how many merges you run. The output of training is not just a word list. It is an ordered list of merge rules. Encoding new text means applying those same merges, in the same order, to the new string. The order matters: later merges are built out of the results of earlier ones.

The emergent behaviour is the good part. Nobody tells BPE that ing is a suffix or that un is a prefix. Those pieces get merged early because they are common, and they become tokens. Common whole words collapse into single tokens because their internal pairs kept winning the frequency count. Rare words never accumulate enough frequency to merge fully, so they stay as several pieces. The tokenizer discovers a rough morphology of the language just by counting pairs.

4. A Toy Merge Loop

The mechanism is easier to trust once you have seen it as code. This is a deliberately small sketch, not an efficient implementation, but the merge loop is the real one.

python

from collections import Counter

def get_pair_counts(ids):
    """Count how often each adjacent pair appears."""
    return Counter(zip(ids, ids[1:]))

def merge(ids, pair, new_id):
    """Replace every occurrence of `pair` with `new_id`."""
    out, i = [], 0
    while i < len(ids):
        if i < len(ids) - 1 and (ids[i], ids[i + 1]) == pair:
            out.append(new_id)
            i += 2
        else:
            out.append(ids[i])
            i += 1
    return out

def train_bpe(text, vocab_size):
    # Base vocabulary: the 256 possible byte values.
    ids = list(text.encode("utf-8"))
    merges = {}

    for new_id in range(256, vocab_size):
        counts = get_pair_counts(ids)
        if not counts:
            break
        top_pair, freq = counts.most_common(1)[0]
        if freq < 2:
            break  # nothing left worth merging
        ids = merge(ids, top_pair, new_id)
        merges[top_pair] = new_id

    return ids, merges

Two things are worth noticing. First, merges is the artefact. To encode unseen text later, you re-apply these rules in insertion order. To decode, you invert them recursively until you are back at bytes, then decode the bytes as UTF-8.

Second, look at the starting line: text.encode("utf-8"). The base units are bytes, not characters. That choice is doing more work than it appears to.

5. Byte Level BPE, And The End Of Unknown Words

A byte is a number from 0 to 255. There are exactly 256 of them, and every digital text, in every language, with every emoji and every corrupted character, is ultimately a sequence of bytes.

So if your base vocabulary is all 256 byte values, nothing can ever be out of vocabulary. There is no string you can type that the tokenizer cannot represent. Unfamiliar text is not rejected, it just falls back to more tokens: unusual sequences never got merged, so they come through in small pieces, possibly as individual bytes.

This is a real change in failure mode. Word level tokenizers fail hard on unknown input: information is destroyed and replaced with <UNK>. Byte level BPE fails softly: the information is fully preserved, it simply costs more tokens to carry. It also explains why models handle code, mixed scripts, and typos better than a fixed vocabulary ever could. Nothing is off the map. Some regions are just more expensive to travel.

6. The Payoff: Strange Behaviours That Are Really Tokenization

This is where the preprocessing detail starts explaining the headlines.

Counting letters and reversing strings. Ask a model how many times a letter appears in a word and it may get it wrong. This is not a reasoning failure so much as a perception one. If a word arrived as a single token, the model received one integer. The letters are not in there. The embedding has absorbed statistical information about spelling from training text, which is why models often get it right, but that is inference, not direct inspection. It is closer to recalling how a word is spelled than to looking at it. Reversing a string has the same shape: character level manipulation of something the model holds at token granularity.

Unreliable arithmetic. Numbers are text, so numbers get tokenized, and how they split depends on what was frequent in the training corpus rather than on anything mathematical. To illustrate the shape of the problem: imagine 1234 splitting as 12 and 34 in one context and 1 and 234 in another. Column by column addition is hard to learn when the columns are not stable units of input. The digits are all present, but the grouping the algorithm needs is not the grouping the model receives.

The leading space. In most byte level tokenizers, the space attaches to the front of the following word. So " hello" and "hello" are different tokens with different IDs and different embeddings. A trailing space in your prompt is not cosmetic. It changes the last token, and the model's next prediction is conditioned on exactly that token.

Capitalisation and position. For the same reason, hello, Hello, HELLO, and " Hello" can all be distinct tokens. The model learns their relationship from data rather than knowing it structurally. Usually that works. Occasionally a capitalised variant behaves slightly differently from the lowercase one, for no reason visible in your prompt.

Language cost asymmetry. Tokenizer merges reflect the training corpus, and those corpora have historically been English heavy. So English words merge into few tokens, while text in less represented languages, especially in non Latin scripts, tends to fragment into many more tokens for the same meaning.

This is not only an efficiency footnote. Tokens are the billing unit and the context limit. If the same sentence costs several times more tokens in one language than another, then some users pay more, hit context limits sooner, and get slower responses, for reasons that have nothing to do with what they asked. It is a fairness problem hiding inside a preprocessing step.

7. What This Means If You Build On LLM APIs

Three practical consequences.

Token count is not word count. A rough words to tokens ratio is fine for a back of envelope estimate in English prose and unreliable for anything else. Code, JSON, numeric data, names, and non English text all deviate, sometimes badly. My recommendation is straightforward: measure, do not guess. Run your actual prompts through the actual tokenizer for the model you are calling and look at the number.

Truncation is not where you think it is. If you trim context by characters or words, you are trimming against the wrong ruler. Budget in tokens.

Prompt spacing and formatting are inputs. Since whitespace changes tokens, treat prompt templates as code. A stray trailing space is a real change.

8. Vocabulary Size Is A Trade Off, Not A Setting

One last design point, because building a small model from scratch forces you to pick a number.

A larger vocabulary compresses text better. Fewer tokens per document means shorter sequences, cheaper attention, and more content fitting in a context window. The cost is that the embedding table and the output layer both grow with vocabulary size, and rare tokens get fewer training examples each, so their embeddings are learned less well. A smaller vocabulary keeps those layers compact and gives every token plenty of training signal, at the price of longer sequences, which attention punishes quadratically.

There is no correct answer, only a balance point that depends on your corpus, your compute, and your context length target. What matters is knowing it is a trade off you are making, rather than a default you inherited.

Conclusion

Tokenization is the least glamorous part of a language model and one of the most explanatory. It is the layer where text stops being text and becomes indices, and everything the model can and cannot perceive is decided right there.

Once you have built the merge loop yourself, a set of separate mysteries collapses into one cause. Letter counting, arithmetic, leading spaces, capitalisation quirks, language cost gaps: these are not deep reasoning failures. They are consequences of a compression algorithm chosen because it makes sequences shorter and vocabularies finite, and which happens to be blind to the structure some tasks require.

That is the real lesson of building a GPT from first principles. The parts you were tempted to skip are usually the parts that explain the behaviour you cannot otherwise account for.

Written while building a small GPT from scratch, mostly to find out what I had been importing.