Small experiments / Token explorer
What does a token look like?
A little toy built out of curiosity. Explore the GPT-2 vocabulary one ID at a time, or turn a piece of text into tokens. This is not the tokenizer used by every modern AI model.
01 / Decode a token
One ID, one vocabulary entry.
Whitespace key: · space / → tab / ⏎ newline. Some vocabulary entries represent partial byte sequences.
02 / Encode some text
Words become a sequence.
Encoding happens in your browser. This tool does not send your text to a server.