AI Token Boundary Colorizer

Tokenizer boundaries are invisible in plain text but they decide cost, context limits, and even model behavior. This colorizer encodes your text with the OpenAI BPE tokenizers (o200k_base for GPT-4o-class models, cl100k_base for GPT-4/3.5) and renders every token with its own background color, plus token IDs from the vocabulary. Statistics show total tokens, characters, and the characters-per-token ratio so you can estimate prompt costs and see why unusual spellings or code fragments tokenize inefficiently. Tokenizer tables load lazily; all encoding happens locally and nothing is uploaded.