tokenizers.si
A tokenizer turns raw text into the integer tokens a language model actually reads, and turns its output back into text. Every prompt, every context window, and every API bill is counted in them.
Why "tokenizers"
No language model reads text. Before a single parameter fires, a tokenizer (BPE, WordPiece, SentencePiece, tiktoken) cuts the input into subword pieces and maps each one to an integer ID. Context limits, latency, and pricing are all denominated in the tokens it produces.
It is the one component every model shares and none can skip. As models go multimodal, the same idea extends to image patches, audio frames, and actions: everything becomes tokens. The term names the interface between human language and machine intelligence, not a passing implementation detail.
Contact
tokenizers.si is for sale, privately. Contact for registration history and transfer terms.
The portfolio
misalignment.si, plus nine names for the layers of modern infrastructure, each offered privately.