tokenization

Tokenization: how an AI chops your words into pieces it can read Lesson

A language model never sees letters or words. It sees tokens, chunks of text turned into numbers. How that chopping works quietly shapes cost, context limits, multilingual fairness, and why models stumble on spelling and arithmetic.