Lesson 1
Tokenization
How AI reads text
🧩 How does AI read text?
AI doesn't understand letters directly. It breaks text into pieces — tokens.
Step 1
We start with the original text as one piece
Hello world!
💡 Key insight:
- •A token isn't necessarily a word. It can be part of a word or even a punctuation mark.
- •Common words = 1 token. Rare words = multiple tokens.
- •More tokens = more expensive AI request.