Multimodal AI
Modern AI models process images, audio, and video alongside text. Images are split into patches, each patch embedded into a vector, and those vectors join the text token stream. The model learns to reason across modalities through contrastive pre-training (CLIP) and visual instruction tuning. The architecture is the same transformer — only the tokenizer changed.
The fridge photo
It looked at your photo. But everything so far has been about text.
A model reads tokens. So what does a photo turn into?
Cut into squares
The image is cut into a grid of small squares.
Each square becomes a list of numbers, the same kind a word gets: its spot on the map of meaning.
Squares join words
The squares line up with your words, and attention works across both.
So “cook” can look at the square with the eggs, the way “bank” looked at “river”.
Bigger image, more squares
A bigger image means more squares, and more squares means more tokens.
Trust what it sees
It can describe things that aren’t there, just like with text. Which answers need checking?
Tap the box where each question belongs.
Same machine
Sound works the same way: cut into short slices, turned into tokens, read with attention.
Different inputs, one idea. Turn it into tokens and let attention connect them.
Everything becomes tokens
Images and sound become tokens too, so one model can read them alongside your words.
Models that take more than text are called multimodal. The image squares are called patches.