Lesson 6Architecture

Multimodal RAG

Search across text and images

The Problem: Text RAG fails with visual content (diagrams, tables, photos, charts). When documents contain images, traditional text-only RAG cannot find or reason about visual information.

The Solution: Three Architectures of Multimodal RAG

Multimodal RAG extends traditional retrieval-augmented generation to handle images alongside text. There are three main architectures: CLIP embeddings encode images directly into vectors for similarity search, LLM-generated summaries convert images into text descriptions that can be embedded and searched normally, and Multi-vector approaches like ColPali produce per-token embeddings for both text and image patches, preserving layout and visual structure. Each architecture trades off retrieval accuracy, indexing speed, and the ability to answer text-based questions about image content.

Think of it like a librarian who can search both by text descriptions and by looking at actual pictures in books:

1. CLIP Embeddings: Image goes directly into CLIP encoder, producing a vector for similarity search. Fast but can't answer text questions about image content.
2. LLM-Generated Summaries: LLM describes each image in text, then text is embedded normally. Searchable by queries but loses visual detail in translation.
3. Multi-Vector (ColPali): Late interaction model produces per-token embeddings for both text and image patches. Best accuracy, preserves layout and visual info.
4. Choosing the right approach: CLIP for image-to-image search, LLM summaries for text-based Q&A about images, ColPali for document understanding with layout-sensitive retrieval.

Where Multimodal RAG Matters

Technical Documentation: Diagrams and schematics alongside text explanations, searchable by both visual similarity and text queries.
Medical Records: X-rays, MRI scans referenced in patient notes, enabling search like "show me cases with similar chest X-ray findings".
E-commerce: Product photos combined with descriptions for visual search: "find similar looking products" or "red dress with floral pattern".
Legal Discovery: Contracts with stamps, signatures, handwritten notes that affect interpretation alongside printed text.

Fun Fact: ColPali (2024) from Illuin Technology showed that a single vision model can match or beat complex OCR+text pipelines for document retrieval, while being 4x faster to index. The key insight: visual tokens preserve layout information that gets lost in text extraction.

Try It Yourself!

Explore the visualization below to see how each architecture processes a document with images — from chunking to retrieval to generation.

Try it yourself

Interactive demo of this technique

Technique Comparison

Demo Mode

Pre-recorded responses

TaskBeginnerAnalysis

Create a quality text description of a diagram for RAG indexing

Without technique

Prompt

Describe this diagram. [Diagram: bar chart of quarterly sales — Q1: $2.1M, Q2: $3.4M, Q3: $2.8M, Q4: $4.2M]

Response

The diagram shows quarterly sales. Sales are growing.

Tokens:290/18

Time:1500ms

Quality:

With multimodal-rag

With technique

Prompt

Create a detailed text description of this diagram for indexing in a RAG system. The description must be searchable — a user should be able to find it with a text query. Include: 1. Chart type and what it shows 2. All data values (exact numbers) 3. Trends and patterns 4. Key takeaways 5. Search tags [Diagram: bar chart of quarterly sales — Q1: $2.1M, Q2: $3.4M, Q3: $2.8M, Q4: $4.2M]

Response

Type: Bar chart Topic: Quarterly sales for the year

Data:

Q1: $2.1M (minimum)
Q2: $3.4M (+62% vs Q1)
Q3: $2.8M (-18% vs Q2)
Q4: $4.2M (+50% vs Q3, maximum)

Trends: Overall growth with Q3 dip. Best quarter — Q4 ( $4.2M). Annual total:$ 12.5M.

Takeaways: Seasonal Q3 dip, strong Q4 finish. Q1→Q4 growth: +100%.

Tags: sales, quarterly, bar chart, revenue, seasonality, growth

👁️Structuring description for maximum search relevance

🧠Exact numbers enable finding the chart with queries like "Q4 sales" or "$4.2M"

✅Tags expand search coverage — synonyms and key concepts

Tokens:400/180

Time:3500ms

Quality:

Why this works

In the "LLM summaries" architecture, description quality = search quality. Structure, exact data, and tags turn an image into a searchable artifact.

1 / 2

Practice Challenges

Create a free account to solve challenges

3 AI-verified challenges for this lesson

Related lessons:Hallucinations Rag

This lesson is part of a structured LLM course.

My Learning Path

The Solution: Three Architectures of Multimodal RAG

Think of it like a librarian who can search both by text descriptions and by looking at actual pictures in books:

1. CLIP Embeddings: Image goes directly into CLIP encoder, producing a vector for similarity search. Fast but can't answer text questions about image content.
2. LLM-Generated Summaries: LLM describes each image in text, then text is embedded normally. Searchable by queries but loses visual detail in translation.
3. Multi-Vector (ColPali): Late interaction model produces per-token embeddings for both text and image patches. Best accuracy, preserves layout and visual info.
4. Choosing the right approach: CLIP for image-to-image search, LLM summaries for text-based Q&A about images, ColPali for document understanding with layout-sensitive retrieval.

Where Multimodal RAG Matters

Technical Documentation: Diagrams and schematics alongside text explanations, searchable by both visual similarity and text queries.

Medical Records: X-rays, MRI scans referenced in patient notes, enabling search like "show me cases with similar chest X-ray findings".

E-commerce: Product photos combined with descriptions for visual search: "find similar looking products" or "red dress with floral pattern".

Legal Discovery: Contracts with stamps, signatures, handwritten notes that affect interpretation alongside printed text.