Question Answering

The Solution: An Expert Hotline

Question answering (QA) is the task of taking a natural-language question and returning a direct, correct answer instead of a list of links or a long document for the user to read. A large language model does this by encoding the question and any supporting text into vectors, attending over that context, and generating the answer token by token. The crucial design decision is where the facts come from, and that splits QA into two modes that behave very differently in production.

Open-book vs. closed-book

In closed-book QA the model answers from its parameters alone — the knowledge baked in during training. This is fast and needs no extra infrastructure, and it works well for stable general knowledge (“What is the capital of France?”). But it has a hard ceiling: the model cannot know anything after its training cutoff, it has no access to your private data, and when it is unsure it tends to hallucinate — stating a wrong fact with full confidence. In open-book QA you instead use RAG (retrieval-augmented generation): you convert documents into embeddings, store them in a vector index, retrieve the passages most relevant to the question, and paste them into the prompt as context. The model then answers from those passages and cites them — a property called grounding. Open-book is the right choice whenever answers must be current, domain-specific, or verifiable.

Tradeoffs, pitfalls, and a worked example

Each mode has costs. Closed-book is cheap but unreliable for facts; open-book is reliable but only as good as your retrieval — if the right passage is never retrieved, the model either guesses or, ideally, says “I don’t know.” Common pitfalls are noisy chunks that drown the real answer, questions that need information from several documents (multi-hop), and answers that sound grounded but quietly mix in unsupported claims. Mitigations include better chunking, re-ranking retrieved passages, and asking the model to quote its sources. Chain-of-thought prompting helps with multi-step questions in both modes. For example, ask “What were the main causes of the 2008 financial crisis?”: a closed-book model produces a plausible essay you must take on trust, while an open-book system retrieves passages on subprime lending, securitization, and rating-agency failures, and answers with citations you can click and check — which is exactly why production QA over real knowledge almost always uses retrieval.

Where Is This Used?

• Customer Support: Answering FAQs automatically
• Search Engines: Direct answers instead of just links
• Education: Tutoring and homework help
• Enterprise: Internal knowledge base queries
• Closed-Book Risk: When the model answers from training data alone, it may hallucinate facts, provide outdated info, or confidently state something wrong — always verify critical answers against sources

Try It Yourself!

Use the interactive example below to ask questions and see how AI formulates accurate, helpful responses.