{"title":"Why Do LLMs Hallucinate?","description":"","content":"Ask ChatGPT about a research paper. It gives you a title, author names, and a publication year. Everything sounds plausible.\n\nThere is just one problem: the paper does not exist. The authors never wrote it. The citations are fabricated.\n\nThis is a hallucination.\n\nLLMs regularly generate confident, plausible-sounding, completely false information. They invent people, fabricate events, cite nonexistent sources, and state incorrect facts with the same fluency they use for accurate information.\n\nFor developers building AI products, hallucinations are not just an annoyance. They are a fundamental problem that can break user trust, cause real harm, and make certain applications unsafe.\n\nUnderstanding why hallucinations happen is the first step to building systems that minimize them.\n\n---\n\n# What is a Hallucination?\n\nA hallucination is when an LLM generates content that is factually incorrect, nonsensical, or completely fabricated, but presents it with the same confidence as accurate information.\n\nExamples:\n\n#### **Fabricated citations:**\n\n\n```shell\nUser: What papers has Yann LeCun written about capsule networks?\n\nLLM: Yann LeCun published \"Dynamic Capsule Routing for Visual\n Recognition\" in NeurIPS 2019, which introduced...\n```\n\n\nThis paper does not exist. LeCun did not write it. The LLM invented it.\n\n#### **Fictional entities:**\n\n\n```shell\nUser: Tell me about the Zelkova Bridge in Prague.\n\nLLM: The Zelkova Bridge is a historic stone bridge in Prague,\n built in 1847 during the reign of Ferdinand I...\n```\n\n\nThere is no Zelkova Bridge in Prague. The LLM created a detailed but fictional description.\n\n#### **Incorrect facts:**\n\n\n```shell\nUser: What is the capital of Australia?\n\nLLM: The capital of Australia is Sydney.\n```\n\n\nIt is Canberra. Sydney is the largest city but not the capital.\n\nThe danger is not just that LLMs make mistakes. It is that they make mistakes with confidence. There is no hesitation, no uncertainty markers, no \"I'm not sure but...\" prefix.\n\nThey sound right even when they are wrong.\n\n---\n\n# LLMs Are Pattern Matchers, Not Knowledge Bases\n\nTo understand hallucinations, you need to understand what an LLM actually does.\n\nAn LLM is a next-token prediction machine. Given a sequence of tokens, it predicts what token is most likely to come next based on patterns it learned during training.\n\n\n```shell\nInput: \"The capital of France is\"\nOutput: \"Paris\" (predicted with high probability)\n\nInput: \"The author of 'On the Origin of Species' is\"\nOutput: \"Charles Darwin\" (predicted based on training patterns)\n```\n\n\nWhen the LLM encounters a question, it does not:\n\n- Look up the answer in a database\n- Verify facts against a knowledge source\n- Check whether it actually knows the answer\n\nInstead, it asks: \"Based on patterns I learned, what text would typically follow this input?\"\n\n\n[Embed: https://link.excalidraw.com/readonly/LkzAH9pLa3nAWns496Tz](https://link.excalidraw.com/readonly/LkzAH9pLa3nAWns496Tz)\n\n\nThis is the fundamental issue: the training objective is to predict plausible text, not to verify truth.\n\n---\n\n# Why Hallucinations Are Inevitable\n\nSeveral factors make hallucinations an inherent property of LLMs:\n\n### 1. The Training Objective Rewards Fluency, Not Truth\n\nDuring pretraining, the model learns to predict the next token. The loss function penalizes wrong token predictions, not factual inaccuracies.\n\n\n```shell\nTraining example: \"The Eiffel Tower is located in ___\"\nCorrect next token: \"Paris\"\nLoss: Prediction error (did it guess \"Paris\"?)\n\nTraining does NOT check:\n- Is this claim factually true?\n- Does the model actually \"know\" this?\n- Can this be verified?\n```\n\n\nThe model learns to produce text that looks like the training data, not text that is verified to be true.\n\n### 2. Knowledge Is Stored as Compressed Patterns\n\nLLMs do not memorize facts like a database. They compress patterns from billions of text examples into model parameters.\n\nThink of it like a lossy compression algorithm:\n\n\n```shell\nOriginal: Trillions of tokens from the internet\n |\n v\nCompression: 100-400 billion parameters\n |\n v\nResult: Approximate, statistical representation\n (not a perfect copy)\n```\n\n\nWhen you ask the model something, it reconstructs information from these compressed patterns. Sometimes the reconstruction is accurate. Sometimes it produces plausible-sounding noise.\n\nThis is like asking someone to recall a book they read once, years ago. They remember the gist but might confuse details, merge memories, or fill gaps with plausible inventions.\n\n### 3. The Model Has No Mechanism to Say \"I Don't Know\"\n\nLLMs are trained on text written by humans who generally write with confidence. Academic papers, Wikipedia articles, news stories: they all state claims assertively.\n\nThe training data rarely contains examples like:\n\n- \"I'm not sure about this, but...\"\n- \"I don't have reliable information on...\"\n- \"This is outside my knowledge\"\n\nSo the model learns to always produce confident-sounding output.\n\n\n```shell\nTraining teaches:\n\nQuestion --> Confident Answer (seen millions of times)\n\nTraining rarely shows:\n\nQuestion --> \"I don't know\" (seen rarely)\n```\n\n\nEven when the model has weak signal about a topic, it produces output with the same confident tone it uses for well-established facts.\n\n### 4. Post-Training Reinforces Confidence\n\nDuring fine-tuning (SFT) and RLHF, models are trained to be helpful assistants. The training data consists of confident, well-structured responses.\n\n\n```shell\nSFT Training Example:\n\nUser: What is quantum entanglement?\nAssistant: Quantum entanglement is a phenomenon where two particles\n become correlated such that the quantum state of one\n instantly influences the other, regardless of distance...\n```\n\n\nThe model learns: fluent, confident, complete answers get rewarded. Hedging and uncertainty do not appear in the training signal.\n\nSo even when the model is uncertain internally (low probability across many options), it still outputs text that sounds certain.\n\n### 5. Rare or Ambiguous Topics Trigger Hallucinations\n\nHallucinations are most common when:\n\n- The topic appeared rarely in training data\n- The question is ambiguous or underspecified\n- The expected output format requires specific details (names, dates, citations)\n\n\n```shell\nHigh Hallucination Risk:\n\n\"Who was the third person to climb K2 in winter?\"\n\"What did Dr. James Morton publish in 1987?\"\n\"List five papers on neural architecture search from 2015\"\n\nLow Hallucination Risk:\n\n\"What is photosynthesis?\"\n\"Explain how TCP/IP works\"\n\"What is the capital of Japan?\"\n```\n\n\nFor well-documented topics, the model has seen the same fact stated many times, so the pattern is strong. For obscure topics, the model fills in gaps with statistically plausible (but possibly fabricated) details.\n\n---\n\n# Types of Hallucinations\n\nNot all hallucinations are the same. Understanding the types helps diagnose and mitigate them.\n\n### 1. Factual Hallucinations\n\nThe model states incorrect facts about the real world.\n\n\n```shell\nUser: When was the Golden Gate Bridge built?\nLLM: The Golden Gate Bridge was completed in 1932.\n\nActual: 1937\n```\n\n\nThese occur when the model's compressed knowledge is inaccurate or when it confuses similar facts.\n\n### 2. Fabricated Entities\n\nThe model invents people, places, products, or events that do not exist.\n\n\n```shell\nUser: Tell me about the Xerxes Protocol in cryptography.\nLLM: The Xerxes Protocol is a post-quantum cryptographic scheme\n developed at Stanford in 2019...\n```\n\n\nThere is no Xerxes Protocol. The model generated a plausible-sounding but fictional description.\n\n### 3. False Citations\n\nThe model produces references to papers, books, or sources that do not exist.\n\n\n```shell\nUser: Cite a paper on attention mechanisms.\nLLM: See \"Attention Dynamics in Deep Networks\" by Zhang et al.,\n published in ICML 2018.\n```\n\n\nThis is particularly dangerous in academic or professional contexts where users expect verifiable sources.\n\n### 4. Contextual Hallucinations\n\nThe model contradicts information provided in the context or generates content inconsistent with the conversation.\n\n\n```shell\nUser: The meeting is on Tuesday at 3pm.\nUser: What time is the meeting?\nLLM: The meeting is scheduled for Wednesday at 2pm.\n```\n\n\nThe model fails to ground its response in the provided context.\n\n### 5. Reasoning Hallucinations\n\nThe model makes logical errors while appearing to reason correctly.\n\n\n```shell\nUser: If all roses are flowers, and some flowers fade quickly,\n do all roses fade quickly?\nLLM: Yes, since roses are flowers and flowers fade quickly,\n all roses must fade quickly.\n```\n\n\nThis is a logical fallacy (affirming the consequent), but the model presents it confidently.\n\n---\n\n# How to Reduce Hallucinations\n\nHallucinations cannot be eliminated entirely, but they can be significantly reduced with the right techniques.\n\n### 1. Retrieval-Augmented Generation (RAG)\n\nInstead of relying solely on the model's internal knowledge, retrieve relevant documents and include them in the prompt.\n\n#### Without RAG:\n\n\n```shell\nUser: What is our company's refund policy?\nLLM: [Hallucinates a plausible policy]\n```\n\n\n#### With RAG:\n\n\n```shell\n1. Retrieve: Search company docs for \"refund policy\"\n2. Inject: Add the actual policy text to the prompt\n3. Generate: LLM answers based on retrieved context\n\nLLM: According to your policy document, refunds are available\n within 30 days of purchase with original receipt...\n```\n\n\nRAG grounds the model's responses in actual data, dramatically reducing fabrication.\n\n\n[Embed: https://link.excalidraw.com/readonly/MFZ3bmEDOteHH0DqxEct](https://link.excalidraw.com/readonly/MFZ3bmEDOteHH0DqxEct)\n\n\n### 2. Lower Temperature\n\nTemperature controls randomness in token selection. Lower temperature means the model picks higher-probability tokens more consistently.\n\n\n```shell\nTemperature = 0.0: Always pick the most likely token (deterministic)\nTemperature = 0.7: Balance between creativity and consistency\nTemperature = 1.0+: More random, more creative, more hallucinations\n```\n\n\nFor factual tasks, use low temperature (0.0-0.3). For creative tasks, higher temperature is acceptable but increases hallucination risk.\n\n### 3. Explicit Uncertainty Instructions\n\nPrompt the model to express uncertainty when appropriate.\n\n\n```shell\nSystem prompt:\n\"If you are not confident in an answer, say 'I'm not certain about\nthis' or 'I don't have reliable information on this topic.'\nDo not fabricate information.\"\n```\n\n\nThis does not guarantee compliance, but it helps shift the model's behavior.\n\n### 4. Tool Use and Grounding\n\nLet the model call external tools instead of generating answers from memory.\n\n\n```shell\nInstead of:\nUser: What is the current stock price of Apple?\nLLM: [Guesses or uses outdated training data]\n\nWith tool use:\nUser: What is the current stock price of Apple?\nLLM: [Calls stock API]\nLLM: According to the latest data, Apple (AAPL) is trading at $178.23.\n```\n\n\nTools provide real-time, verifiable information. The model orchestrates but does not fabricate.\n\n### 5. Citation Requirements\n\nAsk the model to cite sources and verify those citations exist.\n\n\n```shell\nPrompt: \"Provide your answer with specific citations.\n Only cite sources you are certain exist.\"\n\nPost-processing: Verify that cited URLs/papers actually exist\n before showing to user.\n```\n\n\nThis adds a verification layer that catches fabricated references.\n\n### 6. Structured Output with Validation\n\nUse structured output formats (JSON, schemas) with validation.\n\n\n```shell\nInstead of:\n\"Generate a user profile\"\n\nUse:\n\"Generate a user profile as JSON matching this schema:\n{name: string, email: valid_email, age: positive_integer}\"\n\nThen validate the output programmatically.\n```\n\n\nStructured outputs are easier to validate and constrain.\n\n### 7. Human-in-the-Loop for High-Stakes Decisions\n\nFor critical applications, do not let the LLM be the final authority.\n\n\n[Embed: https://link.excalidraw.com/readonly/jrgTxPPr6GLVWGEiRNbG](https://link.excalidraw.com/readonly/jrgTxPPr6GLVWGEiRNbG)\n\n\nUse LLMs for drafts, suggestions, and acceleration, but require human verification before consequential actions.\n\n---\n\n# Why Hallucinations May Never Be Fully Solved\n\nSome researchers believe hallucinations are a fundamental property of how LLMs work, not a bug to be fixed.\n\nHere is why:\n\n#### **1. The training objective creates them**\n\nNext-token prediction rewards generating plausible text. As long as this is the objective, the model will sometimes generate plausible-but-false text.\n\n#### **2. Perfect knowledge is impossible**\n\nNo model can know everything perfectly. There will always be gaps, and the model will always try to fill them.\n\n#### **3. Language itself is ambiguous**\n\nMany questions have multiple valid interpretations. The model picks one and runs with it, even when it should ask for clarification.\n\n#### **4. The boundary between memory and generation is blurry**\n\nLLMs do not clearly separate \"things I know\" from \"things I'm generating.\" It is all token prediction.\n\nThis does not mean we should give up. It means we should:\n\n- Design systems that account for hallucinations\n- Use LLMs as tools, not as sources of truth\n- Combine LLMs with verification, retrieval, and human oversight\n\n---\n\n# Key Takeaways\n\n- **Hallucinations are confident false outputs** that sound just as fluent as accurate responses.\n- **The root cause is the training objective:** LLMs predict plausible text, they do not verify truth.\n- **Contributing factors:**\n - Lossy compression of knowledge\n - No internal \"I don't know\" mechanism\n - Training data rewards confidence\n - Rare topics have weak signal\n- **Types of hallucinations:**\n - Factual errors\n - Fabricated entities\n - False citations\n - Contextual contradictions\n - Reasoning errors\n- **Mitigation strategies:**\n - RAG (ground responses in retrieved documents)\n - Lower temperature for factual tasks\n - Explicit uncertainty prompting\n - Tool use instead of memory\n - Citation verification\n - Human-in-the-loop for critical decisions\n\n**Hallucinations may be inherent** to how LLMs work. Design systems that account for them rather than assuming they can be eliminated.\n\n---\n\n# The Bottom Line\n\nLLMs do not know things the way humans do. They predict statistically likely text based on patterns in their training data.\n\nWhen they have seen a fact stated clearly and repeatedly, they reproduce it accurately. When the signal is weak, ambiguous, or absent, they fill the gap with plausible fabrication.\n\nUnderstanding this is essential for building reliable AI systems.\n\nDo not ask: \"Is the LLM right?\"\n\nAsk: \"How can I verify this, and what happens if it's wrong?\"\n\n---\n\n# Further Reading\n\n- [Survey of Hallucination in Natural Language Generation](undefined) - Comprehensive academic overview\n- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](undefined) - Original RAG paper\n- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](undefined) - Benchmark for hallucination\n- [Do Large Language Models Know What They Don't Know?](undefined) - Research on LLM calibration","pageType":"ai-engineering"}

Why Do LLMs Hallucinate?

Ashish Pratap Singh

Get Premium