Why AI hallucinates, and the fact-checking habit that keeps you safe

Sooner or later everyone using an AI assistant gets handed a fact that is stated beautifully and is simply not true — an invented quote, a case that does not exist, a statistic that sounds right. This is not a rare bug. It is the predictable consequence of how the models are built, and the only defence is a habit, not trust.

Why it happens

As explained in how these models work, a model predicts plausible next words; it is optimised for text that fits, not text that is true. When it lacks a real answer, the gap does not produce silence — it produces the most plausible-looking fill. Fluency and confidence are by-products of the mechanism, not signals of accuracy, which is exactly why a hallucination is so hard to spot: it reads like everything around it.

The fact-checking habit

Treat specific, checkable claims — names, dates, numbers, quotes, citations, legal or medical specifics — as unverified until you confirm them against a primary source. The model is excellent at finding the shape of an answer and at explaining a topic; it is unreliable as the authority for a fact. So use it to draft, summarise and explain, then verify the load-bearing details yourself. If it cites a source, open the source — invented references are common precisely because a plausible-looking citation is what the predictor would produce.

Where the stakes are highest

The cost of a hallucination scales with the decision it feeds. For a casual summary, low stakes; for something legal, medical, financial, or published under your name, a confident falsehood can do real damage — which is why this desk's own rule is no claim without a dated, checkable source, the standard in the methodology. Pair the habit with the privacy discipline in what not to paste into AI: verify what comes out, and be careful what goes in. Used that way, the model is a superb thinking partner; trusted as an oracle, it is a liability.