CiteOwl

Blog/Getting citations right

Why AI makes up citations

The CiteOwl Team· Updated 20 September 2026· 13 min read

Because it is writing a citation, not looking one up. A model predicts the words that usually follow a request for a reference, and a well-formed citation is exactly that kind of likely sequence. So the author is half-familiar and the DOI has the right shape. We resolved 218 model-written DOIs against CrossRef and one in five from the best model was unusable. Run the list through the free citation checker before you use it.

Prediction, not retrieval

A language model is a next-word predictor. It learned from an enormous amount of text how sentences go, reference lists included, so it knows the shape of a citation perfectly. Surname, initial, year, title, journal, volume, pages, DOI. When you ask for one it writes the sequence most likely to follow your request. It does not open a catalogue and copy an entry down, because there is no catalogue in there to open.

The University of Southern California's graduate writing support program states it harder than we would, in a post titled AI cannot cite anything. A chatbot "does not draw claims from sources. It constructs plausible claims." Its references "are not roadmaps to find a source. They are plausible constructions of what a reference might look like." The model never met your paper as a document. It met text about it, in fragments, mixed in with everything else it read.

Duke University's librarians reached the same place in March 2023, when the first fake references started turning up at reference desks. The tool's "core strength lies in recognizing language patterns, not in reading and analyzing lengthy scholarly texts." It reconstructs what a citation should look like from memory, which is why it can hand you an author, a title and a DOI with total composure and be wrong about all three.

Why it guesses instead of saying it does not know

That still leaves the composure to explain. Researchers at OpenAI and Georgia Tech answered it in September 2025 with an argument about incentives rather than architecture. Models "hallucinate because the training and evaluation procedures reward guessing over acknowledging uncertainty", and their analogy is a student in an exam hall. "Like students facing hard exam questions, large language models sometimes guess when uncertain, producing plausible yet incorrect statements instead of admitting uncertainty."

The scoring is what does it. On a benchmark marked right or wrong, a blank scores zero and a guess scores zero unless it happens to land, so guessing is never worse and is sometimes better. A model trained under that rule learns to produce an answer. "I am not sure which paper says that" is the losing move on almost every leaderboard, so it is the move the model learned not to make.

Two consequences follow, and they are the ones that matter to you. Confidence in the writing is a property of the writing, not a measure of truth. And the model cannot audit itself, because the thing that would need checking is the thing it just made up. Pasting a suspect reference back into the same chat and asking whether it is real is the one check that cannot work.

What a made-up citation actually is

A fabricated citation is a reference to a source that does not exist, or to one that exists and says nothing like what it was cited for. What makes it dangerous is that nothing about it looks wrong. The author name is plausible. The journal is real and probably one you have heard of. The formatting is cleaner than your own. The DOI is the right length, carries a registrant prefix that belongs to the right publisher, and sits in the right journal's range of numbers.

Which way it is wrong decides what you do about it, and each kind has a page of its own here. A DOI that will not resolve at all, a DOI that opens a different paper, a real paper cited for a claim it does not make, a quotation the source does not contain, and a venue that is not real scholarship are five different problems with five different repairs.

218 DOIs we resolved ourselves

Every published measurement we could find tests models that are one to three years old. So in August 2026 we ran the experiment against current ones and published the raw dataset so anyone can check the arithmetic or disagree with the method.

The setup is the situation a student is actually in. One plain prompt, no tools, no web access, no retrieval. "List 10 peer-reviewed academic references on the topic of X, with authors, year, title, journal, volume, pages and DOI." Ten topics, chosen before we saw any result and drawn from ordinary undergraduate subjects, from urban heat islands to congestion charging. Replies were parsed by deterministic bibliography parsers, never by another language model, and every reference carrying a DOI, 218 of them, was resolved against CrossRef. Collected 11 August 2026.

The frontier model beats the small one by a wide margin, which is the shape every earlier study found too. One reference in five from the best of them was still unusable. For the rates other researchers have measured, across models and across the published literature, see how common fake AI citations are.

What Claude Haiku did instead, and why its number is not comparable

Claude Haiku 4.5 mostly refused. On most of the ten topics it declined to produce a reference list at all, said plainly that it could not give verified citations from memory, and named databases to search instead. That is the correct behaviour and worth crediting. It is also why its sample is 18 references rather than 100, and why you should not read its 33% next to the other two as though the three measured the same thing.

It cost us a day of work, too, because our first pass scored that refusal as fabrication. The parser turned "I cannot provide a verified list" and a list of database names into 41 unresolvable pseudo-references, and the model came out at 81% fabricated. That number was wrong, we caught it in a hand check before publishing, and it is a good illustration of why a measurement that agrees with your priors deserves the most scrutiny.

What this does and does not show

It is one run on one date. Rates move between runs, and we watched them move while collecting, so treat these as an order of magnitude and not a specification. Ten topics is enough to see a pattern and not enough to rank models. Only references that arrived with a DOI were scored, because the index that resolves title-only entries was over its daily budget that day, and counting an unresolvable title as fabricated would have inflated every figure here. A wrong-paper verdict is a title comparison against the CrossRef record, which we checked by hand on a sample rather than trusting the code.

A DOI that resolves is not a DOI that is right

Here is the part we did not expect, and it changed advice we had given on this site.

Everyone tells you to check a DOI by pasting it into doi.org and seeing whether it resolves. We told you that too. On this data that check misses most of the problem. For GPT-5.6, 7% of DOIs were dead and 12% resolved perfectly to a completely different paper. A bad DOI from that model was almost twice as likely to work as to fail.

One example, representative rather than cherry-picked. "Battery energy storage technology for power systems, an overview" is a real article in Electric Power Systems Research, and its DOI ends .017. The model gave it as 10.1016/j.epsr.2008.09.011. That string is registered, in the right journal, in the right issue, to a paper about multi-objective generation expansion planning. Off by six in the final digit. It resolves. It opens a page. The page is about something else.

The same pattern ran across topics. An urban heat island paper whose DOI opens a study on open windows and thermal comfort, in the correct journal. A remote work paper whose DOI opens "Pay Transparency and the Gender Gap", in the correct journal. The model is not inventing identifiers at random. It is producing an identifier of the right shape for the right venue and being wrong about which article sits there, which is the failure a wrong DOI takes even when no AI is involved.

So the check has to change, and it is barely more work. Open the DOI, read the title and authors on the page it lands on, and confirm they are the ones you cited. Not "does it resolve." Does it resolve to the paper you meant. Our citation checker makes exactly that comparison, and the full manual routine is how to check if a citation is real.

Why web search narrows it and does not close it

When a chatbot actually searches and cites pages it retrieved, the references are far more likely to be real, because they point at something it just read. That is a genuine improvement and it is the first thing to turn on. It is not a cure, and the size of the remaining gap has now been measured.

Delip Rao, Eric Wong and Chris Callison-Burch tested ten commercial models and deep research agents on 53,090 citation URLs, plus three models on a second set of 168,021, and reported in April 2026 that 3% to 13% of citation URLs were hallucinated and 5% to 18% did not resolve at all. The paper is a preprint and has not been peer reviewed. Two findings in it are worth carrying away. Deep research agents, the mode marketed as the careful one, produce more citations per query than search-augmented chat and hallucinate URLs at a higher rate. And the failure rate moves with the field, from 5.4% in business to 11.4% in theology, for the same reason fabrication moves with topic familiarity.

The same study points at what does work. Given an open-source tool that resolves a URL and reports back, the models cut non-resolving citations "by 6 to 79 times, to under 1%", with the size of the gain depending on how competently each one used the tool. That is the shape of the whole problem in one result. The model cannot check a reference by thinking harder about it. It can check a reference by looking it up, and so can you.

Which is also why prompt wording is the weakest lever available. Telling a model to use only real, verifiable sources gives it no new ability to verify anything, and we know of no study showing the instruction alone closes the gap. It changes the tone more than the truth. The settings and prompts that do move the number, and the point at which they stop helping, are in how to get ChatGPT to cite real sources.

Why your own rate runs above the average

Fabrication is not spread evenly across topics, and the pattern is the one the mechanism predicts. Linardon and colleagues generated six literature reviews with GPT-4o and verified all 176 citations in JMIR Mental Health. Across the three topics they covered, the most heavily researched returned fabricated citations 6% of the time. The other two returned 28% and 29%.

The model invents most where there was least real text for it to have learned from, which is exactly where a thesis question sits. A well-worn undergraduate essay topic is dense with training data and the citations come back mostly real. A narrow question at the edge of a subfield, the kind you were told to find because nobody has written it up yet, is the condition under which the model has the least to recall and the most to construct. The narrower your question, the further your personal rate runs above any headline average.

Why a student is the most exposed

This looks like a problem for published authors and large labs. The opposite is true, for three reasons.

You are leaning on general-purpose AI more heavily than a senior researcher with a decade of read papers in their head. The references you cannot personally vouch for are precisely the ones a model is most likely to have invented, because a fabrication thrives where you have the least ability to check it.

The people between you and a mistake are stretched. Fabricated references now survive peer review and reach the published literature in measurable numbers, so a marker working through forty essays in a weekend will miss them too, right up until the one time they decide to check a reference that looked too convenient. The fake does not have to be caught every time to ruin a submission. Once is enough.

And the consequence lands on you alone. A failed assignment, a delayed thesis, an academic-integrity meeting or a note in your record does not care that the tool sounded certain. "The AI gave it to me" is not a defence, because the bibliography is in your name. If that conversation has already started, my professor says my sources don't exist is the hour-by-hour version.

The habit that makes the question go away

Treat a chatbot as a fast, fluent, slightly unreliable research assistant, never as a librarian. Use it to draft, to rephrase, to explain a concept, to suggest where to look. Do not use it as the final word on what is true or what exists. Every factual claim you keep should trace back to a source you have opened, and every citation in your bibliography should be one you could find again on your own.

Two things make that cheap rather than tedious. Check a reference the moment it enters the draft, when it costs seconds, instead of checking forty at midnight. And prefer a tool that finds the paper before it writes the sentence, because when the source comes first there is nothing left to fabricate. That is the difference we set out in the best AI tools for academic writing, and it is the same line that decides whether using AI to write essays counts as cheating. What you want at the end is not a draft that survives a detector. It is a draft whose citations resolve and whose edits you read before you kept them.