AI does not need to deceive us to cause damage. It only needs to produce an answer that looks credible — and meet a human who decides not to verify it.
That distinction matters. A hallucination is an error: a model generates information that is unsupported, distorted, or entirely invented. But an error inside a chat window is still only an error. The real harm begins when someone copies it into a document, a decision, a medical note, a financial model, or a court filing and treats it as fact.
A legal brief with witnesses who never existed
In New Mexico, defense lawyer Stephen Aarons used ChatGPT while preparing an appeal in a murder case. According to the state’s highest court, the submitted brief included invented police testimony and witnesses who did not exist.
Aarons said he had supplied trial materials to the system and expected a reliable summary. The result sounded convincing enough to use. It was not verified against the record before it reached the court.
The New Mexico Supreme Court held him in contempt, imposed a $5,000 fine, and referred the matter for disciplinary investigation. The appeal was reassigned to a public defender.
Confidence is not evidence
AI systems are optimized to generate useful and coherent responses. Coherence can look like knowledge. Detail can look like research. A calm tone can look like certainty.
None of those qualities prove that the underlying claim is true.
This is why the most obvious hallucinations are often the least dangerous. A ridiculous answer triggers suspicion. A detailed answer with names, dates, quotations, and professional language can pass through a busy workflow unnoticed.
The risk grows with the consequence
Using AI to brainstorm a headline is not the same as using it to summarize evidence in a criminal appeal. The same model error can have radically different consequences depending on where the output is used.
- Check every factual claim against the original source.
- Open and read every cited case, article, regulation, or document.
- Compare summaries with the underlying record.
- Keep a human accountable for the final decision.
- Do not allow an AI-generated draft to become an official submission by default.
For consequential work, AI should accelerate analysis — not replace the chain of evidence. A faster draft is useful. A faster path to an unverified conclusion is not.
“The AI said so” is not a transfer of responsibility
Blaming the tool after an error does not solve the governance problem. Professionals remain responsible for what they sign, submit, approve, prescribe, publish, or send.
This principle extends far beyond law. An accountant must verify figures. A doctor must verify clinical information. An engineer must verify specifications. A manager must verify the basis of a decision. AI may assist each of them, but it does not inherit their duty of care.
The practical lesson
Organizations do not need to ban AI to reduce this risk. They need workflows that distinguish between low-stakes assistance and high-stakes reliance.
That means defining which outputs require source checks, which actions need approval, where provenance must be recorded, and who owns the final result. Verification should be a designed step — not something we hope a busy person remembers.
AI can be wrong with extraordinary confidence. Human responsibility begins where that confidence meets the real world.
