Assessing Reliability
Once an LLM gives you a polished, plausible answer, it becomes hard to know how sceptical to be. Total distrust makes the tool useless, uncritical trust makes it dangerous. You’ve just seen both sides in the Gemini Notebook activity: output that was impressive, but that couldn’t be checked or repeated. This page moves from a vague sense that LLMs are sometimes useful and sometimes risky to a more disciplined question: what exactly would justify trusting specific output, for a specific task?
The goal is neither to make you fearful of LLMs nor to reassure you that everything is fine, but rather to help you build a professional standard of judgement. You already do this with other sources and tools.
Failure Modes: Patterns, Not Accidents
We’ve covered several ways in which AI systems fail. In Teachable Machine, the model failed when inputs didn’t resemble its training data. In Hack Merlin, its safety measures gave way, because guardrails are patterns, not rules. In Gemini Notebook, a story title became a person, and the same request produced different data.
These are failure modes: characteristic ways in which a kind of system goes wrong. The key point is that LLMs fail in characteristic, non-random ways. Their failures follow from how they work (LLM Behaviour explained why), which means you can learn to anticipate them. The main classes of failure are:
- Factual errors: hallucination, outdated knowledge, overgeneralization.
- Reasoning errors: faulty logic, arithmetic mistakes, losing track in multi-step work.
- Instruction-following errors: ignored constraints, over- or under-compliance, and overconfidence.
- Sycophancy: agreeing with you, flattering you, or giving way when you push back (Sharma et al. 2024).
- Bias: stereotypes, and values that may not be yours.
- Interaction errors: inconsistency between runs, forgetting earlier context, sensitivity to phrasing, and manipulation.
- Quieter errors: omission, over-interpretation, false certainty, and muddled sourcing.
Failure Modes is a reference page describing each of these, with examples and suggestions to help prevent or mitigate them. Skim it now, and come back to it when you need to put a name to something that went wrong.
Two failure modes deserve a special mention, because they’re the easiest to miss. Hallucination gets the most attention, but many bad outputs aren’t invented from nothing: they leave out important evidence, present a weak inference as settled, or flatten real ambiguity. In humanities work, these quieter errors are especially dangerous, because they look like interpretation rather than obvious falsehood. A model may correctly identify a name in a letter, and then overstate the relationship, or read the tone too confidently from thin evidence. The second is sycophancy, which turns your own expectations into the model’s answer: ask a leading question, and you’ll often get the answer you were leading towards (Sharma et al. 2024).
Degrees of Trust
Trusting LLM output is not a yes-or-no decision. It is a judgement about whether a specific output is fit for a specific use. Can I trust ChatGPT? is too broad a question to be useful. A better one is: Can I trust this output enough for this purpose?
A draft list of seminar discussion questions needs a different standard than a factual claim in a published article. Brainstorming search terms, drafting an email, or generating alternative phrasings can be helpful even without perfect accuracy. Exact quotations, archival descriptions, dates, counts, citations, translations, and extracted datasets demand much stronger checking.
Thinking this way helps you avoid both hype and blanket rejection.
Fluency Without Evidence
The greatest strength of LLMs is also one of their greatest risks: they produce fluent, well-structured prose. Polished writing triggers our normal cues for expertise, which makes it easy to lower our guard. A highly specific answer may feel more credible simply because it includes dates, names, or causal explanations, but specificity can be fabricated or weakly grounded. (Remember the three detailed, confident, and entirely invented biographies of Sidney Colvin in Unit 1.)
Make a habit of separating style from support. Ask yourself: what in this answer could I actually verify? Which claims are direct, and which are interpretive? Is the model showing me evidence, or only giving me conclusions?
Grounding the model in your sources, and asking it to quote them (Prompting Strategies), helps a great deal, because it gives you something concrete to inspect. But a grounded answer can still select the wrong passage, mistake a salutation for a mention in the body of a letter, or turn a tentative statement into a firm conclusion. Grounding improves transparency, not certainty.
Verification in Proportion to the Stakes
How much you check should depend on what the output will be used for.
- Low stakes: brainstorming keywords, drafting an email, rephrasing notes, generating possible avenues of inquiry. These still require judgement, but exhaustive checking may not be necessary.
- Medium stakes: summaries of sources, comparisons of themes, tentative interpretive scaffolding. These need visible evidence and selective checking.
- High stakes: anything that will be quoted, cited, counted, published as fact, or reused as data. These need much stronger verification, and sometimes a different method entirely.
This lines up with what conversational tools are good at:
- Good fits: exploratory, synthetic, and drafting tasks, such as brainstorming search terms, drafting summaries, comparing themes across a few texts, rephrasing notes, or helping write code.
- Weaker fits: exact counts, exhaustive retrieval, reproducible labelling, and precise factual claims, without verification.
Here’s a simple rule to follow: if the output needs to be cited, counted, or reused systematically, ask what evidence supports it, and whether the process can be repeated. If it can’t, as with the Gemini Notebook exports, the output can still be a useful starting point, but it can’t be the finished data. Unit 3 shows the alternative.
Some uses are poor fits for unverified LLM output, even when the answer sounds plausible.
- Exact quotations and citations. Models may paraphrase, compress, or invent wording while sounding precise (Walters and Wilder 2023).
- Exhaustive counts or lists. The model may leave items out without signalling that it has.
- Fine-grained chronology. Sequences can be flattened or rearranged.
- Sensitive interpretation without reviewing the sources yourself.
- Structured data for later analysis. The output may look neat while hiding inconsistencies, and a second run may not match the first.
This doesn’t mean the model can’t help with these tasks. It means the model shouldn’t be the final authority.
Practical Verification Approaches
While a general cautionary attitude is important, verification requires a set of concrete practices to ensure that the output can be trusted for its intended use.
- Ask the model to quote the phrase supporting each claim.
- Open the source, and check whether the quotation actually supports the conclusion drawn from it.
- Restate the question more narrowly, and see whether the answer holds steady.
- Try the same task in a fresh session or another tool, and compare (Manakul et al. 2023).
- Ask the question in a neutral way, without signalling the answer you expect (sycophancy).
- If a claim matters, verify it outside the model entirely.
- For structured tasks, compare the model’s output against a small sample you’ve checked by hand.
- Mark uncertainty explicitly.
Verification is easiest when the task was well scoped in the first place. Don’t just ask if the answer sounds right, ask what you would need to inspect to know whether it is right.
You can use familiar disciplinary habits to interrogate LLM output:
- What is the source of this claim?
- Is it explicit in the text, or inferred?
- What is missing or flattened here?
- Could a reasonable reader interpret this differently?
You already have many of the right instincts: the same critical habits you use with archives and secondary literature apply here. Ask whether a claim is stated in the source or constructed by the model. Ask whether the answer compresses ambiguity into certainty. Ask what it leaves out: models are often strong at producing the central tendency, and weak at preserving edge cases, contradictions, or minority evidence (Peters and Chin-Yee 2025; Kandpal et al. 2023). And ask whether a different framing of the same source would lead to a different interpretation.
From Spot Checks to Systematic Evaluation
Everything above is about checking individual responses. For structured tasks with right and wrong answers, such as finding the names in a set of letters, there’s a more systematic version: people with the relevant expertise annotate a sample of documents, producing a set of correct answers (the ground truth), and the model’s output is compared with it, using the precision, recall, and F1 we covered in Unit 1. How often two annotators agree tells you how reliable the ground truth itself is (Artstein and Poesio 2008).
This is what separates research use from casual use: treating model output as a hypothesis to be evaluated, not an answer to be accepted. You’ll do exactly this in Unit 3: annotate some letters yourself, and measure how well a model (and Gemini Notebook) did on them. The exercise below is a first, reduced version.
Exercise
- Go through the table row by row. Is each name really in the letter, exactly as written? Is the type right?
- Go through the letter. Is any name missing from the table?
- For each problem you find, look up the kind of failure in Failure Modes. Is it a factual error, an instruction-following error (names not recorded as written), a quieter error (an inference presented as a fact), or something else?
- Then compare with the answer key below, made by two people who annotated this letter independently.
Both annotators marked these eight names (as written in the letter, with their type):
| Mention | Type |
|---|---|
| SIDNEY COLVIN | person |
| COLVIN | person |
| LOUIS STEVENSON | person |
| Stephen (twice: Leslie Stephen, editor of the Cornhill) | person |
| EDINBURGH | place |
| Penzance | place |
| CORNHILL | organization (the Cornhill Magazine) |
| TEMPLE BAR | organization (the magazine Temple Bar) |
They disagreed on one: one annotator also marked post-office (at the post-office a letter will find me) as an organization, the other didn’t consider it a named entity. Neither marked Villon or The Sire de Maletroit's Mousetrap, which are titles of Stevenson’s works, not people.
So even the answer key involves judgement calls, and two careful people don’t agree completely. What would you have decided about the post-office?
Key Takeaways
- LLMs fail in characteristic ways, learning the main failure modes lets you anticipate them.
- Many errors are quiet: omissions, over-interpretation, false certainty, and answers that tell you what you expected to hear.
- Trust should be calibrated to the task and its stakes: use freely for low-stakes work, verify strongly anything you’ll quote, count, publish, or reuse as data.
- Verification is concrete: quote, check the source, ask again, compare, and check a sample by hand.
- For structured tasks, systematic evaluation against ground truth is the research-grade version of checking.
If you want to read more, the following are a good place to start:
- Walters and Wilder (2023): a short study that asked a chat assistant for literature reviews, and checked every citation it gave. Many of them didn’t exist.
- OpenAI, Sycophancy in GPT-4o (2025): the company’s own account of an update that made its chat assistant flatter users and agree with them too readily, and why it happened.