Evaluating and Comparing Models
Time: about 45 minutes
Setup: on your own, or in a group
Requirements: the NER pipeline notebook, run through Section 8, and ideally your own annotations from the annotation activity
This is where the ideas from Unit 1 become a concrete calculation. You’ll compare the model’s predictions with human annotations, work out precision, recall, and F1 (Tjong Kim Sang and De Meulder 2003; Nadeau and Sekine 2007; Manning et al. 2008, chap. 8; Jurafsky and Martin 2026, chap. 18), look at what kinds of errors the model makes, and then compare it with a second model, and with Gemini Notebook.
If you need a refresher on the metrics, see the example, and especially From Marbles to Letters, which explains true positives, false positives, and false negatives with these letters.
The repository includes two independent sets of annotations, by two people (person_1 and person_2), of the same eight letters. By default, the model is evaluated against person_1. If you’ve done the annotation activity, the notebook loads yours too, as my.
Part 1: Evaluating the Model (Notebook Section 9)
Run the cells in Section 9. They will:
- Load every set of annotations in
data/annotations/, and show which letters each annotator covered - Choose one annotator as the reference (
REFERENCE,person_1by default) - Compare the model’s predictions with the reference annotations, letter by letter, and calculate precision, recall, and F1 for each entity type
- Repeat the evaluation against every annotator, so you can see how much the score depends on whose annotations count as the truth (Plank 2022)
Before moving on, say what they mean in plain language.
- If precision for people is 0.57: of every hundred names the model tagged as people, 57 really were people, and 43 weren’t.
- If recall is 0.71: of every hundred people the annotator listed, the model found 71, and missed 29.
Error Analysis
The next cell lists every disagreement between the model and the annotations.
- False positives: are they really wrong, or did the annotator miss them? Ground truth can be wrong too.
- False negatives: what do the missed entities have in common? Salutations? Initials? Names that are also ordinary words?
- Are the errors random or systematic? Systematic errors tell you about the mismatch between the model’s training data and your sources (Ehrmann et al. 2023).
In the letter to Colvin (vol1_ch3_letter14), for instance, the model tags Clever (from not so CLEVER perhaps) as a person, and misses COLVIN in the salutation MY DEAR COLVIN.
Inter-Annotator Agreement
The agreement cell calculates Cohen’s kappa for every pair of annotators on the letters they share (person_1 and person_2, and yours against both, if you’ve annotated), and treats the model as one more annotator. Read the explanation in the notebook: raw agreement always looks high, because most words aren’t entities, and kappa corrects for that. The next cell lists where person_1 and person_2 disagree, letter by letter.
- How does the two people’s agreement compare with the model’s agreement with either of them?
- Look at the disagreements between the two people. Which are boundary, type, or coverage disagreements? Which would you resolve, and how?
- How did the model’s F1 change when you switched the reference annotator? What does that tell you about reporting a single score?
Kappa isn’t the only way to measure agreement, many NER projects compare annotators with F1 instead, treating one as the reference (Hripcsak and Rothschild 2005). For more, Artstein and Poesio (2008) is the standard introduction to agreement measures in language research.
Confidence Thresholds
Each prediction has a confidence score. The threshold cell keeps only predictions above a series of cut-offs and recalculates the metrics, so you can see the tradeoff from Unit 1 on real data. Against person_1, keeping every prediction gives a precision of 0.62 and a recall of 0.73. Keeping only predictions with a confidence of 0.99 or more raises precision to 0.72, and drops recall to 0.55: fewer false alarms, but many more misses.
- Which matters more for your research goal, precision or recall? If you’re building an index of everyone mentioned in the letters and will check it by hand, recall probably matters more. If you’re publishing counts, both do.
- What threshold would you choose, and why?
- Some false positives have very high confidence: the model was 98% confident that
Cleveris a person. What does that tell you about using confidence to filter out errors (Guo et al. 2017; Desai and Durrett 2020)?
Part 2: Comparison with Other Approaches (Notebook Section 10)
Choosing a model is a methodological decision, with consequences for your results. Section 10 compares the model with two alternatives.
A Second NER Model
The first part of Section 10 runs a second model, dslim/bert-base-NER (English only, trained on news articles), on the annotated letters, and evaluates it the same way.
Before running it, make a prediction: a model trained specifically on English, or one trained on several languages? Which will do better on these letters, and why? Read both model cards for clues.
Then run the cells, and fill in a comparison like this one, using the notebook’s output and the error-analysis cell (you can adapt it to list the second model’s errors):
Davlan/bert-base-multilingual-cased-ner-hrl |
dslim/bert-base-NER |
|
|---|---|---|
| Precision / recall / F1, people | ||
| Precision / recall / F1, places | ||
| Precision / recall / F1, organizations | ||
| Notable false positives | ||
| Notable misses |
- Where do the two models agree? Where do they disagree?
- Was your prediction right? More specialized doesn’t automatically mean better for your task. Only evaluating on your own data tells you.
- The second model also predicts a
MISCcategory (Tjong Kim Sang and De Meulder 2003), which the evaluation ignores. What might you be missing by fitting its output to your annotation scheme?
Gemini Notebook
The second part of Section 10 evaluates the two data files that Gemini Notebook produced in our test of the Unit 2 activity, from the same prompt, on the same 20 letters, against the same annotations.
Before running it, make another prediction: will a general-purpose LLM do better or worse than the NER models?
Then run the comparison, and the cell after it, which checks whether each approach gives the same output twice.
Gemini Notebook did considerably better than both NER models on people and organizations, with F1 scores of about 0.75 and 0.72, against 0.63 and 0.48 for the main NER model. On places, the main NER model caught up (0.82, against about 0.80). Most of the difference is in recall: it found names that the NER model missed, such as COLVIN in the salutation, initials like Q., and names that are also ordinary words, like Low and Strong.
But the NER model, run twice on the same letter, gives identical output. Gemini Notebook’s two runs shared only 288 of their 327 and 325 rows, and some mentions changed type between runs. Anything built on its rows (counts, a network, a map) would change too. And you can’t see the code that produced them, or which model version did.
- If you had to choose one approach for a research project, which would it be, and why? What would you need to report about it?
- Two runs of Gemini Notebook scored almost the same, but produced different data. Which matters more for your research: the score, or the data?
- Could you get some of both? Advantages of Programmatic Approaches describes some approaches in between.
Options for Improving Results
If a model isn’t good enough for your purpose, you have several options, roughly in order of effort:
- Adjust the confidence threshold to favour precision or recall.
- Preprocess the text better (as
normalize_capsdoes for ALL-CAPS text). - Post-process the predictions: filter known false positives (
God), and add rules for patterns the model misses. - Try a different model, perhaps one trained on historical text (Schweter et al. 2022) or on your language, or an LLM called from code.
- Fine-tune a model on annotated examples from your own sources (Next Steps).
Recording Your Method
The last cell of the notebook prints a record of the models and settings used, and the evaluation. If you used results like these in a publication, your methods section would need at least:
- the model name and its exact version (on Hugging Face, the commit of the model’s repository), a link to its model card (Mitchell et al. 2019), and the version of the library that ran it
- the preprocessing and settings (here, capitalization normalization, the aggregation strategy, and how long texts were split)
- the evaluation data: how many letters, annotated by whom, following what guidelines, and with what agreement
- the results: precision, recall, and F1 by entity type, and against which annotator
- known systematic errors, and what you did about them
Documenting Your Work, in Unit 2, has a methods paragraph for this case study, built from exactly these figures.
Benchmark scores on a model card tell you how the model performed on its test data, typically modern news. Only an evaluation on your own material tells you what you actually need to know (Ehrmann et al. 2020).
Key Takeaways
- Precision, recall, and F1 on your own annotated material tell you what a model’s published benchmark scores can’t.
- Look at the errors, not just the scores. Systematic errors reveal the mismatch between the model’s training data and your sources.
- A score depends on whose annotations count as the truth, and agreement between people sets a limit on what a model’s score means.
- Confidence thresholds trade precision for recall, and high confidence doesn’t mean correct. Choose a threshold based on your research goal.
- The most accurate approach isn’t automatically the right one: on these letters, Gemini Notebook scored highest, but its results couldn’t be reproduced. Choosing a model is a methodological decision: compare candidates on your own data, and record what you used.
If you want to read more, the following are a good place to start:
- Ehrmann et al. (2020): the report of a shared task in which teams built NER systems for historical newspapers in several languages, and compared them on the same annotated material. It shows how much scores drop on historical text, and why.
- Plank (2022): a short argument that disagreement between annotators is often information, not just noise, and what that means for evaluating models against them.