Next Steps

You’ve now worked through a complete pipeline: from raw letters to structured data, visualizations, and an evaluation of how far to trust them. This page pulls together the main lessons of the unit, and points to where you can go from here.

Key Takeaways

Control, not capability. Gemini Notebook was more accurate on these letters than the NER model, at least on people and organizations. The pipeline’s advantage is that you can reproduce it, inspect it, name the model that ran it, and know that every letter was processed.
Different tools for different tasks. Conversational tools are good for exploration and interpretation, and programmatic pipelines are good for systematic extraction, structured output, and reproducibility. Many projects use both.
Models are not magic. They make mistakes, inherit the strengths and blind spots of their training data, and need to be evaluated on your own sources. A high benchmark score on news articles doesn’t guarantee good performance on nineteenth-century letters (Ehrmann et al. 2023).
Evaluation is how you find out. Treat model output as a hypothesis to be tested, not an answer to be accepted. Build your own ground truth, calculate precision, recall, and F1, and look at the errors, not just the scores.
Errors compound. Each step in a pipeline (extraction, resolution, geocoding) adds its own errors, and a polished chart or map can hide all of them.
Make your use visible. Record the model, its version, your settings, and your evaluation. Your code is already a record of what you did, and the prompts you used to write it are part of your method too.
Human judgement remains essential. The model found the entities. You decided how to resolve them, how to evaluate them, and what they mean.

Writing Up AI Methods

If you used a pipeline like this in published research, your methods section would need to describe it precisely enough for a reader to judge, and repeat, what you did. You already have the pieces:

Recording Your Method lists what the methods section needs: the model and its version, your preprocessing and settings, your evaluation, and its known errors.
Documenting Your Work has an example methods paragraph for this case study.
Acknowledging AI Use covers reporting any help you had from an LLM, for example in writing or adapting the code.

Try drafting the paragraph yourself, for the case study as you ran it: with your annotations, your choice of reference annotator, and your threshold. What would you have to go back and check?

Applying the Pipeline to Your Own Sources

The notebook processes any folder of plain-text (.txt) files. To try it on your own material:

Put your texts in a folder, one document per file.
Change letters_dir in Section 3 to point to it (and skip or adapt the metadata merge, which is specific to the Stevenson letters). Coding with LLMs explains how to get the path right.
Run Sections 2–4, and look critically at the results.
Annotate a sample of your own documents and evaluate, as in Evaluating and Comparing Models.

Things that will likely need to change:

Format: PDFs, scans, or spreadsheets need converting to text first. OCR errors will affect the model (Hamdi et al. 2023).
Language: choose a model trained on your language. The multilingual model in the case study covers ten languages, but many other models exist.
Entity types: if you need dates, works, ships, or occupations, look for a model with those labels, consider fine-tuning (below), or try an LLM called from code, evaluated in the same way.
Resolution: you’ll need your own alias dictionary, or a more sophisticated approach.

Further Directions

Hugging Face Model Hub: browse pre-trained models for NER, classification, summarization, translation, and more. Filter by task and language. Read the model card before you use one.
spaCy: a fast, widely used Python library for language processing. Good for production pipelines, rule-based matching (useful for custom entity patterns), and working with Hugging Face models.
Fine-tuning: if an off-the-shelf model isn’t good enough, you can continue training it on annotated examples from your own sources, so that it learns their particular names, spellings, and conventions. It’s transfer learning again, as in Teachable Machine, on a larger scale (Howard and Ruder 2018). It needs far more annotated examples than an evaluation does, and more computing power, but it can improve performance on your material considerably (Brandsen et al. 2020; Brandsen et al. 2021; Schweter et al. 2022). The Hugging Face documentation covers it step by step.
LLMs from code: calling an LLM through an API, with a pinned model version and a saved prompt, or running an open model on your own computer, combines some of an LLM’s flexibility with some of a pipeline’s control (The Best of Both Worlds). Evaluate it exactly as you evaluated the NER model.
The self-paced introduction to Python: if you want to become more comfortable reading and adapting code.
This repository: the notebook and data are yours to adapt, under the licence.

A Return to the Beginning

These materials started from one idea: AI models learn patterns from data, and everything they can and can’t do follows from that. In Unit 1, you saw it in a model that learned shape from a few dozen photos. In Unit 2, you saw it in a chat assistant that was fluent, often right, and impossible to pin down. Here, you saw it in a pipeline whose every step you could inspect, measure, and repeat. Whatever tools you use next, the same questions apply: what was this model trained on, how would I know if it’s wrong, and could someone else see what I did?

Going Further

If you want to read more, the following are a good place to start:

The Programming Historian: peer-reviewed, step-by-step lessons on digital methods for humanities researchers, from text analysis and NER to networks and maps. Many of the techniques in this unit have lessons there.
Ehrmann et al. (2023): a survey of named-entity recognition on historical documents, with the main challenges, resources, and approaches. The introduction and the sections on historical text are readable without the technical detail.

References

Brandsen, Alex, Suzan Verberne, Karsten Lambers, and Milco Wansleeben. 2021. “Can BERT Dig It? -- Named Entity Recognition for Information Retrieval in the Archaeology Domain.” arXiv:2106.07742.

Brandsen, Alex, Suzan Verberne, Milco Wansleeben, and Karsten Lambers. 2020. “Creating a Dataset for Named Entity Recognition in the Archaeology Domain.” In Proceedings of the Twelfth Language Resources and Evaluation Conference, edited by Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, et al. European Language Resources Association.

Ehrmann, Maud, Ahmed Hamdi, Elvys Linhares Pontes, Matteo Romanello, and Antoine Doucet. 2023. “Named Entity Recognition and Classification in Historical Documents: A Survey.” ACM Computing Surveys 56 (2): 27:1–47.

Hamdi, Ahmed, Elvys Linhares Pontes, Nicolas Sidere, Mickaël Coustaty, and Antoine Doucet. 2023. “In-Depth Analysis of the Impact of OCR Errors on Named Entity Recognition and Linking.” Natural Language Engineering 29 (2): 425–48.

Howard, Jeremy, and Sebastian Ruder. 2018. “Universal Language Model Fine-Tuning for Text Classification.” In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), edited by Iryna Gurevych and Yusuke Miyao. Association for Computational Linguistics.

Schweter, Stefan, Luisa März, Katharina Schmid, and Erion Çano. 2022. “hmBERT: Historical Multilingual Language Models for Named Entity Recognition.” arXiv:2205.15575.