Background
The project, its historical setting, and prior research
How well do large language models know the women who shaped early Christianity?
This project evaluates six LLMs — Claude, ChatGPT, Gemini, Grok, DeepSeek, and GLM — on their knowledge of female figures from the first five centuries CE. We ask not only whether models produce factually accurate responses, but whether they reproduce the same androcentric silences and distortions that have long characterized the historical record itself.
The project has three aims:
- Establish a replicable evaluation methodology for assessing LLM outputs on specialized historical content — rubric, annotation protocol, and question bank — designed to transfer to other underdocumented historical populations.
- Identify and document bias patterns specific to early church history: hallucination in sparse historical domains, androcentric training data, conflation of canonical and apocryphal sources, and evasive responses to contested scholarship.
- Build a public benchmark comparing model performance across figures, rubric dimensions, and question difficulty, released openly so that it can be rerun as models evolve.
All materials — methodology, question bank, raw prompts, annotation guidelines, and results — are published here so that other researchers can replicate, extend, or critique the evaluation.
Why “Matristics”?
Matristics — coined by analogy with patristics, the study of the church fathers — names the study of the church mothers: the women whose leadership, patronage, scholarship, and witness shaped Christianity’s formative centuries. The name reflects the project’s central question: whether the digital memory embodied in language models preserves these women any better than the historical record did.
Historical Context: Women in Early Christianity
The first five centuries of Christian history produced a substantial number of women who led communities, developed theology, wrote texts, resolved ecclesiastical disputes, funded institutions, and exercised prophetic authority. Most of them are unknown to a general audience. Many are known only through sources written by men who had reasons — rhetorical, theological, institutional — to shape how they are remembered.
This section provides the historical background necessary for understanding both the figures under investigation and the interpretive challenges the project confronts.
The Documentary Problem
The sources for women’s lives in early Christianity survive unevenly and almost never in women’s own words: of the vast surviving corpus, only a handful of texts — Perpetua’s prison diary, Egeria’s travel journal — are plausibly female-authored. What survives was copied, edited, and canonized by male ecclesiastical institutions, so hagiographic conventions and theological agendas shape even sympathetic portraits such as Gregory of Nyssa’s Life of Macrina. Apocryphal literature, funerary inscriptions naming women as deacons and patrons, and other archaeological evidence provide an important corrective, but they are fragmentary and contested.
Women’s Roles in the First Three Centuries
In the earliest communities women appear across the full range of leadership roles: house-church leaders (Priscilla, Lydia, Nympha, Chloe), a deacon and patron (Phoebe), apostles and missionaries (Junia, Mary Magdalene, Thecla), and prophets (Philip’s daughters, Ammia of Philadelphia, the Montanist prophetesses Maximilla and Priscilla). Martyrdom accounts such as that of Perpetua and Felicitas preserve some of the most vivid — and most nearly first-person — evidence of women’s religious agency. The overall picture is one of genuine but contested authority, already being narrowed by the household codes of the later New Testament writings.
Women’s Roles in the Fourth and Fifth Centuries
With imperial patronage came institutionalization: the female diaconate was formalized in church orders such as the Didascalia and the Apostolic Constitutions — Olympias of Constantinople is its best-documented holder — while aristocratic women such as Macrina, Melania the Elder, and Paula founded monasteries, and intellectual patrons like Marcella turned Roman households into centers of scholarship. At the same time, conciliar legislation and episcopal writing progressively restricted the public roles women had exercised in earlier centuries. Whether this period represents a flourishing of women’s religious life (through asceticism) or a narrowing (through exclusion from office) remains a central debate in the scholarship.
Key Scholarly Literature
The following works are foundational for contextualizing the figures under study.
| Work | Focus |
|---|---|
| Cohick & Hughes 2017 | The most comprehensive recent survey of female figures from the second through fifth centuries. |
| Madigan & Osiek 2005 | Primary sources in translation with critical commentary on ordained women. |
| Kraemer & D’Angelo 1999 | Foundational feminist historiography of the period. |
| Kraemer 1992 | Women’s religious lives among pagans, Jews, and Christians across the Greco-Roman world. |
| Cooper 2013 | Accessible synthesis recovering the world of ordinary early Christian women. |
| Elm 1994 | Scholarly analysis of female asceticism in late antiquity. |
| Clark 1986 | Essays on women, asceticism, and piety in late ancient Christianity. |
| Torjesen 1993 | Argument for women’s formal priestly leadership in early Christianity. |
| Clark 1983 | Patristic primary sources on women in translation, with commentary. |
| Clark 1979 | Essays and translations on the women in Jerome’s and Chrysostom’s circles, including the Life of Olympias. |
| Hartney 2004 | Chrysostom’s preaching and its urban context — the Constantinople of Olympias. |
| Swan 2001 | Sayings, lives, and stories of early Christian women in the desert tradition. |
| Oden 1994 | Anthology of women’s writings across the history of Christian thought. |
| Paintner 2012 | Annotated introduction to the sayings of the desert fathers and mothers. |
LLM Evaluation Context
Large language models are increasingly used as educational tools, research aids, and reference systems. Understanding the limits of their historical knowledge — and the systematic biases in how that knowledge is distributed — is a research problem with practical consequences.
Why Early Church History?
Early church history is a particularly revealing test case for several reasons that are developed fully in the Known LLM Biases in Early Church History section:
- The domain has sparse and contested primary sources, creating high hallucination risk.
- The textual tradition is deeply androcentric: the surviving record was overwhelmingly produced by men.
- The scholarship on women in this period has undergone significant revision over the past four decades, creating a gap between older popular accounts and current specialist knowledge.
- Several key questions (women’s ordained ministry, apostolic authority) are theologically contested in contemporary Christianity, creating pressure on LLMs to avoid or distort historically answerable questions.
Prior Work on LLM Evaluation
The foundational methodology for this project draws on several key papers. Zheng et al. (2023) introduced the LLM-as-judge paradigm and the MT-Bench benchmark, establishing pairwise comparison as a scalable evaluation approach. Liu et al. (2023) proposed G-Eval, a rubric-based scoring framework using chain-of-thought reasoning that informs our own dimension rubric. Wang et al. (2024) and Chen et al. (2024) provide comprehensive surveys of the field as of 2024, covering the strengths and failure modes of automated evaluation.
Prior Work on LLM Bias
On the bias side, UNESCO (2023) documents alarming evidence of regressive gender stereotypes in major generative AI systems. Kotek, Dockum, and Sun (2023) demonstrates that LLMs are 3–6 times more likely to assign occupations stereotypically aligned with a person’s gender. Navigli, Conia, and Ross (2023) provides a systematic taxonomy of bias origins in large language models, from training data selection through reinforcement learning with human feedback.
What This Project Adds
Existing evaluation frameworks measure general capability, and existing bias studies measure stereotyping in contemporary contexts. Neither addresses how LLMs represent historical women in a domain where the training data itself inherits two millennia of androcentric selection — where the silences in the model reproduce silences in the archive. The three aims above are pursued at that intersection, and the first results are reported in the Benchmark.