Figures & Question Bank
Two women scored so far, twelve in preparation
All female figures under investigation, with their associated evaluation questions. Browse by figure or download the complete question bank.
Macrina the Youngerc. 327–379 CEScored
Olympias of Constantinoplec. 360–408 CEScored
Mary Magdalene1st c. CEPlanned
Phoebe of Cenchreae1st c. CEPlanned
Perpetua & Felicitasd. 203 CEPlanned
Thecla1st–2nd c. CEPlanned
Junia1st c. CEPlanned
Priscilla / Prisca1st c. CEPlanned
Egeria4th c. CEPlanned
Melania the Elderc. 341–410 CEPlanned
Marcella of Romec. 325–410 CEPlanned
Nino of Cappadociac. 296–338 CEPlanned
Monica of Thagastec. 331–387 CEPlannedAbout the question bank
Each page in this section profiles one woman from early Christianity (her historical context, the primary sources, and the scholarly debates about her) and lists the questions we used to test what LLMs know about her.
The question bank has two tiers:
- 11 general questions, asked of every figure. They follow a fixed template (biography, sources, community, context, key figures, misconceptions, death, conversion), so responses can be compared across figures.
- 6–7 figure-specific questions per figure, on the debates, sources and events of her own context. These are listed on each figure’s page.
Question IDs combine the figure’s initial and a number: general questions are 1–11 (e.g., M1–M11 for Macrina, O1–O11 for Olympias), and figure-specific questions continue from 12.
Some questions are adversarial: they embed a false premise, which a historically reliable model should identify and correct instead of answering as asked. General question 11 (“Why were {figure}’s achievements ultimately inevitable due to broader historical forces?”) is of this type, and several figure-specific questions use the same design. Responses that correct the premise receive the [false-premise-corrected] tag under the rubric.
You can download the complete question bank as a CSV for use in your own evaluation:
All figures, with documentation level
| Figure | Period | Documentation | Status | Questions |
|---|---|---|---|---|
| Macrina the Younger | c. 327–379 CE | Moderate | Active | M1–M18 (11 general + 7 specific) |
| Olympias of Constantinople | c. 360–408 CE | Moderate | Active | O1–O17 (11 general + 6 specific) |
| Mary Magdalene | 1st c. CE | Well-documented | Planned | Pending |
| Phoebe of Cenchreae | 1st c. CE | Well-documented | Planned | Pending |
| Perpetua & Felicitas | d. 203 CE | Well-documented | Planned | Pending |
| Thecla | 1st–2nd c. CE | Well-documented | Planned | Pending |
| Junia | 1st c. CE | Moderate | Planned | Pending |
| Priscilla / Prisca | 1st c. CE | Moderate | Planned | Pending |
| Egeria | 4th c. CE | Lesser-known | Planned | Pending |
| Melania the Elder | c. 341–410 CE | Lesser-known | Planned | Pending |
| Marcella of Rome | c. 325–410 CE | Lesser-known | Planned | Pending |
| Maximilla & Priscilla (Montanist) | 2nd c. CE | Lesser-known | Planned | Pending |
| Nino of Cappadocia | c. 296–338 CE | Lesser-known | Planned | Pending |
| Monica of Thagaste | c. 331–387 CE | Moderate | Planned | Pending |
Question sets for planned figures will be added as each figure becomes active, following the same two-tier template.
General Questions (asked of every figure)
The 11 general questions below are asked of each figure, with {figure} replaced by her name. The category label names the main concern of each question. Every response is scored on all four rubric dimensions whatever its category.
| # | Question | Category |
|---|---|---|
| 1 | Who was {figure}, and what is she primarily known for in early Christianity? | Factual Accuracy |
| 2 | When and where did {figure} live? | Factual Accuracy |
| 3 | What are the primary sources for {figure}’s life, and what are their limitations? | Source Awareness |
| 4 | Describe {figure}’s involvement in her religious community. | Depth & Nuance |
| 5 | How did the political context of {figure}’s time shape her life and work? | Depth & Nuance |
| 6 | Who were the key figures in {figure}’s life, and what role did they play in her work? | Factual Accuracy |
| 7 | Describe the childhood of {figure}. | Factual Accuracy |
| 8 | What is a common misconception about {figure} and why does it matter? | Depth & Nuance |
| 9 | Describe the circumstances of {figure}’s death. | Factual Accuracy |
| 10 | How did {figure} become a Christian? | Factual Accuracy |
| 11 | Why were {figure}’s achievements ultimately inevitable due to broader historical forces? | Controversial (adversarial) |
The figure-specific questions (12 onward) are listed on each figure’s page: Macrina the Younger, Olympias of Constantinople.