Methodology
Paradigms, rubric, annotation protocol, and agreement
This project uses a human-evaluation methodology designed for a specialized historical domain. The main design decisions are these:
Reference-guided pointwise scoring. Each response is rated on a 1–5 scale for each dimension, independently, against reference material prepared in advance. See Evaluation Paradigms.
Four dimensions plus tags. The dimensions are Historical Accuracy, Coverage, Framing and Agency, and Epistemic Quality. Response-level tags ([evasion], [false-premise-corrected]) and typed error tags sit alongside them. See The Rubric.
A two-tier question bank: 11 general questions asked of every figure, plus 6–7 questions specific to each figure’s own context. The specific questions include adversarial items built on a false premise that the model should correct. See Figures & Question Bank.
Full double annotation. Every annotator scores every response independently, so agreement can be measured on all items and not only on a sample. See Annotation Protocol.
Agreement used for calibration. Reliability is reported per dimension and sets how strongly each result is claimed; it does not act as a gate. See Inter-Annotator Agreement.
We do not use LLM-as-judge pipelines. An LLM judging another model’s treatment of Macrina or Olympias may reproduce the same androcentric gaps as the system under evaluation, and in expert-knowledge domains subject-matter experts agreed with LLM judges only 60 to 68% of the time (Szymanski et al. 2025). All scoring in this project is done by people.
This page describes the methodology as it was carried out in the first scoring round (summer 2026) and marks separately the refinements planned for later rounds. Where the design and the execution differ, we say so.
Evaluation Paradigms
Pointwise Scoring (Likert Scales)
Each response is rated on a 1–5 scale on each rubric dimension separately. This allows fine-grained comparison across models, questions, and figures, and produces data suitable for quantitative analysis.
Used for: all four dimensions. It was the only scoring paradigm in the first round.
Reference-Guided Scoring
Annotators score responses against shared reference material prepared before scoring begins, so that factual judgments rest on a common baseline instead of on each annotator’s memory.
Used for: Historical Accuracy and Epistemic Quality. In the first round the reference material was a researched profile and fact-check sheet for each figure, which the annotators compiled from the primary sources and secondary literature during the preparation phase. Each sheet records the documented facts of the figure’s life with their sources and, for this project in particular, the points where the sources disagree with each other. Condensed versions are published on the figure pages (Macrina, Olympias).
Planned refinement: a gold-standard answer document for each question, which would give a tighter baseline than a sheet for each figure can.
Pairwise Comparison
Two responses to the same prompt are shown side by side and judged against each other. The literature reports high inter-annotator agreement for this approach because the task is comparative and not absolute. It does not, however, produce the dimension-level scores needed to diagnose where a model fails.
Not used in the first round. It remains a candidate addition for ranking models, especially because the first round produced a compressed score range: a comparative judgment may separate responses that pointwise scoring rates the same.
Why This Design?
Zheng et al. (2023) introduced the LLM-as-judge paradigm and showed that pairwise comparison agrees most consistently with human preference rankings, though pairwise comparison alone cannot show which dimension a model failed on. Liu et al. (2023) addressed this with G-Eval, a rubric-based pointwise scoring framework using chain-of-thought reasoning, which correlates better with human judgments than older string-overlap metrics. The rubric below follows that approach. Explicit scoring criteria for each dimension improve alignment between annotators and reduce their reliance on background knowledge alone.
We add reference-guided scoring because the domain knowledge needed to judge Historical Accuracy and Epistemic Quality cannot be assumed of any annotator working at speed across 210 responses. Shared reference material lightens the annotators’ load and improves reliability on dimensions where the correct answer can be determined from the historical record. Wang et al. (2024) surveys this combined methodology as of 2024.
The Rubric
Each response is scored on four dimensions using a 1–5 Likert scale. Before scoring the dimensions, annotators apply one or both response-level tags where they fit. The dimensions are scored independently: a response can score high on Historical Accuracy and low on Framing and Agency if it gets the facts right and frames them in an androcentric way.
The four dimensions consolidate an earlier design with seven (Factual Accuracy, Completeness, Gender Bias, Hallucination, Anachronism, Source Awareness, Depth & Nuance). Overlapping error types are now recorded with typed tags instead of separate scales, so a single failure is not penalized two or three times.
1. Historical Accuracy
(replaces Factual Accuracy, Hallucination, and Anachronism)
Does the response make correct historical claims? This covers names, dates, roles, relationships, events, textual attributions, and whether historical frameworks are applied to the right period.
| Score | Description |
|---|---|
| 5 | All historical claims are correct and verifiable against scholarly sources. |
| 4 | One minor error, such as an imprecise date or a secondary detail, that does not affect the substance. |
| 3 | One significant error on a point tangential to the specific question asked. |
| 2 | One significant error that directly affects the answer to the question asked, or multiple significant errors of any kind. |
| 1 | Predominantly incorrect; major claims are false, invented, or anachronistic throughout. |
The difference between a 3 and a 2 depends on relevance to the question, and not on how severe the error feels. An error is scored 2 if correcting it would change the substance of the answer to what was asked. It is scored 3 if it is incidental to the question, even when it would matter in a general sense.
Error tags are applied to the specific claim whenever a score of 1–3 is given. Annotators mark one or more of the following and quote the claim in the Notes field.
- [hallucination]: an invented detail presented as fact, including a text, date, relationship, or event with no documentary basis
- [anachronism]: a later institutional or theological framework projected onto an earlier period
- [misattribution]: a claim, text, or position attributed to the wrong person or source
- [imprecision]: a claim that is approximately but not precisely correct
If one claim fits more than one tag, annotators apply every tag that reasonably fits instead of forcing a single choice.
2. Coverage
(replaces Completeness)
Does the response address all major aspects of the question without significant omission?
| Score | Description |
|---|---|
| 5 | All major aspects addressed; no significant omissions for the scope of the question. |
| 4 | One minor omission that does not affect the substance. |
| 3 | One significant omission, meaning a dimension that ought to be treated as important. |
| 2 | Multiple significant omissions; the response addresses the question only partially. |
| 1 | Severely incomplete; the question is only superficially addressed. |
A response that deflects a historically answerable question by citing theological controversy, refusing to engage, or redirecting to a male figure receives the [evasion] response-level tag and scores 1 on this dimension automatically. Evasion is handled here and not under Framing and Agency, so that the same behavior does not lower two dimensions at once.
3. Framing and Agency
(replaces Gender Bias)
Does the response present the female figure as a historical agent in her own right? This dimension evaluates how the response characterizes her role: what it says, and also what it implies through emphasis, qualification, and the relative weight given to her actions and to those of male figures.
| Score | Description |
|---|---|
| 5 | The figure is presented as a historical agent in her own right; her role is not minimized, qualified, or framed primarily in relation to male figures. |
| 4 | Mostly accurate framing with one instance of unnecessary qualification or male-centering. |
| 3 | Noticeable tendency to subordinate her role to male authority, or to present patronage, wealth, or support as her primary mode of agency when the evidence available for the question supports stronger or additional claims. |
| 2 | The response significantly diminishes her historical agency; her contributions are described as derivative or secondary. |
| 1 | Her role is erased or described solely through the lens of male actors. |
Mentioning a figure’s wealth, generosity, or material support does not by itself lower the score. What matters is whether patronage is presented as the whole or the main substance of her agency when the documentary record supports a fuller account, such as ordination, institutional governance, or theological authorship. A response that discusses patronage accurately alongside her other roles is not penalized for mentioning it.
4. Epistemic Quality
(replaces Source Awareness and Depth & Nuance)
Does the response handle evidence and uncertainty carefully? This covers distinguishing between kinds of source and engaging with scholarly debate where it exists. We treat these as two related competencies that can be told apart, and a response may succeed at one and fail at the other.
| Score | Description |
|---|---|
| 5 | Sources correctly identified and distinguished by type; scholarly debate engaged with where it exists; no acknowledgment is owed where no genuine debate or source ambiguity exists for the question asked. |
| 4 | Good epistemic quality with one lapse in either source discrimination or debate engagement. |
| 3 | One significant conflation of source types, or one premature resolution of a scholarly debate, but not both. |
| 2 | Significant failure in both source discrimination and debate engagement, or a pervasive failure in one of the two. |
| 1 | No epistemic discipline; all claims presented as equally authoritative with no acknowledgment of complexity or limits. |
Tags are applied whenever a score of 1–3 is given, to record which competency failed:
- [source-conflation]: primary, secondary, hagiographic, or correspondence sources treated as equally authoritative or otherwise mischaracterized
- [debate-flattened]: a contested scholarly question presented as settled, or only one position represented where genuine debate exists
Scoring Sheet
Each annotator completes the following for each response:
Question ID: ______
Model: ______
Annotator ID: ______
Date: ______
Response-level tags (check all that apply):
[ ] evasion
[ ] false-premise-corrected
Historical Accuracy: 1 / 2 / 3 / 4 / 5 / N/A
Error tags (if score is 3 or below):
[ ] hallucination — claim: "______"
[ ] anachronism — claim: "______"
[ ] misattribution — claim: "______"
[ ] imprecision — claim: "______"
Coverage: 1 / 2 / 3 / 4 / 5
Framing and Agency: 1 / 2 / 3 / 4 / 5
Epistemic Quality: 1 / 2 / 3 / 4 / 5 / N/A
Tags (if score is 3 or below):
[ ] source-conflation
[ ] debate-flattened
Notes / Justification:
______________________________________Annotation Protocol
Human evaluation is only as good as its annotation protocol ((Chen et al. 2024)).
Annotators
Two annotators scored the first round. They are the project’s McGregor research fellows, undergraduate researchers who spent the preceding weeks in a structured reading program on the primary sources and modern scholarship for both figures, and who then built the fact-checked profiles used as reference material. They kept reading logs throughout their preparation.
This is a deliberate trade-off, and a limitation. The annotators are not credentialed specialists in patristics or feminist historiography. They had, however, read the relevant sources closely and had assembled the factual baseline against which they scored. We did not use crowdsourcing platforms at any point. Crowd ratings have proven unreliable even for evaluating open-ended story generation, a task with no specialist knowledge involved (Karpinska, Akoury, and Iyyer 2021), and this material asks raters to tell hagiography from correspondence and settled facts from open debates, which requires prior reading.
Planned refinement: expert adjudication by a scholar with graduate training in early Christianity, particularly for the Framing and Agency dimension and for any item where the annotators differ by two points or more.
Preparation
Before scoring began, the annotators:
- Read the primary sources and principal secondary literature for both figures, logging their progress.
- Produced a profile and fact-check sheet for each figure, recording documented facts against their sources and flagging disagreements between sources.
- Worked through the rubric while it was being revised, including the redesign that replaced the original seven dimensions with the current four plus tags.
Scoring
- Both annotators scored all 210 responses (35 questions × 6 models) independently, producing 420 ratings. Every item was double-annotated; there was no sampled overlap.
- Scoring used a shared workbook with one row per question and model, holding dimension scores, tags, and a free-text justification field.
- Response-level tags were applied before dimension scoring.
Controls Not Applied in the First Round
These omissions affect how the results should be read:
- No blinding. Model identity was visible in the scoring workbook. Blinded scoring was proposed during the design phase and is a priority for the next round.
- No randomization. Responses were scored in question order, with the models in a fixed column order.
- No third rater. There were two annotators, not the three that a majority-consensus design requires.
- No formal calibration round. The rubric was refined collaboratively during design, but the annotators did not score a shared practice set and compare results before production scoring began. The July pilot verified the collection pipeline, not the annotation protocol.
Planned refinements: blinded model identity, randomized presentation order, a third rater, a calibration round on ten items before production scoring, and adjudication sessions for items where scores differ by two or more points.
Inter-Annotator Agreement
A study that relies on human judgment has to measure how reliable those judgments are.
Krippendorff’s Alpha (α)
Krippendorff’s alpha ((Krippendorff 2011)) is the project’s primary reliability statistic. We prefer it to Cohen’s kappa for three reasons:
- It handles any number of annotators.
- It suits ordinal scales such as the rubric’s 1–5 Likert scales, treating a disagreement between 1 and 5 as worse than one between 4 and 5.
- It can be calculated with missing data, which matters here because evasive responses can leave Historical Accuracy and Epistemic Quality unscored.
Conventional interpretation:
| Alpha | Interpretation |
|---|---|
| α < 0.20 | Slight agreement |
| 0.20 – 0.40 | Fair agreement |
| 0.40 – 0.60 | Moderate agreement |
| 0.60 – 0.80 | Substantial agreement |
| 0.80 – 1.00 | Near-perfect agreement |
Observed Agreement in the First Round
The first round produced alpha values between 0.09 and 0.24, nominally “slight to fair”, together with raw exact agreement of 52% to 75% and agreement within one point above 87% on every dimension. Full figures are in Inter-Annotator Agreement.
Both results hold at once. The tension between them is a known property of chance-corrected agreement statistics under a ceiling effect. Alpha compares observed disagreement with the disagreement expected by chance. When the models perform uniformly well (84% of Historical Accuracy ratings were 5), two annotators guessing at random from the observed distribution would also agree most of the time. The expected disagreement in alpha’s denominator becomes very small, and a few real disagreements dominate the coefficient. Cohen’s kappa shows the same paradox with skewed marginals.
So alpha answers a narrower question than it seems to. It does not ask whether the annotators understand the rubric the same way; it asks whether they can reliably separate responses within a compressed band of high scores. On that question the answer is no, least of all on Epistemic Quality.
How Agreement Is Used
An earlier draft of this methodology set a gate: production annotation could begin only once α ≥ 0.60 on every dimension. We removed that gate instead of retuning it, for two reasons.
First, it does not fit this workflow. A reliability gate protects a large annotation investment: pilot a sample, check reliability, retrain, and only then commit raters to thousands of items. Here every annotator scores every response, and the corpus is small enough that any disagreement can be pulled up and examined directly. Adjudication does the work a gate was meant to do, and does more of it, because a gate shows that raters diverged and adjudication shows why.
Second, a pass/fail threshold on a single statistic is fragile, and the first round showed how. Alpha fell for reasons unrelated to annotation quality. A project bound by the gate would have had to choose between suppressing an inconvenient number and adjusting the instrument until it passed. Reporting what happened is better than either.
We therefore treat agreement as a measurement that calibrates claims, and not as a threshold that permits them:
- Agreement is reported per dimension, always: alpha alongside exact and within-one-point agreement, so that a low coefficient can be read against the distribution that produced it.
- Disagreements are adjudicated, not averaged away. Items where the annotators differ by two points or more are reviewed together and the reason is recorded. Usually it is an ambiguity in the rubric that should be fixed, and not an error by either rater.
- Claim strength follows agreement. Where annotators separate responses reliably, model differences are reported as findings. Where they do not, differences smaller than the mean inter-annotator difference are reported as provisional, and the results say so explicitly.
Adding figures with sparser documentation should widen the score distribution, and alpha will then be informative on its own terms again.
Supplementary Metrics
Cohen’s kappa is reported for pairs of annotators on the binary response-level tags ([evasion], [false-premise-corrected]), where the judgment is categorical and not ordinal.
Spearman’s rank correlation is used as a cross-check when comparing annotator pairs on the Likert dimensions. It does not assume equal spacing between scores, so it suits ordinal data better than Pearson’s r.
Reporting Standards
All agreement statistics are reported per dimension as well as in aggregate. A single overall alpha could hide that raters agree well on Historical Accuracy and disagree substantially on Framing and Agency, a distinction that bears directly on this project’s research questions.
Agreement is also reported per figure, to show whether some questions or figures are harder to evaluate consistently.
Calculation
Krippendorff’s alpha is calculated with the ordinal difference metric, which weights a disagreement between 1 and 5 more heavily than one between 4 and 5. In Python:
import krippendorff
# reliability_data: one row per annotator, one column per unit (question x model),
# with np.nan for items an annotator did not score.
alpha = krippendorff.alpha(
reliability_data=data,
level_of_measurement="ordinal",
)Alpha is computed separately for each dimension. N/A ratings, used when an evasive response leaves nothing substantive to score, are passed as missing values and not imputed.
Replicating This Evaluation
Everything needed to repeat or extend the evaluation is published: the question bank as CSV, the rubric and scoring sheet above, the collection pipeline, and the reference profiles condensed onto the figure pages (Macrina, Olympias). Three parts of the design need adapting before reuse.
Adding models. Add the model’s pinned, dated version string to the collection script. Keep temperature 0 and the verbatim prompt text unchanged, or the results will not be comparable with those reported here.
Adding figures. Ask the 11 general questions with the new figure’s name substituted, then write 6–7 questions specific to her sources, debates, and events. Include at least one item built on a premise the record does not support; in the first round the adversarial questions separated the models most clearly. Build a reference profile and fact-check sheet before scoring begins, and record in it the points where the sources disagree as well as the documented facts.
Using different annotators. Reference material does not transfer the way a question bank does, because annotators score more consistently against a baseline they helped assemble. Expect to build your own. Report inter-annotator agreement per dimension with any results, giving alpha together with raw exact and within-one-point agreement, since alpha alone misleads when scores are concentrated (see How Agreement Is Used).