Results
The numbers behind the benchmark charts
Six frontier models answered the same 35 questions about Macrina the Younger and Olympias of Constantinople. Two annotators scored every response independently on four rubric dimensions, giving 420 ratings over 210 responses. The interactive charts are on the Benchmark page; this section gives the numbers and their interpretation.
Summary
- The models are close together. Overall means run from 4.56 to 4.73 on a 1 to 5 scale, a spread of 0.17. No model failed on this material, and no annotator gave a 1 on any dimension.
- Epistemic Quality is the weakest dimension. It averages 4.46, against 4.75 for Historical Accuracy, and it is where the models differ most (4.19 to 4.71). The models know the facts better than they know the limits of the evidence.
- The false-premise questions separate the field. On the five questions built on a premise the sources do not support, Claude corrected the premise in 10 of 10 ratings and Gemini in 4 of 10.
- Five of the six models score lower on Olympias than on Macrina, although Olympias is the better documented of the two.
- Outright refusal almost never happened. We recorded one [evasion] tag in 420 ratings. Flattening was common: [source-conflation] was applied 25 times and [debate-flattened] 24 times.
The Ranking
Claude and GPT-5.5 finish 0.01 apart and get there by different routes. GPT-5.5 has the best factual record of any model (4.87) and is fourth on Epistemic Quality; Claude is the only model above 4.7 on that dimension. Gemini has the most uneven profile: first on Coverage (4.83) and last on Epistemic Quality (4.19). It says a great deal about each question and distinguishes its sources least reliably. GLM is last overall yet first on Framing and Agency (4.74), so the overall rank does not tell you which kind of failure a model is prone to.
Hard Questions
Some questions are hard for every model and some for only one. O17, the invented arson plot, is the lowest item in the bank (4.02) and pulls down four of the six profiles; Claude (4.75) and Grok (4.62) are the exceptions. M11, an “inevitability” probe, is the next lowest (4.38). Some dips belong to a single model: Claude at M6 (key figures in Macrina’s life, 3.88), Grok at M11 (3.75), and GLM at O13 (Sergia’s account of the relics, 3.50).
Where the Olympias Gap Comes From
The Olympias gap has a different source for each model. GPT-5.5 drops by about a quarter of a point on three dimensions at once: Historical Accuracy (5.00 to 4.74), Coverage (4.89 to 4.64) and Framing and Agency (4.81 to 4.50). DeepSeek loses most on Historical Accuracy and on Framing and Agency (about 0.19 each). GLM’s drop is in Coverage (4.64 to 4.35), while its Historical Accuracy rises on Olympias. Claude, Gemini and Grok change by 0.13 or less on every dimension.
Detailed Tables
Mean score per dimension, averaged over both annotators (n = 70 ratings per model). This table holds the same numbers as the leaderboard.
| Model | Historical Accuracy | Coverage | Framing & Agency | Epistemic Quality | Overall |
|---|---|---|---|---|---|
| Claude Sonnet 5 | 4.81 | 4.66 | 4.71 | 4.71 | 4.73 |
| GPT-5.5 | 4.87 | 4.77 | 4.66 | 4.57 | 4.72 |
| Grok 4.5 | 4.73 | 4.79 | 4.57 | 4.63 | 4.68 |
| Gemini 3.5 Flash | 4.77 | 4.83 | 4.64 | 4.19 | 4.61 |
| DeepSeek v4 Pro | 4.69 | 4.73 | 4.63 | 4.27 | 4.58 |
| GLM 5.2 | 4.63 | 4.50 | 4.74 | 4.39 | 4.56 |
| All models | 4.75 | 4.71 | 4.66 | 4.46 | 4.65 |
Distribution of all 420 ratings on each dimension:
| Dimension | Mean | 5 | 4 | 3 | 2 | Best model | Weakest model |
|---|---|---|---|---|---|---|---|
| Historical Accuracy | 4.75 | 84% | 9% | 5% | 2% | GPT-5.5 (4.87) | GLM (4.63) |
| Coverage | 4.71 | 78% | 16% | 5% | 1% | Gemini (4.83) | GLM (4.50) |
| Framing & Agency | 4.66 | 72% | 23% | 4% | 1% | GLM (4.74) | Grok (4.57) |
| Epistemic Quality | 4.46 | 57% | 32% | 10% | 1% | Claude (4.71) | Gemini (4.19) |
No rating of 1 was given on any dimension.
Epistemic Quality is where the models struggle. Only 57% of responses earned a 5 on it, against 84% on Historical Accuracy. The tags show why: [source-conflation] (25 uses) and [debate-flattened] (24 uses) together outnumber all the factual-error tags combined. The models generally state correct facts, but they treat hagiography, correspondence and modern scholarship as equally authoritative, and they settle scholarly debates that the sources leave open.
Framing and Agency did not behave as we expected. It is not the lowest dimension, and its failures are mild: 23% of ratings sit at 4 rather than 5, usually for one instance of unnecessary qualification or of centering a male figure, not for erasure. The dimension does drop on the questions written to probe bias (4.50, against 4.71 on the rest), which suggests the framing weakness appears under pressure more than by default.
| Model | Macrina the Younger | Olympias of Constantinople | Difference |
|---|---|---|---|
| Claude Sonnet 5 | 4.74 | 4.71 | −0.02 |
| GPT-5.5 | 4.83 | 4.59 | −0.24 |
| Grok 4.5 | 4.67 | 4.68 | +0.01 |
| Gemini 3.5 Flash | 4.62 | 4.60 | −0.02 |
| DeepSeek v4 Pro | 4.65 | 4.51 | −0.14 |
| GLM 5.2 | 4.61 | 4.51 | −0.10 |
| All models | 4.69 | 4.60 | −0.09 |
Five of the six models score lower on Olympias than on Macrina. Documentation alone would predict the opposite. Olympias is attested by a dedicated Life, seventeen surviving letters from Chrysostom, and the accounts of Palladius and Sozomen, while Macrina is known almost entirely through two works by one of her brothers. The gap is widest for GPT-5.5, which is the strongest model on Macrina and fourth on Olympias.
Two features of the Olympias questions probably contribute. The set includes the two hardest items in the bank (O17 and O13, both of which require careful discrimination between sources), and the questions about her ordination as a deacon and her institutional authority invite the patronage-centered framing that the rubric penalizes.
Mean overall score by question difficulty:
| Difficulty | Mean | Ratings |
|---|---|---|
| Basic | 4.75 | 24 |
| Intermediate | 4.76 | 96 |
| General | 4.62 | 264 |
| Advanced | 4.45 | 36 |
Advanced questions score lowest, as intended. The general template questions score below Basic and Intermediate mainly because they include the two “inevitability” probes (M11, O11) and the misconception questions.
The flagged bias-probe questions (96 ratings) average 4.57, against 4.67 for the rest, and the drop is concentrated in Framing and Agency (4.50 against 4.71).
Five questions embed a premise the record does not support: that Macrina left writings (M16), that Olympias wrote theological treatises (O16), that an anti-Christian plot lay behind the arson charges against Olympias (O17), and that each figure’s achievements were historically inevitable (M11, O11). A model that identifies and corrects the premise receives the [false-premise-corrected] tag.
| Model | Premises corrected (of 10) | Mean score on these items |
|---|---|---|
| Claude Sonnet 5 | 10 | 4.95 |
| GLM 5.2 | 9 | 4.53 |
| GPT-5.5 | 8 | 4.43 |
| Grok 4.5 | 6 | 4.40 |
| DeepSeek v4 Pro | 6 | 4.38 |
| Gemini 3.5 Flash | 4 | 4.25 |
O17, the invented plot behind Olympias’s arson trial, is the hardest item in the bank (mean 4.02). Claude, Grok and GLM rejected the premise and scored 5 on Historical Accuracy. GPT-5.5, Gemini and DeepSeek elaborated the fabricated conspiracy and were scored 2 or 3 with a [hallucination] tag. Of all the items, this one shows most directly how sparse documentation turns into confident invention.