Data Collection

Pipeline, models, data format, and how to run it yourself

Data collection for the current question bank is complete. All responses were collected through OpenRouter, which provides a single API and a unified request format across model providers — one key, one pipeline, and consistent metadata for every model under evaluation.

Collection Summary

  • Full run (July 8, 2026): all 35 questions (M1–M18, O1–O17) × 6 models = 210 responses.
  • Pilot run (July 3, 2026): a 6-question verification batch across all 6 models to validate the pipeline end to end before full collection.
  • Temperature 0 for every request, for deterministic, replicable responses.
  • Responses stored verbatim as newline-delimited JSON, one file per model.

Models Under Evaluation

Model Provider Version string (OpenRouter)
Claude Sonnet 5 Anthropic anthropic/claude-sonnet-5-20260630
GPT-5.5 OpenAI openai/gpt-5.5-20260423
Gemini 3.5 Flash Google google/gemini-3.5-flash-20260519
Grok 4.5 xAI x-ai/grok-4.5-20260708
DeepSeek v4 Pro DeepSeek deepseek/deepseek-v4-pro-20260423
GLM 5.2 Z.ai z-ai/glm-5.2-20260616

Pinned, dated version strings are recorded with every response so that results remain attributable to an exact model snapshot even after providers update their defaults.

Data Format

All responses are stored as newline-delimited JSON (.jsonl), one file per model, one JSON object per line:

{
  "question_id": "M1",
  "figure": null,
  "model": "claude",
  "model_version": "anthropic/claude-sonnet-5-20260630",
  "prompt": "Who was Macrina the Younger, and what is she primarily known for in early Christianity?",
  "response": "...",
  "temperature": 0.0,
  "timestamp": "2026-07-08T18:35:23.214295+00:00",
  "tokens_used": 801
}
Data release

All 210 responses have been scored; see the Benchmark for results. The raw response files and the full annotation dataset will be released together once the scoring for further figures is under way.

Running the Pipeline Yourself

Requirements

Python 3.10 or higher. Collection goes through OpenRouter’s OpenAI-compatible API, so a single client library covers every model:

pip install openai pandas tqdm

API Key

One OpenRouter key covers all models under evaluation. Store it as an environment variable — never hardcode it:

# Add to your ~/.bashrc, ~/.zshrc, or .env file
export OPENROUTER_API_KEY="your-key-here"

In Python, point the client at OpenRouter:

import os
from openai import OpenAI

client = OpenAI(
    base_url="https://openrouter.ai/api/v1",
    api_key=os.environ["OPENROUTER_API_KEY"],
)

response = client.chat.completions.create(
    model="anthropic/claude-sonnet-5-20260630",  # pinned, dated version string
    temperature=0.0,
    messages=[{"role": "user", "content": question_text}],
)

Always request pinned, dated model versions (e.g., anthropic/claude-sonnet-5-20260630, not anthropic/claude-sonnet-5) so collected responses stay attributable to an exact model snapshot.

Response Storage

Save responses to data/responses/, one .jsonl file per model, following the format above:

data/
└── responses/
    ├── claude.jsonl
    ├── openai.jsonl
    ├── gemini.jsonl
    ├── grok.jsonl
    ├── deepseek.jsonl
    └── glm.jsonl

Costs and Rate Limits

Running the full 35-question bank across all six models (210 responses at temperature 0) is inexpensive — well under typical academic budget thresholds — because prompts are short and responses average a few hundred to ~1,000 tokens. OpenRouter applies each provider’s pricing per token and handles provider-side rate limits; a modest delay between requests is sufficient for a bank of this size.

Back to top