Methodology

llmmetric is built by Respan and is editorially independent. Our goal is a neutral, verifiable reference. Here is exactly how each number is produced.

Quality / benchmark scores

Aggregated from public sources only (LMArena human-preference Elo, SWE-bench, and official model cards for MMLU-Pro, GPQA, AIME, HumanEval, and similar). Every number links to its original source. We never publish our own eval scores as a benchmark.

Independent benchmark runs (Epoch AI)

Labs publish different benchmarks, which makes models hard to compare. To give every model common axes we also include Epoch AI's own evaluations, where one team runs every model under one setup. These rows are always labelled "(Epoch AI run)" and link to Epoch's page. For scaffold-free benchmarks (GPQA Diamond, SimpleQA Verified) an Epoch run fills the same axis as a lab-reported score, and the lab's own number wins whenever both exist. For agentic benchmarks such as SWE-bench Verified the harness changes the result, so Epoch's run stays a separate axis. Data: Epoch AI, 'Capabilities & Benchmarking', epoch.ai/benchmarks, licensed CC-BY 4.0. AIME 2025 and AIME 2026 scores labelled "(MathArena)" come from MathArena (ETH Zurich SRI Lab), which runs every model without tools and averages repeated attempts. MathArena flags rows where the model was released after the competition; we carry that flag as a ⚠ because contamination is possible.

Operational metrics (latency, uptime, status)

Measured by independent probes that call each provider's API directly from neutral infrastructure. These probes do not route through the Respan gateway, so the numbers reflect the provider itself, not our routing. When direct credentials are unavailable, the availability percentage is the observed state from the provider's official feed. Official incident windows mark affected days and power incident history, but are not treated as full-provider downtime because many incidents are regional, partial, or limited to one product.

Pricing

Taken from official provider pricing pages and updated regularly; each figure links to its source and shows a last-updated date.

Corrections

Found something wrong or out of date? The data is open via our API and we welcome corrections.