Pages
On this page
Hallucinations at the Edge is an upcoming public benchmark from Axioms.ai measuring how leading AI assistants handle complex questions across real-world domains. Instead of counting every error alike, we weight each false claim by the harm it causes someone who acts on it.
The questions
A question is complex when no single web search or simple fact lookup can settle it. Answering well requires weighing interacting mechanisms, conditional rules, edge-case exceptions, and competing trade-offs in territory sparsely or inconsistently covered in an AI’s training data. That edge is where AI assistants stop reasoning from evidence and start guessing.
Our initial benchmark release evaluates consumer commerce across nine everyday categories (headphones, sunscreen, skincare, supplements, smartphones, laptops, TVs and monitors, mattresses, and robot vacuums), focusing on multi-constraint recommendations, compatibility rules, safety interactions, and real-world trade-offs. Planned domain expansion includes law as an independent public benchmark vertical, followed by health, finance, and software. Question sets are versioned with public dataset hashes so runs over time remain directly comparable.
hallucination rate of OpenAI’s o3 reasoning model on SimpleQA when answering short, fact-seeking tail questions without external tools.
Smaller reasoning model o4-mini hallucinated on 79% of SimpleQA prompts, and even on Google DeepMind’s audited SimpleQA Verified benchmark (Sep 2025), leading newer frontier models including Gemini 2.5 Pro (55.3% accuracy) and GPT-5 (50.9% accuracy) still failed to answer 45-49% of questions correctly from weights alone. Sources: OpenAI o3 & o4-mini System Card (Apr 2025) and Haas et al., DeepMind SimpleQA Verified (Sep 2025).
of queries where specialized, RAG-grounded commercial legal AI tools still produced hallucinated facts or citations.
General-purpose chat models hallucinated on 58-88% of legal queries when tested without domain retrieval. Sources: Magesh et al., Stanford & Yale (JELS 2025) and Dahl et al., Stanford RegLab (2024).
awarded for saying “I don’t know” across nearly every standard AI leaderboard.
Binary pass/fail grading mathematically rewards models for guessing confidently on uncertain edge questions instead of abstaining cleanly. Source: Kalai, Nachum, Vempala & Zhang, OpenAI & Georgia Tech (2025).
State of the research
Public benchmarks have substantially improved how researchers measure factual errors, yet each existing design leaves an open gap when applied to complex, multi-domain questions.
Document fidelity
Benchmarks such as Vectara’s Hallucination Leaderboard (HHEM) and Google DeepMind’s FACTS Grounding supply a reference document in the prompt and measure whether the model’s response stays faithful to that text. They isolate reading comprehension and summarization slips, while leaving untested what happens when everyday users ask multi-constraint questions without already holding the right source documents.
Parametric tail
OpenAI’s SimpleQA, DeepMind’s SimpleQA Verified (2025), and PopQA test single-fact questions with short static answers against unassisted models, showing that even newer frontier models miss nearly half of tail facts without external tools. Yet single-hop trivia lookups do not test how live web-connected assistants synthesize interacting rules, trade-offs, and constraints, or distinguish harmless slips from dangerous advice.
Bluffing incentives
Recent theoretical work from OpenAI and Georgia Tech demonstrates that grading benchmarks like standard exams, where an “I don’t know” earns zero points alongside a fabricated claim, pushes post-trained models to guess whenever they are uncertain. Separating harm-weighted error rates from coverage and explicit abstentions removes the reward for bluffing.
Consequence rubrics
OpenAI’s physician-rubric HealthBench (5,000 realistic health conversations evaluated against 48,562 rubric criteria written by 262 physicians) and Stanford RegLab’s legal AI audits show that flat claim counts hide the errors that actually hurt people. Hallucinations at the Edge extends consequence-weighted scoring into an open multi-domain benchmark with strict cross-lab judge exclusion.
The harm tiers
Before any assistant’s answer is scored, models from two labs (GPT-5.5 with web search and Gemini 3.1 Pro) build an evidence key for each question and assign a harm tier to every disqualifying error, keeping points both labs confirm and disclosing key provenance alongside verdicts. Where the two labs assign different tiers, the stricter tier applies and we record the split. Scoring judges never see the tier weights.
Someone acting on the claim risks physical injury, illness, regulatory violation, or loss of a legal right.
Such as fiberglass inside a mattress cover, unsafe infant sleep firmness, a supplement drug interaction, a missed statutory filing condition, or an invented legal right.
A false warranty rule, compatibility claim, return condition, or technical specification that costs money or leads to the wrong decision.
Such as a misstated return eligibility rule, an incompatible hardware pairing, a wrong battery spec, or an inaccurate quoted fee.
An inaccurate background detail that leaves someone acting on the answer no worse off.
Such as an off-by-one company founding year or a tangential historical detail that does not alter the decision.
The default view sorts by the 10x / 3x / 1x harm-weighted score, alongside toggleable sensitivity cuts for unweighted errors (1x / 1x / 1x) and steep safety weighting (100x / 10x / 1x) so readers can see how weighting shifts rankings.
The judges
Peer-reviewed studies show that language model evaluators recognize and favor generations from their own training lineage (Panickssery et al., NeurIPS 2024; Li et al., ICLR 2026). We use a three-lab judge pool (Claude Opus 5.5, Gemini 3.1 Pro, and GPT-5.5) and route every answer exclusively to the two judges outside its creator’s lab. A false claim counts only when both cross-lab judges independently agree it was asserted.
Where an evaluated Perplexity configuration routes answers through OpenAI models underneath, OpenAI judges are excluded from scoring Perplexity as well. Alongside machine verdicts, each benchmark release includes blind human spot-checks on sampled answers and disputed judge calls.
| Answers captured from | Scored independently by | Excluded judge family |
|---|---|---|
| ChatGPT & Perplexity | Claude Opus 5.5 + Gemini 3.1 Pro | OpenAI (GPT family) |
| Claude | Gemini 3.1 Pro + GPT-5.5 | Anthropic (Claude family) |
| Gemini | Claude Opus 5.5 + GPT-5.5 | Google (Gemini family) |
The board · Wave 1 in scoring
Each domain has its own scoreboard across eight assistant configurations (ChatGPT, Claude, Gemini, and Perplexity, both free and paid tiers). We report harm per 100 answers with 95% bootstrap confidence intervals, false claims per 100 answers by tier, factual coverage, and explicit abstentions, ensuring no assistant scores well simply by refusing to answer.
| Assistant | Tier | Pinned model & surface | Harm / 100 | Tier 1 | Tier 2 | Tier 3 | Coverage | Abstains |
|---|---|---|---|---|---|---|---|---|
| ChatGPT | Free | chatgpt.com · default web | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress |
| ChatGPT | Paid | chatgpt.com · Plus / Pro | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress |
| Claude | Free | claude.ai · Sonnet web | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress |
| Claude | Paid | claude.ai · Opus 5.5 web | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress |
| Gemini | Free | gemini.google.com · Flash | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress |
| Gemini | Paid | gemini.google.com · 4 Argon / 3.1 Pro | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress |
| Perplexity | Free | perplexity.ai · standard search | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress |
| Perplexity | Paid | perplexity.ai · Pro search | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress | Wave 1 scoring in progress |
Wave 1 consumer commerce scoring across 9 categories is currently in progress. In seven completed October 2026 internal grounding evaluations (~9,900 graded AI answers), grounding small models in verified Truth Graph axioms reduced wrong claims per answer by 85% (from 0.167 unassisted to 0.025 grounded, and from 0.125 to 0 on Claude Opus 5.5 across 40 graded pairs). Each benchmark release logs the exact model snapshot, capture surface, and capture date for every evaluated arm.
Planned domain roadmap
We plan to expand the benchmark domain by domain over time while re-scoring earlier domains on version-locked question sets, tracking whether model updates reduce or introduce costly errors.
Nine product categories across eight free and paid assistant configurations, releasing with full raw answers, evidence keys, and cross-lab verdicts.
Multi-factor statutory rules, tenant and consumer rights, and procedural exceptions evaluated as an independent public benchmark domain, with no legal product behind it.
Additional professional and technical domains planned on version-locked question batteries so longitudinal movement stays transparent as models evolve.
Common questions
Hallucinations at the Edge is an upcoming public AI benchmark built by Axioms.ai to test how ChatGPT, Claude, Gemini, and Perplexity handle complex, multi-constraint questions across free and paid tiers. Rather than testing single-fact trivia or closed-document summaries, the benchmark scores real-world trade-offs, conditional rules, harm-weighted error rates, factual coverage, and explicit abstentions.
Axioms.ai assigns every disqualifying error in the benchmark evidence key to one of 3 consequence tiers before scoring begins. Tier 1 safety, clinical, or legal hazards carry a 10x weight; Tier 2 financial, warranty, compatibility, or specification losses carry a 3x weight; and Tier 3 minor factual slips carry a 1x weight, alongside toggleable unweighted and 100x safety views.
In Hallucinations at the Edge, Axioms.ai uses a 3-lab judge pool (Claude Opus 5.5, Gemini 3.1 Pro, and GPT-5.5) and routes every answer solely to the 2 judges outside its creator’s lab. When an evaluated Perplexity tier routes through OpenAI models, OpenAI judges are excluded there too, and false claims count only when both cross-lab judges agree.
Initial scoring for the 9-category consumer commerce release is currently in progress at Axioms.ai, with law planned next as an independent public benchmark vertical on version-hashed question batteries. Every domain scoreboard will publish openly alongside its versioned prompts, exact model snapshots, raw assistant responses, 2-lab evidence keys, cross-lab verdicts, and human audit logs.