Pages
On this page
Axioms.ai studies how external inputs to language models, from structured text, numbers, and executable code to live RAG retrieval and reasoning rules, interact with pre-trained weights to change output accuracy, domain depth, latency, and cost.
Our mission
We study grounding, connecting a model’s probabilistic generation at runtime to verifiable facts, structured numbers, code, and explicit rules instead of relying on slow, lossy fine-tuning alone. Effective grounding stays lean, because oversized, noisy prompts dilute attention and inflate latency.
We also work on neuro-symbolic computing, pairing neural language models (built for fluent synthesis) with deterministic symbolic structures such as code, knowledge graphs, formal constraints, and external verifiers that never guess. Our lab tests where neural intuition should end and symbolic grounding should take over.
How text, numerical tables, code, RAG passages, and reasoning guardrails compete with pre-trained priors and shift accuracy, coverage, token cost, and latency.
State of the researchWhich tasks a language model should reason through in its weights, which need distilled static axioms, and which must route to deterministic code or live APIs.
State of the researchWhere frontier models guess on complex, multi-constraint questions across commerce, law, health, finance, and software, weighted by real-world harm.
Explore the benchmark designMeasuring the cost, speed, and human-audited reliability of adversarial multi-lab verification pipelines as models evolve.
Read our methodsState of the research
Our experiments build directly on peer-reviewed findings from academic labs and frontier AI research groups. Six results across the recent literature define the problem space we test.
Weights vs. RAG
When models encounter rare or newly updated facts, scaling parameters or running standard fine-tuning rarely sticks without massive paraphrase repetition. Across 14,000 questions in PopQA, unassisted GPT-3 (davinci-003) dropped to 19% exact accuracy on the 4,000 least-popular tail entities, while selective retrieval let smaller models beat unassisted giants. At the same time, post-training weights specifically to read retrieved documents alongside distractors (RAFT) outperforms both standalone fine-tuning and naive RAG.
RAG limits
Fetching raw text passages does not guarantee truthful generation. In Stanford and Yale’s pre-registered audit of commercial RAG-grounded legal research tools (Lexis+ AI, Westlaw AI-Assisted Research), systems still hallucinated on 17-33% of queries. Stanford’s ClashEval further shows a persistent tug-of-war between model priors and injected context, where clean evidence lifts accuracy to 94% while wrong or outdated snippets override a model’s correct prior over 60% of the time, and up to 57% of RAG citations in tested setups act as post-hoc rationalizations.
Structured RAG
Splitting documents into arbitrary token windows severs relationships across facts. Recent work shows major gains from structuring grounding inputs before generation. Microsoft’s GraphRAG builds hierarchical entity and community summaries for global synthesis, OSU’s HippoRAG (NeurIPS 2024) indexes entity graphs for single-step multi-hop retrieval, and Self-RAG (ICLR 2024) trains models to trigger retrieval on demand and critique whether each generated segment is supported.
Context physics
Advertised million-token windows hide steep reasoning cliffs. On NVIDIA’s RULER benchmark, half of models claiming 32K+ context failed to maintain usable accuracy at 32K tokens. Accuracy drops 20+ points when key facts sit in the middle of a prompt, even falling below closed-book baselines on multi-document QA (Lost in the Middle); reasoning degrades as input tokens grow on identical tasks (Same Task, More Tokens); and irrelevant sentences slash math and logic consistency unless models are given explicit filtering rules.
Neuro-symbolic
Autoregressive language models excel at proposing candidates and interpreting nuance, but cannot reliably self-verify formal rules or arithmetic. DeepMind’s AlphaGeometry solved 25 of 30 Olympiad geometry problems (up from 10) by pairing a neural model with a symbolic deduction engine; Kambhampati et al.’s LLM-Modulo framework pairs LLMs with sound external verifiers; UCLA’s Ctrl-G enforces deterministic logical constraints at decoding time; and Chain of Code (+12 points on BIG-Bench Hard) offloads numeric and algorithmic steps to code execution.
Evaluation
OpenAI and Georgia Tech researchers demonstrate mathematically that grading benchmarks like zero-penalty multiple-choice exams rewards models for guessing confidently instead of saying “I don’t know.” Meanwhile, NeurIPS 2024 and ICLR 2026 studies show that LLM evaluators recognize and favor their own generations (Panickssery et al.) and exhibit up to 23.6% preference leakage when generator and judge share synthetic-data training lineage (Li et al.), requiring cross-lab judge panels and human calibration.
Our research program
All studies below are currently in progress. Empirical grounding and agent studies carry pre-registrations, while our benchmark versions question sets with public dataset hashes; full results, raw model outputs, cross-lab judge verdicts, human spot-check logs, and scoring code publish as each study completes.
In progress
Our public benchmark in development measuring how frontier AI assistants handle complex, multi-constraint questions where answers depend on interacting mechanisms, conditional rules, and domain trade-offs. Errors are weighted by real-world harm (10x for safety, clinical, or legal risk; 3x for financial or spec loss; 1x for minor slips) alongside coverage and abstention rates, beginning with consumer commerce on version-hashed question sets and expanding to law and additional domains over time.
Explore the benchmark designIn progress
Our flagship multi-arm empirical study, building on seven controlled October 2026 evaluations (~9,900 graded AI answers) where grounding in verified axioms cut wrong claims per answer by 85% on small models (from 0.167 to 0.025, and from 0.125 to 0 on Claude Opus 5.5 across 40 graded pairs), and Step 2 source rewriting lifted statement citation accuracy from 41/100 to 81/100 across 1,111 forged axioms (604 rewritten). The unified multi-arm comparison tests unassisted baselines, domain-free reasoning guardrails, equal-length placebo text, raw retrieval at an identical token budget, deep research reports, and rules-first compiled grounding with source-bound axioms, validated by cross-family AI judges and blind human spot-checks.
Read our 5-step protocolIn progress
Pre-registered studies in progress on how Markdown Truth Packs shift accuracy, depth, and coverage across small and frontier models. A preliminary internal test of our Indoor Air Quality sample pack (57 prompts, 1,949 blind-judged answers, 6 models) cut small-model wrong claims 54% (0.212 to 0.098) and Gemini 4 Argon errors 78%, and lifted adjacent-topic coverage 9.4 points.
Try the sample packIn progress
In progress with Product.ai Research for publication on product.ai/research. Because commercial agent vendors publish no task-level behavioral logs, this pre-registered study (with pilot runs underway) dispatches live browsing agents across ten fixed multi-step shopping errands, halting at payment to record what they check, where they misread prices or policies, and where they stall.
How we work
The pipeline that drafts an axiom never scores it, and every Axioms.ai benchmark and grounding study enforces strict cross-lab judge exclusion. Because language models recognize and favor text from their own training lineage (Panickssery et al., NeurIPS 2024; Li et al., ICLR 2026), our AI judges always come from competing labs, and we pair cross-lab machine scoring with blind human spot-checks on sampled outputs.
cross-lab AI judges score every answer against an independent evidence key.
An Anthropic answer is judged by Google and OpenAI models; an OpenAI answer is judged by Anthropic and Google models. A claim counts as supported only when both judges agree.
open artifacts committed with every study we release, including prompts, raw answers, verdicts, and scoring scripts.
Empirical study pre-registrations and versioned benchmark dataset hashes publish openly, and negative or refuted findings are reported alongside positive ones.
Human calibration
Automated AI judges let us evaluate thousands of responses consistently, while targeted blind human reviews check that our scoring rubrics hold up against primary source evidence without revealing which model or grounding condition produced an answer.
Rather than locking in fixed human-audit quotas while studies are still in design, we scope human spot-checks to each study, focusing on sampled claims and disputed edge cases where cross-lab AI judges split, and publish the exact sample size and review protocol alongside each release.
model names or condition labels shown to reviewers during blind human spot-checks.
Contributing readers are credited by name with their consent.
Transparent reporting
Every published results table will record the exact model snapshot, grounding assets, token budget, and capture date, reporting error rates with 95% confidence intervals alongside answer coverage so no model tops a table simply by refusing to answer.
Common questions
Axioms.ai investigates how runtime context changes language model accuracy, domain depth, token cost, and latency compared with relying on pre-trained weights alone. Our research lab tests 6 core areas: parametric weight limits on tail facts, retrieval noise in standard RAG, graph-structured context, prompt-length attention decay, neuro-symbolic code and verifier handoffs, and multi-lab AI evaluation methods.
Peer-reviewed evaluation studies show that language model judges recognize and favor text from their own training lineage (Panickssery et al., NeurIPS 2024) and exhibit up to 23.6% preference leakage when generator and judge share synthetic-data roots (Li et al., ICLR 2026). In every Axioms.ai study, answers are scored solely by 2 competing-lab judges alongside blind human sample audits.
Axioms.ai and Product.ai split research publication cleanly by audience and subject. Model-level grounding experiments, context-budget physics, cross-lab judge calibration, and multi-domain benchmarks publish here on axioms.ai/research/. Applied consumer shopping studies, category prediction scorecards, and live browsing evaluations such as our upcoming Autonomous Shopping Agents study publish on Product.ai Research at product.ai/research/.
Public data repositories, pre-registrations, and scoring code will publish alongside each study as our initial benchmark and grounding evaluations wrap up.