Kimi K3.1 Pro Gets Half Its Knowledge Questions Right and Answers the Other Half Confidently

Kimi K3.1 Pro Gets Half Its Knowledge Questions Right and Answers the Other Half Confidently

The benchmark that separates Kimi K3.1 Pro from its reputation is not a reasoning test. It is AA-Omniscience, which asks the model what it knows and then measures two things separately: how often it is right, and how often it is confidently wrong. Kimi K3.1 Pro scores 51.02% accuracy and a 43.89% hallucination rate, for an Omniscience Index of 29.52 [INDEPENDENT, index v4.3.2, 26 October]. Claude Opus 5.5 scores 66.21% accuracy and 46.42 on the index. The gap on the composite is 16.9 points — nearly seven times the 2.53-point gap on the general index. OrcaRouter‘s Kimi K3.1 card carries both rows next to a $2.00/$10.00 rate, and the pair is the most actionable thing on it.

Put plainly: this model reasons near the frontier and knows considerably less than the frontier, and when it does not know, it does not reliably say so.

The row, in context

ModelOmniscience indexAccuracyHallucination
Claude Opus 5.546.4266.21%—
Claude Fable 5.143.4567.23%—
GPT-6 Astra43.4062.60%—
Gemini 4 Argon42.3549.90%—
GPT-6.1 Sol41.5362.10%—
Claude Sonnet 5.532.3053.95%—
Grok 4.732.0347.45%—
Kimi K3.1 Pro29.5251.02%43.89%
Kimi K3 (Max)19.7045.75%77.11%
GLM-5.314.3033.85%—

Two things in this table are worth staring at.

Kimi’s accuracy is middle of the pack — but its index is near the bottom. Gemini 4 Argon answers 49.90% correctly, slightly *worse* than Kimi’s 51.02%, yet scores 42.35 on the index against Kimi’s 29.52. The index weights wrong-but-confident answers differently from admitted ignorance, and that is the whole difference: Kimi gets about half the questions right and, on the others, produces a confident wrong answer 44% of the time.

The improvement over its own predecessor is enormous. Omniscience went from 19.70 to 29.52; accuracy from 45.75% to 51.02%; hallucination from 77.11% to 43.89%. In the previous generation, three out of four wrong answers were confidently wrong. In this one, fewer than half are. That is a change in behaviour, not just in score, and it is arguably the most valuable thing in the release for anyone deploying this model without retrieval.

Why knowledge is the weak axis

Three mechanisms, all of them consistent with what the model is.

It is a post-training release on an unchanged backbone. Moonshot describes K3.1 Pro as a post-trained version of Kimi K3 with the same 2.8T-parameter, 104B-active mixture-of-experts backbone [VENDOR]. Post-training changes behaviour — how the model reasons, formats, refuses, uses tools. It does not add facts to the weights in any organised way, and reinforcement learning on task outcomes can *reduce* the model’s willingness to say “I don’t know” if hedging is scored as failure.

The training focus is agentic and reasoning-heavy. The headline rows are Terminal-Bench, code benchmarks, SWE Marathon, financial modelling. Knowledge benchmarks are not where the compute went.

The 1M context is an implicit substitute for knowledge. A model sold on reading a million tokens of your corpus does not need to memorise facts about your corpus. The vendor’s positioning and the model’s weaknesses are the same fact viewed from two sides.

See also: Building a Certificate and Credential Management System: Auto-Generation, Verification, and Blockchain

What this does to a deployment

The distinction that matters is whether the answer lives in the prompt or in the weights.

In the prompt — use this model freely. Extraction, summarisation, code transformation, reasoning over supplied documents, comparison, classification. The knowledge row is irrelevant because the knowledge is right there, and the reasoning row is strong.

In the weights — do not use it alone. Factual questions, citations, domain constants, regulatory details, dates, product specifications. Here the failure is not a refusal; it is a plausible wrong answer delivered in the same register as a correct one, which is the expensive kind of error. A model with lower accuracy and better calibration would be safer than one with higher accuracy and worse calibration, and this model’s calibration is the weaker half of its pair.

Mixed — separate the two. A two-step pattern handles most real workloads: retrieve the source material first, then ask the model to reason over it. A retrieval step costs a hundredth of what a wrong answer costs, and it moves the task from the model’s weakest axis to its strongest.

When the context is empty and you must use it — a last-resort pattern that works better than trusting a single pass: ask the same question twice with different framings and compare the answers. Kimi’s hallucination rate means a disagreement between two runs is a strong signal of uncertainty, and the second call is cheap relative to the error it prevents.

The number behind the number

A 44% hallucination rate gets misread as “44% of answers are wrong”. It is not that. It is the share of *incorrect* answers that are delivered confidently rather than hedged or refused, on the specific knowledge questions in this evaluation. On the same evaluation, 51% of answers were correct. So the model is not fabricating at random; it is answering at the accuracy of a mid-tier knowledge model and covering its uncertainty too often.

The comparator that makes the point sharpest is Claude Fable 5.1: 67.23% accuracy, the highest in the table, and an index of 43.45 — below Opus, which has slightly lower accuracy. Even at the top of this column, the index is punishing confidence, not ignorance.

What to tell a reviewer

If a stakeholder asks whether Kimi K3.1 Pro is “good”, the accurate answer has three clauses. It is fourth on the general index, within 2.53 points of first. It leads every frontier competitor we compare it to on Terminal-Bench 4.0 for terminal work and on long-context recall. It is eighth of ten on factual knowledge, above only its own predecessor and GLM-5.3, with a hallucination rate of 44%. All three are true of the same model on the same day, and the deployment decision is which clause describes your traffic.

For most enterprise work the first and second clauses are the ones that matter, because most enterprise traffic supplies its own context. The third clause is the one that decides whether you need a retrieval layer, and the answer is usually yes.

Sourcing note: all Omniscience, accuracy, hallucination, HLE and MMMU-Pro figures are Artificial Analysis on index v4.3.2, read 26 October 2026, and include the predecessor row (K3 at 19.70 index, 45.75% accuracy, 77.11% hallucination). AA’s launch note quotes the hallucination change as 53% to 44%, which describes the same movement on a slightly different basis; we use the model page figures throughout for consistency. The post-training description and architecture are Moonshot’s. Deployment patterns and the two-pass technique are ours.

Share your love

Leave a Reply

Your email address will not be published. Required fields are marked *