Agent memory benchmarks can move 18 points without the memory changing at all. In two audits published this month, the same answers scored far apart once the grader, its prompt, or the evidence it was allowed to see changed. The gap is now large enough to decide a leaderboard, and it is the part of the score vendors disclose least.
The clearest case comes from MemStrata, a local-first memory tool that surfaced on Hacker News on 19 October 2026. Its author-run preprint reports both grading methods side by side for the same answers, instead of publishing only the higher number.
The grader is part of the agent memory benchmarks score
On LongMemEval-S, MemStrata scored 95.0 percent (475 of 500) under source-aware grading, where the judge checks the answer against the full source history, and 92.6 percent (463 of 500) under reference-only grading, where the judge sees only the short benchmark reference. On LoCoMo, the gap widened: 90.91 percent (1,400 of 1,540) source-aware against 78.25 percent (1,205 of 1,540) reference-only, a difference of 12.66 points from grading alone.
MemStrata is explicit about the limits. The results are author-run, use a single automated judge model, GPT-5.5, rely on a local Qwen 3.8 27B Q4 reader that is not part of the product, and are, in its own words on its site, not independent replication or a held-out customer evaluation. A same-seed replay of unchanged requests also changed 1.5 to 2.3 percent of labels. That candour is the point: a reader can see how much of the headline is the system and how much is the measurement.
One fixed set of answers, four judges
A second audit, published 21 October 2026, took one fixed set of LoCoMo predictions and ran it through four judge configurations. Only the judge model and the grading prompt changed. Overall accuracy moved from 67.1 percent to 85.1 percent, an 18-point spread, and multi-hop accuracy moved 35 points. Holding the judge fixed and switching prompts alone moved overall accuracy 10.8 points on one judge and 14.2 points on another.
The same audit found a second problem underneath the first. It excluded 164 of 1,986 LoCoMo questions, 8.3 percent, as unreliable, including wrong gold answers and citations that did not contain the answer, and it argues that any system reporting above roughly 91 percent is partly being scored on agreement with benchmark noise. Most papers also exclude LoCoMo's 446 adversarial questions, the only part that penalises guessing, so a system can gain on recall while getting worse at refusing a question it should not answer, and the headline score will show only the gain.
These are not isolated quirks. A field guide to AI memory benchmarks notes there is no standard evaluation protocol for LoCoMo, LongMemEval or BEAM, that each paper picks its own judge, reader, retrieval depth and dataset subset, and that vendor self-claims and independent evaluations routinely diverge by 18 to 24 points on LoCoMo.
What to ask before trusting a memory score
Four disclosures would make most published numbers comparable, and the October audit lists them directly: report a deterministic score such as F1 alongside judge accuracy, publish the full judge configuration including model, prompt and temperature, report adversarial results separately, and report raw and error-corrected numbers with the error list named. A buyer does not need a new benchmark to ask for those four items in a security or evaluation review.
The same questions apply inside a company, where the stakes are higher than a leaderboard. A shared record that agents read from, as we set out in Shared Agent Memory Is Here. The Record It Creates Is the Hard Part, amplifies whatever it holds. If the evaluation that certified that record cannot name its judge, its prompt, or its error list, the company has no way to tell a correct fact from a confidently graded one. The measurement problem and the record problem are the same problem, which is why context engineering has to include how answers are graded, not only what context is retrieved, and why How Jilu Works treats provenance and a review gate as part of the record itself.
A memory score is a claim about a record. Like any claim in that record, it should carry its source, its method and its date, and it should survive a second reader checking it against the original. Until agent memory benchmarks report the grader as routinely as they report the score, the honest reading of a headline number is not that a system remembers 95 percent of what it was told. It is that, under one undisclosed way of counting, it was marked that way once.