Preface
This benchmark is recent. Its vendor published it on 27 October 2026, and Japanese-language coverage followed. We are reading it because it measures a kind of failure that this blog’s one claim depends on: a system that gives a correct answer to the question it was asked, and never raises the question that mattered.
A checker can verify the answer it is given. It cannot verify the question nobody asked.
This post reads one benchmark post closely, notes what we could and could not confirm, and ends with a guess that is marked as one.
1. The Specimen
The benchmark is DAYJOB: Tax, published by Surge AI, a company that evaluates AI systems. It is the third in a family that earlier covered healthcare and finance. The post describes 50 environments across eight areas of tax practice, with 1,789 workspace files in total, a median of 30 files per assignment and as many as 123.
We read the text of the vendor’s blog post, its leaderboard page, the README of its evaluation repository and its dataset card, as they were supplied to us, along with secondary coverage. We did not read the code itself, and we did not run anything. Everything below is the vendor’s own account of its benchmark. It has not been independently verified.
2. What the Assignments Test
The post draws a contrast with earlier tax benchmarks. Those, it says, start from a defined problem: a question to answer, a return to calculate, or information to collect. These assignments start from a client request and a pile of records, and the agent has to work out what the problem is.
The post describes the records as messy: filed and draft returns, ledgers, payroll records, contracts, correspondence, and in one case a photograph of a handwritten receipt. It says tax experts built the tasks from real workflows, wrote the expected outcomes and grading criteria, and put each task through a three-layer expert review. The dataset card adds that the materials include synthetic documents modelled on the forms and records practitioners use.
3. What Came Back
On the vendor’s leaderboard, the highest pass rate was 10.8 percent for Claude Sonnet 5.5, followed by 9.6 percent for Claude Fable 5.1 and 9.2 percent for Claude Opus 5.5. The leaderboard page lists 16 entries, each tagged with its own reasoning-effort setting, such as High, xHigh or Max.
How to read those numbers matters. The post reports pass@1 at a 95 percent rubric threshold: an attempt passes only if it meets at least 95 percent of the task’s criteria. Those criteria cover calculations, supporting evidence, consequential issues, and the finished deliverable. The repository README says leaderboard runs use five attempts per task. With 50 tasks, that is 250 attempts in all, by our own arithmetic. The dataset card gives a median of about 30 rubric criteria per task, so by our arithmetic a task of that size leaves room to miss at most one criterion. We could not see in these texts how a trial’s reward and the 95 percent threshold are combined into the headline rate.
A low pass rate under a strict threshold is not the same as saying the systems cannot do tax work. It says they rarely met nearly every criterion in a single attempt.
4. Two Specimens
The post gives two examples, and we retell them as it describes them. We have not checked the tax rules involved.
In the first, a client named Anna had moved out of her condo and asked whether to sell in October 2026 or January 2027. The post says an agent correctly judged that either date met the residence requirement. It also says the agent left out that eligibility could lapse around the end of August 2027. Across five attempts, one system, Astra, never raised the deadline. Opus raised it in all five.
In the second, a separating couple asked how to divide a Roth IRA and a 401(k). The post says Opus 5.5 assumed in all five attempts that they were not legally married, even when it acknowledged uncertainty, although their own records pointed to a possible marriage. It says Astra recognised the need to verify marital status in four of five attempts.
The same system led on one and missed on the other. Neither example is about a wrong calculation. Both are about a question that was available in the files and did not get asked.
5. A Freer Note
This section is a guess, not a finding.
We can imagine that omission is a failure that cannot be seen from the answer alone. A wrong figure can be recomputed. A missing question leaves nothing on the page to recompute. To catch it, someone has to hold a list of what should have been asked, and that list has to live outside the system being checked.
This benchmark has such a list. The post says experts wrote the criteria, including a category for consequential issues. We read that as a human-written record of the questions the systems were supposed to find. That is the kind of external reference this blog keeps coming back to.
We can also imagine a weakness. If the one doing the checking shares blind spots with the one being checked, a missing question could be missed twice. The repository README says the grading is done by an automated language-model judge, Claude Opus 4.8, and the top three leaderboard entries are also Claude models. We do not know whether this affects the results, and we state the two facts without drawing a conclusion from them.
We hold this loosely. One vendor, 50 environments, and two worked examples cannot carry a general claim.
6. Checkpoints
We will look again on 31 January 2027 and 31 July 2027.
Checkable by anyone
- Does the vendor publish how a trial’s reward and the 95 percent threshold combine into the headline pass rate?
- Has anyone published a re-grading with an independent judge?
- Does any newer model, or any independent group, report a different pass rate?
Checkable only by the vendor
- How many attempts are run per task, and is run-to-run variation reported?
Checkable in the world
- Has any benchmark begun to score omission, the unasked question, as a category of its own?
7. What This Does Not Show
- It does not show that AI cannot do tax work. It shows low pass rates under a strict threshold on a vendor-designed set.
- It does not show whether the choice of judge affects the results. The repository names the judge. We did not test any other.
- We did not verify the tax rules in the examples. We retell them as the post presents them.
- The post is the vendor’s own. The vendor sells evaluation and invites readers to request benchmark access. The results have not been independently reproduced.
- We did not read the code or run anything. We read the repository README and the dataset card as text.
- Only two sample tasks are public. The dataset card says so, which limits outside checking of the full set.
- It does not show that omission is the main cause of failure. It shows two examples.
- The models named are as the post and leaderboard name them. We cannot confirm all of them.
8. Conclusion
In two worked examples, the systems gave answers and left out something the files already pointed to. The benchmark found this because experts had written down what should have been found.
We do not know how far this reaches. For this blog, it is a small, concrete case of why an answer can be checked from the inside and a question cannot.
A checker can verify the answer it is given. It cannot verify the question nobody asked.
We will look again on 31 January 2027.
Yoshimichi Kumon Organizer, LSI Inventor, ARDS/ARKS (PCT/JP2026/006503) Visiting Researcher, Waseda BFC MIT Sloan + CSAIL
References
- Surge AI, “DAYJOB: Tax — Benchmarking AI Agents on Professional Tax Work,” 27 October 2026 (vendor post; text supplied to us). https://surgehq.ai/blog/dayjob-tax
- Surge AI, DAYJOB: Tax benchmark and leaderboard page (text supplied to us; it does not describe the grading method). https://surgehq.ai/benchmarks/dayjob-tax
- Surge AI, DAYJOB evaluation repository, README (text supplied to us; it names the judge and the five-attempt convention). https://github.com/surge-ai/dayjob
- Surge AI, DAYJOB: Tax dataset card on Hugging Face (text supplied to us; it states that two sample tasks are public). https://huggingface.co/datasets/surgeai/DAYJOB-tax
- braindetox, “DAYJOB: Tax Benchmark Tops Out at 10.8%: How to Read It” (secondary; its description of the grader and the five attempts matches the repository README). https://braindetox.kr/en/posts/dayjob_tax_ai_agent_tax_benchmark.html
- Groundy, “Should AI agents prepare tax returns? Reading DAYJOB’s 10.8% result” (secondary). https://groundy.com/articles/should-ai-agents-prepare-tax-returns-reading-dayjobs-10-8-result/
- Japanese-language coverage of the benchmark (secondary; site not identified in the text supplied to us).



Ⅽomment