Surge AI's DAYJOB: Tax benchmark: best agent passes 10.8% of tasks

Surge AI has added DAYJOB: Tax to its DAYJOB benchmark family, which earlier covered Healthcare and Finance. The new release tests AI agents on professional tax work: preparation, compliance and advisory. It contains 50 environments across eight areas of tax practice: individual and family tax, small-business tax, partnerships, property and investments, estates, gifts and trusts, international tax, corporate transactions, and tax review and disputes. Agents get a client request plus supporting records and must produce a tax return, workpapers, a technical review or client advice. The point, the authors say, is to test professional judgment alongside tax knowledge, calculations, retrieval and instruction following: deciding what deserves investigation, resolving conflicting evidence, and recognizing when information is insufficient.

The headline result is low. Claude Sonnet 5.5 leads the leaderboard with a 10.8% pass rate, followed by Claude Fable 5.1 at 9.6% and Claude Opus 5.5 at 9.2%. Surge reports pass@1 at a 95% rubric threshold: an attempt passes only if it satisfies at least 95% of the task's criteria, which cover calculations, supporting evidence, consequential issues and the completed deliverable. The post also compares performance with average cost per trial and says Sonnet 5.5 and Opus 5.5 sit at the high-performance end of the measured Pareto frontier. It gives no per-model cost figures.

The post opens with a worked example. Anna plans to sell her condo and moved out in August 2024. She wants to know whether closing in October 2026 or January 2027 is better for taxes. An agent correctly concluded that either date satisfies the residence requirement for a tax break worth up to $250,000 in excluded gains. It missed a deadline: homeowners must normally have lived in the property for at least two of the five years before the sale, so Anna would stop meeting the requirement around the end of August 2027, and a sale delayed beyond that could cost her eligibility. Across five attempts, an agent called Astra never identified the deadline; Opus identified it in all five.

A second example runs the other way. A separating couple asked an agent how to divide their Roth IRA and 401(k). Opus 5.5 told them they were not legally married and recommended keeping the accounts separate, with a cash payment to even out balances; one response opened, "You aren't legally married, so neither account can be split without triggering taxes and penalties." But the couple's records raised doubts. They now lived in Florida, where new common-law marriages are generally not recognized, but they had earlier lived together in Texas, which recognizes informal marriages under certain conditions, and had held a ceremony in Colorado. A valid marriage established elsewhere may remain valid in Florida, and that could change the tax consequences. Across all five attempts, Opus built its advice on the assumption that the couple was unmarried, even when it acknowledged uncertainty. Astra recognized the need to verify marital status in four of five attempts, with a response such as "We first need to confirm your legal marital status."

The workspaces are large and messy. The benchmark holds 1,789 files, a median of 30 per assignment and up to 123: 1,487 PDFs, 117 Excel workbooks, 38 Word documents, plus emails, images, CSVs, JSON records and text files. They include filed and draft returns, K-1s, W-2s, IRS notices, general ledgers, payroll records, contracts and correspondence. One assignment includes a photograph of a handwritten receipt for merchandise bought for resale, which the agent must reconcile with spending records to work out the effect on inventory and cost of goods sold. Another involves a silo-services company whose client assumes it operates only in Alabama, while its dispatch log shows crews working in five states. A third involves a cross-border household whose Polish employment and property records must be reconciled with cryptocurrency exports and wallet histories.

Tax experts build the tasks from real-world professional workflows, writing the supporting records, expected outcomes and grading criteria. Each task goes through a three-layer expert review for correctness, completeness and solvability. The named backgrounds include a tax attorney with over 10 years at PwC, Deloitte and Exxon, a Baker Tilly Tax Director with experience at PwC, Moss Adams and Windes, and a tax professional with experience at Deloitte and EY.

Surge contrasts DAYJOB with other tax benchmarks. TaxBench has 114 tax questions, each with a defined problem and rubric. TaxCalcBench checks return accuracy against expected filing values. TaxLab tests whether agents can question a simulated taxpayer, gather missing information and complete a federal individual return. The authors say these start from a defined problem, while DAYJOB asks whether the agent finds the issue the client never raised. Their conclusion: even when agents can perform individual steps, exercising professional judgment and reliably carrying out real-world tax workflows remain out of reach. "Agents haven’t yet passed the audit." The full leaderboard is on the DAYJOB: Tax benchmark page, and access to the full benchmark is by request at benchmarks@surgehq.ai.

Key facts

DAYJOB: Tax has 50 environments across eight areas of tax practice, with 1,789 workspace files (median 30 per assignment, up to 123), mostly PDFs.
Claude Sonnet 5.5 leads at a 10.8% pass rate, ahead of Claude Fable 5.1 (9.6%) and Claude Opus 5.5 (9.2%); the metric is pass@1 at a 95% rubric threshold.
In the Anna condo-sale example, Opus spotted the end-of-August-2027 residence deadline in all five attempts; Astra missed it in all five.
In the separating-couple example, Opus 5.5 assumed the couple was unmarried in all five attempts, while Astra flagged the need to verify marital status in four of five.
Tax experts write the tasks and grading criteria, and each task gets a three-layer review for correctness, completeness and solvability.

Why it matters

Surge argues that most tax benchmarks start from a defined problem: a question to answer, a return to calculate, or missing information to collect. Real tax work often starts with incomplete instructions and conflicting records, where the consequential issue never appears in the client's request. DAYJOB: Tax is built to test that end-to-end responsibility, and the results are stark: the best model passes 10.8% of the time under a strict 95% rubric threshold.

Who it affects

Teams building or buying AI agents for tax preparation, compliance and advisory work are the direct audience, as are the tax professionals who would rely on them. The task areas span individual, small-business, partnership, estate, international and corporate work, so the findings touch most of the profession. Model developers get a hard target, since the three leaders sit within about 1.6 percentage points of each other.

How to use it

The full leaderboard with every evaluated model is on the DAYJOB: Tax benchmark page. Access to the full benchmark is by request through benchmarks@surgehq.ai. The post's worked examples are a practical checklist for testing any tax agent: does it flag a deadline the client did not ask about, and does it verify a status such as marital status before advising on dividing retirement accounts?

How solid is it

This is a first-party benchmark from the vendor that built it, and Surge reports the results itself. The post describes expert-created tasks with a three-layer review for correctness, completeness and solvability. Only the top three scores are given in the text, and the post does not report confidence intervals, the number of models or attempts per task overall, a human-expert baseline, or who does the grading. The gaps between 10.8%, 9.6% and 9.2% should not be read as clear rankings. The post does not say who Astra is, so it should not be matched to any leaderboard model.

Risks and caveats

The two examples are benchmark assignments, not real client cases. Five attempts per example is a small sample, and the Anna and couple cases point in opposite directions for Opus, which caught one issue and missed the other. The pass@1 metric at a 95% threshold is very strict, so a low pass rate does not show how close attempts came. The full benchmark is not described as publicly downloadable, which limits independent checking. No per-model cost figures are given behind the Pareto frontier claim.

“Agents haven’t yet passed the audit.”

— Surge AI, DAYJOB: Tax post