Can AI agents do professional work? The rules they miss are in your product.
Not yet, on the strictest public tests. On GDPx, published by SemiAnalysis on 2 November 2026, the best frontier model scored 12% across eleven high-value domains, and in finance under pass or fail grading nearly every model scored zero. The mistakes the agents made break domain rules, and a vertical SaaS product already enforces most of them.
- 12%
Best frontier model's score on GDPx, across eleven high-value domains.
Source: SemiAnalysis, 2 Nov 2026 - 0 of 20
Usable redlines from 20 runs of four models on a Series B legal task.
Source: SemiAnalysis, 2 Nov 2026 - 10.8%
Best pass rate on Surge AI's tax benchmark, at a 95% rubric threshold.
Source: Surge AI, 27 Oct 2026
What did GDPx test, and how did the best agents score?
The GDPx benchmark, published by SemiAnalysis on 2 November 2026, gives frontier agents long pieces of real work in eleven domains, and the best model scored 12%. Finance and legal sit beside chip design, cybersecurity, biology, quant trading, recursive self-improvement, hardware design, materials science, alignment and software engineering. Each agent gets up to eight hours per task and runs in its own native harness where one exists: Claude Code for Opus 5.5, Codex for GPT-6 Astra. The headline score is pass@1 over three attempts, with every domain weighted equally.
In finance, graded pass or fail, nearly every model scored zero. Legal also came out at 0% under binary grading, so the authors report partial scores for it instead. Two of their own notes belong next to those numbers: they say safety classifiers hold down the scores of Opus 5.5 and GPT-6 Astra, and the finance domain has 24 tasks.
A benchmark published a week earlier points the same way. Surge AI's DAYJOB: Tax sets 50 tax assignments built by tax practitioners formerly at PwC, Deloitte, EY and other firms, and counts a pass only when a response meets 95% of the rubric. The best pass rate Surge reported was 10.8%, from Claude Sonnet 5.5. Surge sells evaluation and post-training data, and the full benchmark is available on request.
How close did the agents get on finance and legal work?
Close enough to look convincing in a demo, and short of anything a client could use. GDPx's Bellwood task in finance is a quality of earnings review, the diligence an analyst runs before a deal. Across the 25 attempts SemiAnalysis reported, no agent solved it. The best models met about half of its 11 checks on average, and the best single run passed 8 of 11.
The legal task asks for a redline in a Series B financing. In the GDPx results, four models ran it 20 times between them, and by the authors' judgement none of the 20 redlines was usable. Opus 5.5 averaged 44% of the rubric, GPT-6 Astra 25%, and the best single run reached 57%.
Each of these is one task with 20 to 25 attempts, so it shows how the work goes wrong. A rate for a whole profession would need far more tasks than that.
On the finance diligence task, the best agents met about half the checks
Checks met on the Bellwood quality of earnings task, out of 11, mean across attempts
Which mistakes did the agents make?
The agents broke domain rules, according to the failure notes SemiAnalysis published with GDPx on 2 November 2026: they trusted the most plainly stated figure over the authoritative document, misread a waived adjustment, let a client's email overrule a signed term sheet, and left a subsidiary a way round the charter. Those notes are the most useful part of GDPx for a product leader. Take each mistake and ask which rule it breaks.
The plainest figure won. In the finance notes, models often weighed a source by how explicitly it stated a figure instead of by which document was authoritative. The rule it breaks is the source of truth for each field: a number from the wrong document is wrong however clearly it is printed.
A waived adjustment read as a settlement. In 10 of 15 runs by Opus 5.5, GPT-6 Astra and GPT-6.1 Sol, the agent read a waived $418 premium audit adjustment as settling the prior workers' compensation policy, which understated LTM EBITDA by $4,728. The rule it breaks is how each kind of adjustment is applied, which an accounting or insurance product encodes once and applies to every record.
An email overruled a signed term sheet. In 14 of 20 legal runs, the agent applied terms from the client's email without flagging them, treating the email as if it overrode the signed term sheet. The rule it breaks is precedence: which document outranks which, and when a conflict goes to a person.
A subsidiary left a way round the charter. In 20 of 20 runs, the redline left a subsidiary able to bypass the protective provisions in the company's charter. The rule it breaks is a constraint that has to hold across every entity the company controls.
The authors sum up the finance results plainly: models can "fail basic mechanics in ways a junior analyst would readily catch."
Four mistakes, four rules a vertical product already enforces
What does this mean for an AI-first challenger in my category?
In most vertical categories the AI-first challenger is a frontier model with a prompt and an interface around it, and GDPx tested those models. In its results, OpenAI and Anthropic models generally led, and the authors place SpaceXAI, Meta and open-weight models meaningfully behind.
Our reading: switching to another model does not close the gap on this kind of work, and a challenger built on these models inherits these results until it acquires the rules. Your product holds them already, in its field definitions, validations and the order in which documents are reconciled. Our earlier post, Was your eval set written by the model you want to replace?, covers what an entrant cannot hire; GDPx adds a public score to that argument.
Which rules belong to the category, and which facts to one customer?
Category rules are the same for every account. What a waived adjustment does and which signed document outranks a later email are written into your product's code and configuration. Customer facts and exceptions live in one account's own records: a move-out date, a marital status, a term agreed once with one client.
DAYJOB: Tax has an example of each model missing one. In one assignment, a client's condo sale would lose its residence exemption around August 2027: the rule is the category's, and the date it applies to was in the client's file. Across five attempts GPT-6 Astra never found the deadline, and Opus 5.5 found it every time. In another, a couple were separating: Opus 5.5 assumed they were unmarried in all five attempts, while GPT-6 Astra flagged their status for verification in four of five. Surge's comment on the second case: "The evidence that should have prompted that investigation was already in the files."
Each model caught one customer fact and missed the other, and five attempts is a small sample. For a vertical vendor the interesting part is where the evidence sat. Customer facts and exceptions surface in your product as overrides, support escalations and edits to an agent's output, inside accounts you already run.
What should a product leader do before the next roadmap review?
Work from one workflow your customers already ask an agent to do, and build two lists for it.
- The category list. Write down the rules your product enforces on that workflow today: the source of truth for each field, which document outranks which, and the mechanics a junior is corrected on.
- The customer list. Pull last quarter's overrides and escalations for three customers, and mark each one as a category rule or a customer fact or exception.
- The visibility check. For every item on both lists, find out whether your current AI feature can see it when it runs.
Before your next roadmap review
Bring both lists, a platform lead and a domain expert to a 60 minute session.
When your team corrects an AI agent's output, what is usually missing?
Anonymous. One vote per browser.
Where does SynOS fit?
SynOS runs inside your product, white-labelled, and turns the domain data and rules you already hold into context your agents work from. It builds a Brain for each customer from that customer's own data and corrections, so the facts in one account's files are part of the context of the agent working on that account. Every run is traced and scored, so you can see where an agent ignored a rule, and writes are recorded rather than executed.
Questions
Can AI agents do professional work yet?
Rarely, on the strictest public tests as of November 2026. On SemiAnalysis's GDPx the best frontier model scored 12% across eleven high-value domains, and in finance under pass or fail grading nearly every model scored zero. On Surge AI's DAYJOB: Tax the best pass rate was 10.8% at a 95% rubric threshold.
Why do AI agents fail finance and legal tasks?
GDPx's failure notes point to domain rules. Agents weighed a source by how explicitly it stated a figure instead of by which document was authoritative, misread a waived $418 adjustment in 10 of 15 runs, let a client's email overrule a signed term sheet in 14 of 20 runs, and missed mechanics a junior analyst would catch.
How far can I trust the GDPx and DAYJOB results?
Read them as strong signals with small samples. GDPx finance has 24 tasks, the task examples here rest on 20 to 25 runs, the article is partly paywalled, and the authors say safety classifiers hold down two models' scores. Surge sells evaluation data, the DAYJOB examples rest on five attempts, and the full set is available on request.
Where should a vertical SaaS company start with AI agents for domain work?
Start where customers already want an agent: one workflow your product runs every day. The rules your product enforces there, and the facts and exceptions behind each customer's overrides, are the context that agent needs. Find out how much of it your current AI feature reads before it acts, and close that gap first.
Sources
- GDPx: Measuring Frontier Model Capabilities Across the Most Valuable Domains, SemiAnalysis (Max Kan, Andrew Megalaa, Dylan Patel) (2 November 2026, partly paywalled; 12% headline, method, finance and legal results, Bellwood and Series B failure notes, model spread)
- DAYJOB: Tax, Surge AI (27 October 2026; 50 environments, pass rates at a 95% rubric threshold, condo and separating couple examples. Surge sells evaluation and post-training data; the full benchmark is available on request)