Was your eval set written by the model you want to replace?
When Property Finder checked the evaluation set it used to choose a vision model, 96.8% of the labels had come straight from the model already in production (Property Finder and AWS, AWS Machine Learning Blog, 27 October 2026), so every alternative was graded on how closely it copied the incumbent. One expert relabelled 356 images blind, and on that key an open-weight model the old set had held back scored 0.80 micro F1 against the incumbent's 0.56, figures the authors call optimistic. Before you trust last year's bake-off, check who wrote the answers.
- 96.8%
Of Property Finder's eval labels came directly from its production model's predictions.
Source: Property Finder and AWS, Oct 2026 - 356
Images one expert labelled blind to rebuild the answer key.
Source: Property Finder and AWS, Oct 2026 - 0.56 to 0.80
Micro F1, incumbent against Llama 4 Maverick, on the rebuilt key. The authors call these scores optimistic.
Source: Property Finder and AWS, Oct 2026
Why does every cheaper model lose your eval?
Because the eval may have been written by the model it is trying to beat. When Property Finder audited the evaluation set behind its choice of vision model, 96.8% of the labels had come directly from the predictions of the model already in production, according to the write-up Property Finder and AWS published on the AWS Machine Learning Blog on 27 October 2026. The other 3.2% were human labels, and even those included out-of-spec categories such as “Office”.
If you run AI inside a vertical product, you have probably run this test. A cheaper model, or an open-weight one you could host yourself, went up against the API behind a live feature. It lost, the feature stayed on the API, and the result went into a slide nobody has opened since.
Property Finder, a property app in the Middle East and North Africa, ran that test on listing photos. It processes tens of thousands of property images a day, about 15 per listing, and a vision language model classifies each one across 11 quality attributes, among them rotation, people in the frame and watermarks.
The authors state the consequence plainly: any alternative model was “structurally disadvantaged before a single image was even processed”. On a key like that, a challenger can only score well by making the incumbent's mistakes. Where it read a photo correctly and the incumbent had not, the key marked it wrong.
How does a production model end up writing its own eval labels?
Nobody sets out to build a circular evaluation. Production outputs are the cheapest labels a team has: they already exist, in volume and in the right format. Someone reviews a sample, fixes what they catch, and a year later the set is called ground truth. The errors nobody caught have become the standard every new model is measured against.
Property Finder found two kinds. Watermark text with character errors had been marked correct, and photos showing more than one room had been forced into a single label. A challenger that read the watermark correctly, or saw both rooms, lost points for it. The figure below draws that loop beside the key the team built to replace it.
Two ways to write an answer key
How do you rebuild an eval set with expert ground truth?
The fix started with one person. A strategy and analytics expert at Property Finder annotated every image independently, with no reference to earlier model outputs. The images were 356 photos from UAE production listings that had never been labelled, the Property Finder and AWS write-up of 27 October 2026 reports. The first 256 became the development set for prompt work, and 100 more were added after the development runs.
Labelling showed the old schema was wrong too. Doorways and overlapping features meant one label per photo misdescribed many images, so the team moved to multi-label classification. The scoring changed with it: micro F1 as the primary metric, macro F1 beside it, precision and recall for each label, intersection over union, and exact match on the nine attributes that still take a single label.
The authors size the set with care. By their account, 356 images is roughly enough to detect moderate differences between models, and smaller differences on rare attributes may not show. That is a few hundred images from one expert, far short of a labelling programme.
Does the open-weight model beat the incumbent on the expert's key?
Yes, and the authors attach two discounts to the result. On the expert's key, model A, Property Finder's production baseline, scored 0.55 micro F1 on the development set and 0.56 on the full 356-image evaluation, according to Property Finder and AWS on 27 October 2026. The write-up does not name it. Meta's Llama 4 Maverick 17B Instruct scored 0.75 on the development set with the shared v21 prompt, 0.81 once its prompt reached v27, and 0.80 on the full evaluation with v27_PF, which is v27 plus Property Finder's own rules.
On the expert's key, the open-weight model scored higher at every stage
Micro F1, 0 to 1. Model A had one round of prompt tuning; Maverick had several plus Property Finder's rules. The eval set includes the dev images, so the authors call these scores optimistic.
Dev set (256 images)
Eval set (356 images)
Two discounts belong next to those numbers, and the authors apply both. The first is that tuning was unequal. Model A got one round of prompt work, v22. Maverick got v24 through v27, Property Finder's rules and a pass of the MIPROv2 prompt optimiser. The closest the record comes to a fair comparison is Maverick on the shared v21 prompt, before any tuning of its own: 0.75 on the development set, against 0.55 for model A after its one tuned round.
The second is that the scores are optimistic. The 356-image evaluation contains the 256 development images both prompts were tuned on, and the authors say so. Read 0.56 and 0.80 as a ranking on this key, with the true size of the gap still to be measured on images no prompt has seen.
Within those limits the ranking holds across the board. Maverick matched or exceeded model A on all 11 attributes and tied on photo classification.
The write-up also reports cost and speed. It estimates the new setup at $2.35 per 1,000 images, an upper estimate at December 2025 pricing in US East (Ohio), assuming 9,000 input and 200 output tokens per image. That is 86% below Property Finder's estimate for its previous setup on the v21 prompt; the accuracy comparison used v22, whose cost may differ. Property Finder also reported 40 to 60% lower inference latency, from its own monitoring.
If the answer key was copied from the incumbent, the bake-off measured agreement with the incumbent.
Where should expert review time go after a model switch?
Maverick is uneven by attribute, so the expert's review belongs on its weakest ones. On the nine single-label attributes, Property Finder and AWS report exact match running from 100% on rotation down to 65% on room size and 55% on extracting watermark text.
Room size and watermark text are where review time should go
Exact match by attribute, percent, Llama 4 Maverick. The model reports no calibrated confidence, so the authors suggest disagreement between models as a review signal, checked on the dev set first.
The model gives no calibrated confidence to route hard images to a person; the authors suggest disagreement between models as a review signal, tested on the development set first. Generic prompts failed, and regional rules had to be written down: decorative Arabic text laid over a listing photo is not a watermark. A two-bucket split gave no gain.
The expert's work moves from labelling every image to reviewing room size and watermark text, and to writing the rules the prompts need.
What can an AI-native entrant in your category not hire?
An AI-native competitor in your category can call the same APIs you call and fine-tune the same open-weight models. What the entrant lacks is the people who know the Arabic overlay rule, or the equivalent rule in your vertical. For how far domain training moves those models, see our comparison of 36 domain-trained open-weight models.
If your eval set is copied from a vendor model, that knowledge never reaches the model decision, and you are choosing models on the same footing as the entrant.
In claims software, the expert is the reviewer who knows which denial code a payer applies to a resubmitted claim. In a tax product, it is the preparer who knows when a home office expense belongs in a different category. Ask those people to write the answers before any model sees the items.
Before you rerun the bake-off
Four rules for the Head of AI or platform lead who will own the rerun.
- Labels written blind by a domain expert, never seeded from any contender's output.
- A held-out slice frozen before any prompt work, and used once. Property Finder's eval overlapped its dev set; yours should not.
- The same tuning budget for every contender: the same number of prompt rounds, the same optimiser, the same rules added.
- Multi-label scoring wherever classes overlap, with results for each attribute instead of one headline number.
How do you audit an eval set for incumbent bias?
Run these six steps on your own eval set this month, in order. The first two use only the labels you already have.
- Pick the eval set behind your highest-volume or highest-cost model feature.
- For each label, record its origin: production model output, model output reviewed by a person, or written by a person from scratch.
- If most labels came from a model, have one domain expert label a few hundred items blind.
- Split before tuning: a dev set for prompt work and a test set nobody touches.
- Rerun the incumbent and one challenger under the four rules above.
- Read the errors for each attribute with the expert, and write down the domain rules they apply.
Where did most labels in the eval set behind your main model feature come from?
Anonymous. One vote per browser.
Where does SynOS fit?
Agents built on SynOS rehearse against your real systems, with intended writes recorded rather than executed, and they are scored against test cases your own people write. The corrections your experts make and the traces of each run are kept and curated into training data for a fine-tuned open-weight model you own, on your own infrastructure. That keeps the answer key and the training data in your experts' hands. It does not make an evaluation fair by itself: the four rules above still apply to any model you test. Our page for vertical SaaS teams shows how that runs inside a product. If you have a model decision to revisit, bring that workflow.
Questions about eval sets and model bake-offs
Why does a cheaper model lose an eval labelled by the production model?
Because the eval then measures agreement with the incumbent. In the Property Finder and AWS write-up of 27 October 2026, 96.8% of evaluation labels came from the production model's own predictions, so a challenger that read an image correctly where the incumbent had not was marked wrong.
How many expert labels does a rebuilt eval set need?
Property Finder used 356 images and describes that as roughly enough to detect moderate differences between models; small differences on rare attributes may not show. Start with a few hundred items, then add items for the attributes where models disagree.
Was the comparison fair if the challenger got more prompt tuning?
Not fully, and the authors say the scores are optimistic. Model A had one tuned prompt round; Llama 4 Maverick had several, plus Property Finder's rules and a MIPROv2 pass. Maverick reached 0.75 micro F1 on the dev set with the shared v21 prompt, before any tuning of its own; model A reached 0.55 after its one tuned round, v22. That suggests unequal tuning does not explain the whole gap. In your own rerun, give every model the same tuning budget.
Can a frontier model still generate our training data?
Yes, as long as it does not also write the answer key. In a Red Hat Developer write-up of 27 October 2026, Atos used GPT-4.1 to generate synthetic training data for small models (Phi-4 Mini, Qwen 2.5 1.5B and 7B) and scored every model against a human-validated gold standard on a 34-point rubric. Fine-tuned Qwen 2.5 7B scored about 30.8 against 30.5 for GPT-4.1 with a detailed prompt. Red Hat reports no variance or sample sizes, so treat the gap as a tie.
Can model disagreement replace expert review?
Not on the evidence in this record. Property Finder's models gave no calibrated confidence, and the authors suggest disagreement between models only as a review signal to validate on the dev set before relying on it.
Sources
- Property Finder and AWS on evaluation bias in vision model selection, AWS Machine Learning Blog (27 October 2026; Khaled Thabit, Meriem Boughrara, Mohamed Desouky, Mina Metias, Mohamed Noordeen Alaudeen, Vadim Selyutin; image volumes and the 11 attributes, the 96.8% and 3.2% label origins, the hidden errors, the quoted line, the expert rebuild of 356 images and the dev split, multi-label scoring and metrics, prompt versions and tuning effort, micro F1 scores, the optimistic-scores caveat, exact match by attribute, cost, latency and the review signal)
- Smaller model, sharper insights: how Atos approached structured outputs, Red Hat Developer (27 October 2026; Aditi Saluja, Mustafa Eyceoz, Marius Kiskemper; GPT-4.1 as teacher for synthetic training data, the student models, the 34-point rubric against a human-validated gold standard, the rubric scores and the missing variance and sample sizes)