voxpop · fidelity leaderboard
Which models can actually represent a population?
We measure real, named models against a Polish reference context - the same instruments, the same frozen rules, the same abstention discipline. This is the qualitative preview: verdicts, not numbers. It reads as one message the whole site carries - a pricier model is not a more faithful one.
Qualitative preview. The full numbers and the methodology stay with us - this page shows the verdicts only.
Snapshot: 8 Jul 2026 · Polish reference context · specific model variants
The whole field, one map
Which model handles what
The more green, the more faithfully a model reproduces a population; models are ordered best-first. Hover any cell to see what its verdict means for that model. No model dominates every axis - a verdict here is a set of axes, not a single number, and that is precisely the point of our method.
- Aya 8B
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- DeepSeek V4 Flash
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- Llama 3.1 8B
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- Mistral NeMo 12B
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- Gemini Flash-Lite
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- GPT-4o-mini
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- Mistral 7B
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- Bielik 11B
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- Claude Sonnet
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- Gemma 2 9B
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- Phi-4 14B
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- Qwen 2.5 14B
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
- Qwen 2.5 7B
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
Calibrated-fidelity verdicts are withdrawn. We ran a pre-registered control on our own calibration metric - the pass rule frozen before the data - and it did not hold: the calibrated score could be reached by random noise. So we pulled every calibrated reading while we rebuild the metric. The raw-model verdicts, the contamination pre-flight and the structure axis all stand. Catching our own headline metric before we leaned on it is the trust layer doing exactly its job.
The toggle only ever moved reality, diversity and structure - pre-flight and coherence never depended on calibration. A dot marks a cell that has no calibrated reading of its own.
A diamond marks the best raw reading on that axis in the latest local batch (for the pre-flight: the cleanest). Structure carries no marker while that check is under an internal audit. Best on an axis is not the same as passing it.
How to read the verdicts
- Green - in the healthy human range.
- Amber - marginal, partial, or on hold.
- Red - outside the human range.
- Grey - not measured, uninterpretable, indeterminate, or withdrawn.
Each cell is a categorical verdict, not a number. Green passes, amber is marginal or on hold, red fails. Where a model may have seen the reference data, the reality axis reads uninterpretable - we will not rank a location we cannot trust. Where a measurement cannot be adjudicated, we abstain rather than guess. A star marks a single run whose replication is still pending.
★ Star marks a single run - replication pending.
How we measure
Six checks behind every row.
- 01
Contamination pre-flight
Before anything else, we check whether a model may have seen the reference data in training. If it might have, we say so plainly - and treat the affected measurements as uninterpretable, rather than pretend they are clean.
- 02
Reality
Do a segment's answer distributions line up with real survey data? It is always scored against a held-back slice the model never saw during tuning - never the slice used to calibrate it.
- 03
Diversity
Does the simulated population carry a real spread of opinion, or does it collapse until every respondent sounds like one person? A confident monoculture is a common - and invisible - failure.
- 04
Attitude structure
Do the relationships between attitudes hang together the way they do in people - not each attitude on its own, but how they move together?
- 05
Calibration
How much our layer changed the raw model, shown per axis. We put this metric through our own pre-registered control and it did not hold - the calibrated-fidelity reading could be matched by random noise - so those readings are withdrawn while we redesign it. The raw-model, contamination and structure checks are unaffected.
- 06
Persona coherence
A separate certificate for the voice layer: does the model hold one consistent character across rephrasings and orderings? It ships with a positive control - a scrambled persona that must fail, or the check itself is not trusted.
Filter
Pre-flight
- DeepSeek V4 Flash★APIAPI
- Pre-flight
- Reality
- Diversity
- Structure
The only frontier with a clean, interpretable reality axis and healthy diversity - and it is far from the most expensive one.
- Llama 3.1 8B★~8BLocal
- Pre-flight
- Reality
- Diversity
- Structure
Rare healthy diversity straight out of the box - one of the few that does not collapse raw. Its calibrated reading is withdrawn while we redesign that metric. A single run, replication still pending.
- Aya 8B★~8BLocal
- Pre-flight
- Reality
- Diversity
- Structure
Wins the raw distribution accuracy of the latest local batch and is the cleanest on the pre-flight - yet it also shows the weakest pattern of links between answers in that batch. Flat and easy to shift, that profile is exactly the one our own control test taught us to treat with care: raw accuracy read on its own can mislead. Clean is not the same as faithful.
- Mistral NeMo 12B★~12BLocal
- Pre-flight
- Reality
- Diversity
- Structure
Clean, with a healthy spread straight out of the box - but raw its reality stays weak, and its calibrated reading is withdrawn while we redesign that metric. A bigger model is not a more faithful one. A single run, replication still pending.
- Gemma 2 9B★~9BLocal
- Pre-flight
- Reality
- Diversity
- Structure
Raw, its diversity is mid-collapse and its contamination pre-flight comes back inconclusive - and on raw distribution accuracy it lands below the level of pure random noise, the only model in the roster that does. Its calibrated reading is withdrawn while we redesign that metric.
- GPT-4o-mini★APIAPI
- Pre-flight
- Reality
- Diversity
- Structure
The one model in the bank that recognizes the survey instrument itself - a contamination flag we carry on every reading of this row - even though its answer distributions stay clean. Its raw reality holds up on data it could not have seen, yet collapsed diversity still puts a faithful result out of reach.
- Phi-4 14B★~14BLocal
- Pre-flight
- Reality
- Diversity
- Structure
We had logged this as our strongest calibrated result ever - a generic, non-Polish model whose segments seemed to line up with real data better than any model before it. Then our own pre-registered control showed that same calibrated reading could be produced by random noise, so we withdrew it. Raw, it reproduces nothing, and its contamination pre-flight is inconclusive. We treat catching our own record before we shipped it as the audit layer working, not a setback.
- Qwen 2.5 14B★~14BLocal
- Pre-flight
- Reality
- Diversity
- Structure
From the same family as the smaller Qwen and, like it, may partly recognize the reference - so the contamination pre-flight comes back inconclusive. Raw, its reality stays weak, and its calibrated reading is withdrawn while we redesign that metric. A single run, replication still pending.
- Qwen 2.5 7B★~7BLocal
- Pre-flight
- Reality
- Diversity
- Structure
We had read its repair as a property of the method rather than one lucky model. That calibrated reading is now withdrawn pending a metric redesign; what stands is that it fails raw and its contamination pre-flight is inconclusive.
- Bielik 11B~11BLocal
- Pre-flight
- Reality
- Diversity
- Structure
Once our strongest calibrated result - after our layer, most segments had looked indistinguishable from real data. That calibrated reading is now withdrawn: our own control showed the score could be reached by noise. Raw, it stays far from reality, and its contamination pre-flight is inconclusive.
- Mistral 7B★~7BLocal
- Pre-flight
- Reality
- Diversity
- Structure
A generic, non-Polish model. We had read it as matching our best after our layer; that calibrated reading is now withdrawn, since our own control showed the score could be reached by noise. Raw, it reproduces little. A single run, replication still pending.
- Claude Sonnet★APIFlagship
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
The deepest diversity collapse we measured; even the coherence check comes back uninterpretable, because near-identical answering fools the test.
- Gemini Flash-Lite★APIAPI
- Pre-flight
- Reality
- Diversity
- Structure
- Coherence
Collapsed as a population - but the one model certified to hold a single, consistent character. Our cheapest voice beats the flagship.
Star marks a single run - replication pending.
Calibration
raw → calibrated
How much our layer changed the raw model, shown per axis. We put this metric through our own pre-registered control and it did not hold - the calibrated-fidelity reading could be matched by random noise - so those readings are withdrawn while we redesign it. The raw-model, contamination and structure checks are unaffected.
Calibrated-fidelity verdicts are withdrawn. We ran a pre-registered control on our own calibration metric - the pass rule frozen before the data - and it did not hold: the calibrated score could be reached by random noise. So we pulled every calibrated reading while we rebuild the metric. The raw-model verdicts, the contamination pre-flight and the structure axis all stand. Catching our own headline metric before we leaned on it is the trust layer doing exactly its job.
What's not here - and why
- No numbers. Metric values, thresholds, counts and costs stay with us - this page is a qualitative preview.
- No instrument names. The specific tests and estimators are our craft; naming them here would give the method away without helping you read the result.
- Honest 'not measured'. Where we have not run a measurement, we say so - we never fill a gap with a guess.
- Context matters. Every verdict is about a specific model variant answering in a Polish reference context - not a general statement of model quality. A weak population simulator can be an excellent assistant.
- A snapshot in time. Models and their behavior drift; this reflects what we measured on the snapshot date, not a permanent ranking.
- Star marks a single run - replication pending.
Have a model to submit? Building the standard with us?
We add models by request and work with research and insight teams on an open fidelity standard for synthetic populations answering in Polish.