Benchmarking Overton Pluralism in LLMs
← Paper, dataset,
and code!
- LLMs are central to how information is produced and consumed, influencing human beliefs and values at scale.
- Traditional alignment methods that aggregate over diverse preferences risk:
- erasing viewpoints → algorithmic monoculture
- collapsing genuine disagreement → value monism
- response homogeneity → epistemic breakdown
Pluralistic alignment offers an alternative: models should be able to represent diverse perspectives + values faithfully.
Given a subjective query (no single correct answer), an Overton pluralistic model should represent the “Overton window”1—the full spectrum of reasonable perspectives—in its response.
We introduce OvertonBench, a novel framework for measuring Overton pluralism in LLMs grounded in human viewpoints.
1 Generalized from political science: “the spectrum of ideas on public policy and social issues considered acceptable or viable by the general public at a given time” (OED, 2023).
For a subjective query $x$, the Overton window is the set of reasonable answers $W(x) = \{y_i\}_{1}^{k}$. If humans holding viewpoint $y \in W(x)$ feel well-represented by a model response, $y \in \mathcal{M}(x)$ is covered.
The OvertonScore for a model $\mathcal{M}$ over a set of queries $X = \{x_1, \ldots, x_n\}$ is the average Coverage:
By construction, the max OvertonScore is 1.0.
Example question: “Should the government enforce strict regulations on carbon emissions, or allow companies to emit carbon to grow the economy?”
“Strict carbon rules protect health and slow climate change... While higher initial costs exist, long-term gains outweigh them...”
“The government should take a balanced approach to encourage innovation and economic growth while reducing emissions...”
“Some argue for strict rules... Others disapprove of gov’t regulation... While some support a balance...”
Table 1: OvertonBench results (all qs, unweighted).
95% bootstrap CIs (1k samples) • τ = 4.0
💡 Key Takeaway: All model scores (0.35–0.41) remain far below the maximum of 1.0 → LLMs capture only a fraction of the Overton window.
Table 2: OvertonBench results split by question source (unweighted).
💡 Key Takeaway: No model is uniformly most pluralistic. o4-mini performs best on political topics but worse on diverse topics; DeepSeek V3 shows the reverse.
Repeated human studies are costly → we use Gemini 2.5 Pro to predict human ratings, enabling automated scoring of unseen models.
💡 Key Takeaway: Our LLM judge reproduces human scores with high rank correlation (ρ = 0.88), providing a scalable evaluation proxy.
💡 Key Takeaway: Political neutrality2 and pluralism are negatively correlated and distinct concepts.
2 The model slant metric (Westwood, 2025) measures bipartisan political slant, where scores closer to zero indicate greater perceived neutrality by humans.