[Submitted on 18 Nov 2025 (v1), last revised 30 Jul 2026 (this version, v2)]

View PDF HTML (experimental)

Abstract:Supervised classifiers output a distribution over classes but are typically trained against a single label obtained by collapsing multiple annotators into a majority vote. On tasks where annotator disagreement reflects genuine ambiguity -- natural language inference, politeness, visually ambiguous categorization -- this collapse discards information and forces models to express uniform confidence on inputs where humans systematically disagree. We compare soft-label training, which uses the full annotation distribution as the target, against hard-label training across three datasets spanning vision and NLP (ChaosNLI, POPQUORN, CIFAR-10H). Soft-label training matches or exceeds hard-label accuracy on every dataset, reduces KL divergence to the annotator distribution by 32% on average (p < 10^-4), and produces predictions whose per-sample entropy correlates 61% more strongly with annotator entropy -- models trained on distributions are uncertain precisely where humans are. We argue these benefits follow from a basic observation: when annotators legitimately disagree, the annotation distribution is the correct learning target, not a noisy estimate of it.

Submission history

From: Agamdeep Singh [view email]
[v1] Tue, 18 Nov 2025 04:02:29 UTC (998 KB)
[v2] Thu, 30 Jul 2026 07:08:07 UTC (1,021 KB)