Benchmark Contamination

Benchmark contamination refers to the problem where evaluation data — the questions and answers comprising a benchmark — appears in a model's training data, either accidentally or intentionally. As a result, the model appears to perform better on that benchmark than it actually generalizes to unseen data — it has 'memorized' benchmark answers rather than acquired underlying capabilities. Contamination is a systemic challenge: modern language models train on vast quantities of web data; popular benchmarks (MMLU, HumanEval, GSM8K, MATH) are freely available online, making accidental inclusion likely at scale. Economic incentives also create conditions for intentional contamination. Symptoms include: dramatically better benchmark scores than real-world task performance; large discrepancies between benchmark results and user experiences; the 'MMLU shuffle' effect — where randomly reordering answer choices significantly alters scores — a well-documented contamination signal. Countermeasures: private hold-out benchmarks kept secret before release; dynamic benchmarks with daily newly-generated questions; contamination detection through n-gram overlap analysis between training and test data; relying on independent external evaluations rather than self-reports. Organizations like METR, HELM, and ARC Evals develop increasingly contamination-resistant methodologies.