Harness Contamination

Harness contamination describes the situation where the test environment or evaluation harness itself influences or distorts a benchmark result, rather than merely measuring the model. The term gained prominence after several studies demonstrated that apparent model improvements were actually attributable to different harness configurations: two API parameters such as temperature and system prompt moved a model from 13.3 percent to 38.3 percent on the same benchmark, with no change in model weights. Harness contamination arises from factors including inconsistent prompt templates, differing tokenization or decoding strategies, inadequate isolation of test data from training data, or unbalanced sampling parameters. The problem is particularly insidious because it manifests not as a model defect but as a configuration variance in the test setup, and is therefore often not reported in benchmark publications. For engineering teams, harness contamination means that benchmark results are only trustworthy when the full evaluation path — from prompt through decoding parameters to answer extraction — is documented, versioned, and held identical across comparisons. Contamination can operate in both directions: a poor harness can unfairly penalize a strong model, and an optimized harness can artificially inflate a weak model's score. For organizations evaluating or procuring AI models, harness contamination is a systematic risk in model assessment: comparable benchmarks are only meaningful when the test setup is identically documented. Otherwise, procurement decisions can rest on false premises. We help teams design evaluation pipelines so that harness influences are isolated, documented, and controlled — ensuring benchmark results measure the model, not the test setup.