When a measure becomes a target, it ceases to be a good measure.

Training on rephrased test questions, exploiting automated judges’ preference for longer answers, and testing multiple variants but publishing only the highest score can all inflate leaderboard scores without a corresponding gain in capability.

Flawed benchmarks can steer model development in the wrong direction. Flawed post-training data can reinforce biases and erode existing capabilities.

Benchmarks’ Last Exam examines evaluations and training data themselves: whether an evaluation faithfully reflects a model’s capabilities, and whether higher scores correspond to real progress.