Home | Benchmarks’ Last Exam

中文

When a measure becomes a target, it ceases to be a good measure.

Goodhart’s lawMarilyn Strathern, 1997

Training on rephrased test questions, exploiting automated judges’ preference for longer answers, and testing multiple variants but publishing only the highest score can all inflate leaderboard scores without a corresponding gain in capability.

Flawed benchmarks can steer model development in the wrong direction. Flawed post-training data can reinforce biases and erode existing capabilities.

Benchmarks’ Last Exam examines evaluations and training data themselves: whether an evaluation faithfully reflects a model’s capabilities, and whether higher scores correspond to real progress.

Benchmarks’ Last Exam

All reviews

Terminal-Bench 4.0 Review Report

66 tasks: 10 Accepted, 6 Doubtful and 50 Refused.

2026-11-04

Terminal-Bench-Science Review Report

70 scientific tasks: 31 Accepted, 5 Doubtful and 34 Refused.

2026-10-30

AutomationBench Review Report

100 sampled tasks: 19 Accepted, 10 Doubtful and 71 Refused.

2026-10-25

GDP.pdf Review Report

100 tasks: 40 Accepted, 13 Doubtful and 47 Refused.

2026-10-20

Benchmark Analysis

Explore score distributions, model rankings and original sources.

Contact Us

Find flaws in benchmarks, post-training data and reinforcement learning environments.

Let every step toward AGI rest on real, verifiable advances in capability.