In reply to @natesilver538

Nate Silver

Nate Silver

@natesilver538 · Twitter ·

Benchmarks present all the classic Goodhart's Law problems. Teaching to the test, focusing resources on beating the benchmarks vs. broader use cases, etc. Plus, almost all of this data is self-reported. And on top of that, the models are almost certainly cheating in some cases!