LLM Benchmarks Mostly Measure Exactly Nothing
The Allen Institute for AI has dropped BenchMIRT, a diagnostic framework designed to figure out what LLM benchmarks are actually measuring. As the industry drowns in endless claims of superhuman AI performance, this tool attempts to separate genuine capability from sophisticated data contamination and test memorisation.
* AI labs keep inventing