StateBench Proves Language Models Still Lose Their Place

The claim: AllenAI presents StateBench, a new evaluation framework designed to rigorously test how language models track state changes and variable updates across extended document lengths.

The reality: It is a benchmark confirming what anyone debugging a 100,000-token prompt already knows: Claude and GPT-4 still lose track of whether a boolean was flipped three chapters ago.

Why should I care? Could be big
Another benchmark proving context length is a marketing metric rather than a solved engineering problem.

Read the original: StateBench: Testing how language models track state changes over long documents