Frontier and Center: Who evaluates the evaluations?

AI evaluation methods are being re-examined to provide more nuanced insights into AI agent capabilities. Current methods focus on pass/fail exams, but a more detailed approach is needed to understand where AI agents fall short. A new approach rooted in information theory is being explored to add fidelity to benchmarks and better understand AI performance. This approach has exposed issues with the quality of evaluation cases.