Physicists Marked GPT-6 Astra Answers Wrong, Then Found Errors in Their Own Solutions
The authors of TheoPhysEval, a benchmark of 50 unpublished research-level problems in theoretical physics, report that on several tasks their initial reference solutions turned out to be incorrect. After the fixes, outside experts confirmed that GPT-6 Astra had answered correctly.
Running in Codex with tools and web search, Astra solved 49 of 50 problems for a score of 98%, at an average cost of under $3 per problem and without human hints. A typical task would take a narrow specialist up to a full working day.
The authors are explicit about scope: the benchmark tests difficult calculations using known methods, and creating new physical theories demands more. Still, they argue, an assistant that makes experts correct their own reference answers is worth attention.
Related: Fields Medal Laureates Declare Misalignment in AI Mathematics, OpenAI Claims Solution to Navier-Stokes Millennium Prize Problem