grade-school-math

This seems to refer to the GSM8K (grade school math) benchmark [1]. GPT-4 scores 0.92 on that one. A breakthrough could mean it gets all of them correct. This would have major implications for long-term planning. For instance, if you have a sequence of 10 steps, each with a 90% success rate, the overall success rate after all 10 steps falls to just 34%. This is one of the reasons why agents like AutoGPT often fail in complex tasks. [1] https://github.com/openai/grade-school-math

It’s likely that the grade school math that was referenced is GSM8K https://github.com/openai/grade-school-math which LLMs perform poorly on.