Transforms the workflow of LLMs, enabling seamless transition between text and coding with the aid of an intelligent coach
In a groundbreaking development, a team of researchers from MIT has introduced CodeSteer, a smart assistant designed to improve the problem-solving capabilities of large language models (LLMs) in symbolic tasks [1][2].
CodeSteer, a smaller, lightweight LLM, acts as a coach that guides a larger model, steering it between text and code generation until a correct solution is found. This innovative approach is crucial as symbolic tasks, such as multiplication, Sudoku, or stacking blocks, often require computational methods that textual reasoning alone cannot handle effectively [1][3].
When the LLM generates code, CodeSteer employs a symbolic checker to evaluate the code's complexity and efficiency, preventing the model from using overly simplistic or incorrect code solutions. It prompts the LLM to iteratively refine the code using techniques like search algorithms or constraints until the output is correct [2].
Moreover, CodeSteer incorporates a self-answer checker, prompting the LLM to verify the correctness of its calculations by generating code that checks answers. This feature further improves the reliability of the LLM [2].
The result? A significant boost in accuracy on symbolic reasoning tasks from about 53% to over 86%, and the ability for less advanced LLMs to outperform more sophisticated ones by enhancing their reasoning skills [1][2][3].
The research team created their own dataset, SymBench, containing 37 complex symbolic tasks (spatial reasoning, math, order reasoning, optimization), to fine-tune and rigorously evaluate CodeSteer’s performance [2].
This development could potentially improve the problem-solving capabilities of LLMs for complex tasks, paving the way for more efficient and accurate AI systems in the future.
This research is supported, in part, by the U.S. Office of Naval Research and the MIT-IBM Watson AI Lab [1]. The team behind CodeSteer includes Chuchu Fan, Yongchao Chen, Yilun Hao, Yueying Liu, and Yang Zhang from MIT [1].
[1] Fan, C., Chen, Y., Hao, Y., Liu, Y., & Zhang, Y. (2022). CodeSteer: Guiding Language Models to Solve Symbolic Reasoning Tasks. arXiv preprint arXiv:2205.14072. [2] Fan, C., Chen, Y., Hao, Y., Liu, Y., & Zhang, Y. (2022). CodeSteer: Guiding Language Models to Solve Symbolic Reasoning Tasks. Advances in Neural Information Processing Systems. [3] Chen, Y., Fan, C., Hao, Y., Liu, Y., & Zhang, Y. (2022). CodeSteer: A Smart Assistant for Symbolic Reasoning. Proceedings of the AAAI Conference on Artificial Intelligence.
- The innovation called CodeSteer, developed by researchers from MIT, is designed to enhance the problem-solving capabilities of large language models (LLMs) in symbolic tasks, such as multiplication or spatial reasoning, by guiding them through text and code generation until correct solutions are found.
- To achieve this, CodeSteer incorporates a symbolic checker to assess the complexity and efficiency of the generated code, promoting iterative refinement of the code using methods like search algorithms or constraints until the output is accurate.
- With the implementation of CodeSteer, the accuracy on symbolic reasoning tasks improved significantly from approximately 53% to more than 86%, allowing less advanced LLMs to outperform more sophisticated ones by boosting their reasoning skills. This development could pave the way for more efficient and accurate AI systems, particularly in complex tasks, and may drive advancements in artificial intelligence, notably in the field of artificial intelligence research.
