{"id":599,"date":"2025-11-04T17:58:18","date_gmt":"2025-11-04T09:58:18","guid":{"rendered":"https:\/\/www.midytech.com\/?p=599"},"modified":"2025-11-04T17:58:18","modified_gmt":"2025-11-04T09:58:18","slug":"stanfords-7b-agent-outperforms-gpt-4o","status":"publish","type":"post","link":"https:\/\/www.midytech.com\/index.php\/2025\/11\/04\/stanfords-7b-agent-outperforms-gpt-4o\/","title":{"rendered":"Stanford’s 7B Agent Outperforms GPT-4o"},"content":{"rendered":"\n

On one hand, training an “all-in-one” large model to simultaneously handle reasoning, planning, and tool use offers the advantage of integration. However, it often suffers from unstable training and limited scalability in long-horizon reasoning tasks.On the other hand, prompt-based agent systems, while flexible, lack the ability to learn and self-optimize, preventing them from continuously evolving through interactions.How can we break through this bottleneck?A research team from Stanford University, in collaboration with Texas A&M University, the University of California, San Diego (UCSD), and Lambda, has proposed a novel solution: enabling agent systems to engage in online reinforcement learning within a “stream” of reasoning, thereby achieving continuous self-improvement and capability evolution.They introduced the AgentFlow<\/strong>framework, which adopts a modular architecture where four specialized agents work collaboratively, paired with a specially designed Flow-GRPO algorithm<\/strong>. This setup allows the system to continuously optimize its decision-making strategies in real-world interactive environments.Experimental results show that even a 7B-parameter AgentFlow outperforms both GPT-4o (~200B parameters) and Llama-3.1-405B across multiple tasks including search, mathematics, and science.The team leader shared their work on Twitter, receiving significant attention.The project has also climbed to the second place on the Hugging Face Paper Daily Leaderboard<\/strong>, as well as being among the most popular Hugging Face projects for the week<\/strong>.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

The Challenge of Credit Assignment in Long-Horizon Reasoning<\/h3>\n\n\n\n

The core challenge in training agent systems lies in multi-turn credit assignment<\/strong>: in environments with long time horizons and sparse rewards, how can we accurately determine the contribution of each decision step to the final outcome?Traditional single-model approaches integrate all functions into one large language model (LLM), outputting thoughts, tool calls, and responses in a unified manner using special tags (e.g., <tool_call><\/code>). While effective for short-horizon tasks, this approach tends to encounter issues in complex scenarios such as:<\/p>\n\n\n\n

\u2022Training instability due to excessively long reasoning chains,<\/p>\n\n\n\n

\u2022Difficulty tracing errors when incorrect tools are selected,<\/p>\n\n\n\n

\u2022Inability to dynamically adjust strategies based on environmental feedback.<\/p>\n\n\n\n

Existing agent systems (such as LangGraph, OWL, Pydantic, AutoGen) have achieved modularity but mostly rely on fixed prompt engineering and lack mechanisms for learning from experience.<\/p>\n\n\n\n

AgentFlow: Real-Time Multi-Agent Interaction and Learning in a “Stream”<\/h3>\n\n\n\n

The design philosophy of AgentFlow is to\u00a0decompose complex reasoning tasks among specialized agent modules<\/strong>, while enabling the\u00a0core decision-making module to continuously learn during interactions<\/strong>.<\/p>\n\n\n\n

\"\"<\/figure>\n\n\n\n

Four-Module Collaborative Architecture<\/h4>\n\n\n\n

The system consists of four memory-equipped, specialized agents:<\/p>\n\n\n\n

1.Planner <\/strong>Analyzes task requirements, formulates execution strategies, and selects the most appropriate tools.\u2192 This is the\u00a0core decision-making module<\/strong>and the\u00a0only trainable component<\/strong>of the system.<\/p>\n\n\n\n

2.Tool <\/strong>Responsible for actually invoking tool APIs and integrating the returned results.<\/p>\n\n\n\n

3.Evaluator<\/strong>Assesses intermediate results based on accumulated historical memory, determining whether they align with task goals and constraints.<\/p>\n\n\n\n

4.Solver <\/strong>Integrates all information and validation feedback to generate the final answer or propose the next action.<\/p>\n\n\n\n

    <\/ol>\n\n\n\n

    The key innovation<\/strong>is that the Planner is not static<\/strong>\u2014it is continuously optimized in real-time through online (on-policy) reinforcement learning<\/strong>within the reasoning stream.After each interaction round, the system updates the Planner\u2019s decision strategy based on the success or failure of the final outcome, incorporating these optimizations into the system\u2019s memory to form a closed-loop adaptive learning process.<\/p>\n\n\n\n

    \"\"<\/figure>\n\n\n\n

    Flow-GRPO Algorithm: Solving the Credit Assignment Problem<\/h3>\n\n\n\n

    The team proposed the Flow-GRPO (Flow-based Group Relative Policy Optimization)<\/strong>algorithm, specifically designed for multi-turn reasoning scenarios. Its core idea is to broadcast the final trajectory reward (success\/failure) back to each individual action<\/strong>, transforming the complex problem of multi-step reinforcement learning into a series of single-step policy updates.The approach works as follows:<\/strong><\/p>\n\n\n\n

    1.Collect the complete reasoning trajectory (from initial task to final result).<\/p>\n\n\n\n

    2.Compute the\u00a0outcome reward<\/strong>based on the final result.<\/p>\n\n\n\n

    3.Distribute this reward across each planning action in the trajectory.<\/p>\n\n\n\n

    4.Use a\u00a0relative advantage function<\/strong>to compute the advantage of each action and perform policy gradient updates.<\/p>\n\n\n\n

      <\/ol>\n\n\n\n

      This method effectively alleviates the issue of sparse rewards while maintaining training stability.Online learning enables the system to:<\/strong><\/p>\n\n\n\n

      \"\"<\/figure>\n\n\n\n

      \u2022Quickly correct erroneous tool calls,<\/p>\n\n\n\n

      \u2022Explore better ways of decomposing subtasks,<\/p>\n\n\n\n

      \u2022Dynamically adjust reasoning depth based on environmental feedback.<\/p>\n\n\n\n

      Experimental Results: The Underdog Model Wins<\/h3>\n\n\n\n

      The research team conducted systematic evaluations across 10 cross-domain benchmarks, covering four major categories: knowledge retrieval, agent tasks, mathematical reasoning, and scientific reasoning.<\/p>\n\n\n\n

      Performance Comparison<\/h4>\n\n\n\n

      Using Qwen-2.5-7B-Instruct<\/strong>as the base model, AgentFlow significantly outperformed baselines<\/strong>in all categories:<\/p>\n\n\n\n

      \u2022Knowledge Retrieval:<\/strong>+14.9% improvement over baseline<\/p>\n\n\n\n

      \u2022Agent Reasoning:<\/strong>+14.0%<\/p>\n\n\n\n

      \u2022Mathematical Reasoning:<\/strong>+14.5%<\/p>\n\n\n\n

      \u2022Scientific Reasoning:<\/strong>+4.1%<\/p>\n\n\n\n

      More surprisingly, cross-scale comparisons revealed:<\/strong><\/p>\n\n\n\n

      \u2022A\u00a07B AgentFlow<\/strong>outperformed\u00a0GPT-4o (\u2248200B)<\/strong>by\u00a08.2%<\/strong>on search tasks<\/p>\n\n\n\n

      \u2022It also surpassed\u00a0Llama-3.1-405B<\/strong>by\u00a015.8%<\/strong>on agent tasks<\/p>\n\n\n\n

      \u2022Even a\u00a03B AgentFlow<\/strong>was able to beat the 405B baseline on multiple tasks<\/p>\n\n\n\n

      \"\"<\/figure>\n\n\n\n

      Key Findings from Ablation Studies<\/h3>\n\n\n\n

      1.Online Learning vs Offline Learning<\/strong>Comparative experiments showed that training the Planner using traditional supervised fine-tuning (SFT) methods led to an\u00a0average performance drop of 19%<\/strong>, proving that\u00a0online learning in real interaction environments is essential for efficient reasoning<\/strong>.<\/p>\n\n\n\n

      2.Autonomous Exploration of New Strategies<\/strong>The trained system learns to select appropriate tool combinations based on task characteristics. It also spontaneously discovers new tool usage patterns\u2014such as combining Wikipedia Search with enhanced Web Search\u2014to achieve deeper information mining. These patterns were rarely observed in untrained reasoning flows.<\/p>\n\n\n\n

      3.Dynamic Reasoning Depth<\/strong>On dense reasoning tasks like multi-hop search, the trained AgentFlow demonstrates\u00a0“intelligent laziness”<\/strong>: it uses fewer reasoning steps for simple tasks and increases depth only for complex ones.As the maximum step limit increases, performance improves steadily while the average number of steps does not rise proportionally.<\/p>\n\n\n\n

      4.Value of Modular Collaboration<\/strong>While the reasoning flow alone brings performance gains, untrained systems tend to fall into loops or get stuck.After reinforcement learning, the system shows clear improvements in\u00a0tool call accuracy<\/strong>,\u00a0subtask planning granularity<\/strong>, and\u00a0overall performance<\/strong>.The authors provide an illustrative example where, prior to Flow-GRPO training, the system would repeatedly output the same subgoals and tool calls upon encountering errors (e.g., Python variable definition mistakes), wasting time and efficiency.After training, the Planner adjusts its strategy based on past errors, using more precise subgoals to guide subsequent steps\u2014and often succeeds in just one step.This vividly demonstrates the\u00a0great potential of reinforcement learning in real-world agent reasoning<\/strong>.<\/p>\n\n\n\n

      Technical Significance & Future OutlookThe value of the AgentFlow work lies in:<\/h3>\n\n\n\n

      1.A New Training Paradigm<\/strong>It proves that agent systems can acquire learning abilities akin to large models through online reinforcement learning\u2014and can even be more efficient on specific tasks.<\/p>\n\n\n\n

      2.Validation of “Small but Mighty”<\/strong>It shows that, with proper system design, small models leveraging modular collaboration and continuous learning can outperform large general-purpose models on complex reasoning tasks.<\/p>\n\n\n\n

      3.Ideas for Scalable AI<\/strong>The modular architecture allows flexible addition of new tools and adjustment of module functions.<\/p>\n\n\n\n

        <\/ol>\n\n\n\n

        AgentFlow demonstrates at least one thing clearly: the future of Agentic AI doesn\u2019t have to rely solely on scaling up model size. Innovation in system architecture + efficient training methods may be a more promising direction to explore.<\/strong><\/p>\n","protected":false},"excerpt":{"rendered":"

        On one hand, training an &#822…<\/p>\n","protected":false},"author":2,"featured_media":605,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[8],"tags":[142,24,141,143,16],"class_list":["post-599","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-deep-tech","tag-7b","tag-ai","tag-gpt-4o","tag-stanford","tag-technology"],"_links":{"self":[{"href":"https:\/\/www.midytech.com\/index.php\/wp-json\/wp\/v2\/posts\/599","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.midytech.com\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.midytech.com\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.midytech.com\/index.php\/wp-json\/wp\/v2\/users\/2"}],"replies":[{"embeddable":true,"href":"https:\/\/www.midytech.com\/index.php\/wp-json\/wp\/v2\/comments?post=599"}],"version-history":[{"count":1,"href":"https:\/\/www.midytech.com\/index.php\/wp-json\/wp\/v2\/posts\/599\/revisions"}],"predecessor-version":[{"id":606,"href":"https:\/\/www.midytech.com\/index.php\/wp-json\/wp\/v2\/posts\/599\/revisions\/606"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.midytech.com\/index.php\/wp-json\/wp\/v2\/media\/605"}],"wp:attachment":[{"href":"https:\/\/www.midytech.com\/index.php\/wp-json\/wp\/v2\/media?parent=599"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.midytech.com\/index.php\/wp-json\/wp\/v2\/categories?post=599"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.midytech.com\/index.php\/wp-json\/wp\/v2\/tags?post=599"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}