Resource	Info
Paper	http://arxiv.org/abs/2502.17419
Code & Data	https://github.com/zzli2022/Awesome-System2-Reasoning-LLM
Public	arXiv
Date	2025.03.01

Main Content

实现人类水平的智能需要完善从快速，直观 System1 到较慢，更故意的 System2 推理的过渡。虽然 System1 在迅速的启发式决策中表现出色，但System2 依赖于逻辑推理来进行更准确的判断和减少偏见。

推理LLM具有逐步处理信息的机制，使他们能够做出更准确，更合理的决策。

Monte Carlo Search

Four steps:

1. Selection, which chooses the child node with the highest priority using the UCB1 formula:
   $$\mathrm{UCB1}=\frac{w_i}{n_i}+c\sqrt{\frac{\ln N}{n_i}}$$
   where $w_i$ is the total reward of node $i$, $n_i$ is its visit count, $N$ is the parent node's visit count, and $c$ balances exploration and exploitation.
2. Expansion adds new nodes
3. Simulation performs random rollouts to evaluate them
4. Backpropagation updates node statistics.

MCTS has been widely used in tasks such as optimizing strategies in board games like GO and in robotic path planning, where it helps robots navigate dynamic enviornments effectively.

Analysis of the Features of Reasoning LLMs

Ouput Behaviour Perspective

Slow-thinking models engage in a latent generative process, particularly noticeable during the prediction of subsequent tokens.

Verification and Check Structure:

Analysis of OpenAI's o1 and o3 models indicates that their reasoning frameworks incorporate both macro-level actions for long-term strategic planning and micro-level actions, including "Wait", "Hold on", "Alternatively", and "Let's pause". These micro actions facilitate meticulous verification and iterative checking processes, ensuring precision in task execution.

Longer Inference Length & Time:

最近的研究表明，推理LLM通常会产生超过2000个令牌的产出，以解决编码和数学中的复杂问题。但是，这种扩展的输出长度有时会导致过度思考，在这种情况下，该模型在问题上花费了过多的时间，而无需改善解决方案。

Overly Cautious & Simple Problem Trap:

当前，推理LLM在竞争级数学，复杂编码，医学问题回答和多种方面的翻译等领域中表现出了强劲的性能。这些方案要求模型对问题进行细粒度分析，并根据给定条件执行仔细的逻辑推理。有趣的是，即使对于“ 2+3 =？”之类的直接问题，推理LLM也会表现出过度自信或不确定性。最近的研究指出，类似 o1 的模型倾向于生成多个解决方案回合，以方便数学问题，通常会探索不必要的路径。这种行为与缺乏更简单的问题的不同探索性动作形成鲜明对比，这表明该模型推理过程的效率低下。

Training Dynamic Perspective

Studies suggest that constructing Slow-thinking CoT datasets with a focus on hard samples leads to better generalization in fields like medicine and mathematics.

In comparison to simple CoT, Slow-thinking Supervised Fine-Tuning (SFT) data exhibits remarkable sample efficiency, often delivering comparable results with just 1/100th of the sample size.

Training LLMs for slow-thinking, as characterized by the LongCoT approach, results in relatively uniform gradient norms across different layeers. In contrast, fast-thinking, exemplified by the simplified CoT method, generates larger gradient magnitudes in the earlier layers, along with significant variability in gradient norms across layers.

Core Method

Structure Search

As shown in Table 1, we classify the actions in prior work into four categories:

1. Reasoning Steps as Nodes: Actions represent intermediate reasoning steps or decisions, such as selecting rules, applying transformations, or generating subquestions.
2. Token-level Decisions: Actions involve generating tokens or sequences (e.g., the next word, phrase, or code snippet).
3. Task-specific Structures: Actions are domain-specific, such as moving blocks in blocksworld, constructing geometry in geometry problem-solving, or modifying workflows in task planning.
4. Self-correction and Exploration: Actions forcus on revisiting, refining, or backtracking to improve previous reasoning steps.

Additionallye, as illustrated in Table 1, we classify the reward design into five categories:

1. Outcome-based Rewards: Rewards focus on the correctness or validity of the final outcome or solution, including the validation of reasoning paths or task success.
2. Stepwise Evaluations: Rewards are assigned at intermediate steps based on the quality of each step or its contribution toward the final outcome.
3. Self-evaluation Mechanisms: Rewards rely on the model’s own confidence or self-assessment (e.g., likelihood, next-word probability, or confidence scores).
4. Domain-specific Criteria: Rewards are tailored to specific tasks, such as symmetry and complexity in geometry or alignment with human preferences in text generation.
5. Iterative Preference Learning: Rewards are derived from comparing multiple solutions or reasoning paths, guiding learning dynamically.

Reward Modeling

• Outcome supervision emphasizes the correctness of the final answer at a higher level of granularity, and the resulting model is referred to as the Outcome Reward Model (ORM).
• Process supervision provides step-by-step labels for the solution trajectory, evaluating the quality of each reasoning step. The resulting model is known as the Process Reward Model (PRM).

PRM offers significant advantages in complex reasoning tasks for several key reasons. First, it provides fine-grained, step-wise supervision, allowing for the identification of specific errors within a solution path. This feature is especially valuable for RL and automated error correction. Second, PRM closely mirrors human reasoning behavior, which relies on accurate intermediate steps to reach correct conclusions. Unlike ORM, PRM avoids situations where incorrect reasoning can still lead to a correct final answer, thus ensuring more robust and interpretable reasoning.

PRM face challenges:

1. Lack of Explanations: Current PRMs often generate scores for reasoning steps without sufficient explanations, limiting interpretability and hindering their usefulness in refining reasoning during test-time.
2. Bias in Training Data: Data collection methods, such as MCTS, tend to introduce distributional biases, assigning disproportionately higher scores to majority of questions. As a result, PRMs struggle to effectively identify erroneous reasoning steps.
3. Early-Step Bias: PRMs show lower accuracy in predicting rewards for earlier reasoning steps compared to those closer to the final answer. This issue stems from the increased randomness and uncertainty associated with the initial steps in the reasoning process.

Self Improvement

Macro Action

Reinforcement Fine-Tuning

Reinforcement Fine-Tuning (RFT) is an innovative technique recently introduced by OpenAI, designed to enable developers and engineers to fine-tune existing models for specific domains or complex tasks.

Key advantages include:

1. Simplified Training Pipline: RL supervision streamlines data construction and training processes, eliminating the need for complex stepwise search mechanisms.
2. Enhanced Scalability: Online RL training facilitates efficient scaling on large datasets, particularly for complex reasoning tasks.
3. Emergent Properties: DeepSeek-R1 demonstrates unique emergent capabilities, such as Long-CoT reasoning, which are difficult to achieve through SFT alone.

RFT faces the following challenges:

1. Unclear Mechanism behind Reasoning: The underlying mechanisms driving the reasoning improvements in DeepSeek-R1 remain poorly understood. For example, while DeepSeek-R1 exhibits emergent properties (e.g., “Emergent Length Increasing”, “Aha moments”), studies such as suggest that capabilities like Long-CoT might already exist in the base model, rather than solely emerging from RL training. Furthermore, performance gains observed in smaller models (e.g., Qwen-Math2B/7B) occur without noticeable “Aha moments”, complicating causal interpretations.
2. Reward Model Saturation: Many existing RL algorithms face reward model saturation, typically manifested as exploration collapse after around 100 training steps. Although DeepSeek-R1 alleviates this issue through specialized reward formatting, methods like ReFT and Satori propose alternating sampling and SFT distillation to combat reward hacking and exploration collapse.
3. Unstable Long-CoT Generation: Long reasoning chains generated by RFT are prone to instability, including context overflow, failure to return final answers, and sensitivity to reward shaping. For instance, methods like inadvertently introduce cosine reward functions, which degrade performance with increased iterations. O1-Prune uses post-hoc length pruning techniques (via RL/SFT) to stabilize outputs.

Future directions for RFT may include several exciting and innovative advancements, such as:

1. Efficient and Stable RL Frameworks
2. Scaling RFT
3. Controlling Long-CoT Stability
4. Theoretical and Empirical Analysis

推理LLM的一个关键挑战是快速思考的功能的丧失，这会导致效率低下时，当简单的任务需要不必要的深层推理时。与人类在快速（系统1）和慢速（系统2）思考之间的流畅切换不同，当前的推理LLM难以维持这种平衡。虽然推理LLMS确保有意和彻底的推理，但快速思维的系统依靠先验知识来快速响应。

Benckmarks

🤖ChatGPT

Others