Resource	Info
Paper	http://arxiv.org/abs/2505.13417
Code & Data	https://github.com/THU-KEG/AdaptThink
Public	arXiv
Date	2025.06.20

Summary Overview

作者认为对于简单的问题，没有必要使用 thinking 模式，而是直接进行回答也能获得很高的准确率。从效率上来说，模型应该有选择的使用 thinking，而不是所有的都开。通过强化学习训练模型，使得模型能够在推理时，自动地选择是否使用 thinking。

AdaptThink, a novel RL algorithm to teach reasoning models to choose the optimal thinking mode adaptively based on problem difficulty.

1. a constrained optimization objective that encourages the model to choose NoThinking while maintaining the overall performance;
2. an importance sampling strategy that balances Thinking samples during on-policy training, thereby enabling cold start and allowing the model to explore and exploit both thinking modes throughout the training process.

Main Content

Can the reasoning model learn to select Thinking or NoThinking mode adaptively based on the difficulty of the input problem, thereby achieving more efficient reasoning without sacrificing or even improving performance?

Constrained Optimization Objective

an ideal selection policy should prefer to choose NoThinking as long as the overall performance is not diminished. Let $\pi_{\theta_\text{ref}}$ denote the reference model, which is the initial $\pi_\theta$ and remains unchanged during training.

To solve this constrained optimization problem, we incorporate the constraint into the objective as a penalty term, with a penalty weight $\lambda\geq0$:

By dividing the both side by $\lambda$, letting $\delta=\frac{1}{\lambda}$, and reorganizing the terms about $\pi_{\theta{\text{ref}}}$, we have:

In pracitice, $\mathbb{E}_{y^\prime\sim\pi_{\theta_{\text{ref}}}(\cdot|x)}R(x,y^\prime)$ can be approximated by pre-sampling before training. Specifically, we sample $K$ responses from $\pi_{\theta_{\text{ref}}}(\cdot|x)$ for each $x$, and calculate their mean reward:

Then the optimization objective becomes:

Since $\mathbb{1}(y_1 = \langle/\text{think}\rangle)$ and $R(x,y)$ are not differentiable, we employ policy gradient method to solve this optimization. Specifically, let $\pi_{\theta_\text{old}}$ be a distribution equal to $\pi_\theta$ without gradient update, and define the advantage function: $A(x,y)=\mathbb{1}(y_1 = \langle/\text{think}\rangle)\cdot\delta+R(x,y)-\bar{R}_\text{ref}(x)$. Then the objective can be converted into a PPO-style loss without KL penalty:

To solve cold-start challenge, we employ the technique of importance sampling. Specifically, we define a new distribution $\pi_{\text{IS}}(\cdot|x)$:

During training, we sample responses $\{y^i\}_{i=1}^K$ from $\pi_{\text{IS}}(\cdot|x)$ instead of $\pi_{\theta_{\text{old}}}(\cdot|x)$, so that half of the samples in a batch are in Thinking mode and the other half are NoThinking.

🤖

Others