Resource	Info
Paper	https://openreview.net/pdf?id=qZMLrURRr9
Code & Data	/
Public	ICML
Date	2025.06.26

Main Content

Eureka 在 ERL 中应用了协同优化的想法，用于奖励生成：EA 将 LLM 生成的奖励函数进行群体进化，而 RL 则用于训练策略来评估最佳 reward function。但是 Eureka 具有两个限制：

1. Greedy Exploitation: Eureka 仅根据每次迭代的最佳 reward function 而改善，这可能会导致奖励函数不是最优的结果。
2. Suboptimal Parameter Assignment: Reward functions 通常包含许多参数，例如权重或系数。Eureka 借助于 LLM 直接进行设置，这通常是次优的。这样可能导致无法提供精确的知道。

作者为了解决上述问题，进一步扩展 ERL，提出了 R*

Two Stages: the initialization stage and the evolutionary improvement stage.

The initialization stage consists of two key steps: (1) Initializing the reward function population, which provides the foundation for subsequent reward function design; (2) Initializing the critic population, wich is used to collect trajectory segment pairs with high-quality labels.

1. Rward Population Initialization. R* leverages LLM to generate the initial reward function population $\mathbb{P}_{\text{Reward}}$ by providing the task description $L$ and environment code infomation $C$. To facilitate more efficient structure search in subsequent stages, we design the prompt $P_{\text{reward}}$ to guide the LLM in generating reward functions composed of multiple reward modules.
2. Critic Population Initialization. In addition to the reward population, R* also needs to generate a critic population $\mathbb{P}_{\text{Critic}}$, with each individual represented as a code function. These critics are used to compare trajectories and provide step-wise comparison labels. Similar to the reward population, R* takes task descriptoins $P$, environment code $C$, and the prompt $P_{\text{critic}}$ as inputs to generate the critic population $\mathbb{P}_{\text{Critic}}$.

After completing the initialization stage, we proceed to the reward function evolutionary improvement stage, which consists of three steps: population evaluation, evolution, and parameter optimization.

1. Reward Population Evaluation. For each reward function $F_i$ in the reward population $\mathbb{P}_{\text{reward}}$, we train PPO in parallel to solve the task using Isaac Gym. During training, we record the success rates of the learned policies, along with other relevant statistics, such as the mean, maximum, and minimum values of each reward module. The reward functions, along with their fitness values, are then stored in the reward function archive $D_F$. The success rate of the policy serves as the reward function fitness.
2. Reward Population Evolution. We select the reward function $F_\text{best}$ with the highest fitness, along with its associated learning information, to serve as feedback. Through the LLM reflection mechanism, we perform evolution operations such as modifying, deleting or adding modules to improve the reward function, thereby creating a new improved reward function. To fully exploit the high-quality reward functions discovered, we select parent reward functions from the reward function archive $D_F$ based on their fitness. Module-level crossover is then applied to these parents to generate new individuals, facilitating further exploration of the reward design space.
3. Reward Parameter Optimization. Using the critic population $\mathbb{P}_{\text{Critic}}$, we extract trajectory segments and apply comparision labels to the segments, creating a labeled trajectory buffer $D_T$. The reward functions are then optimized based on $D_T$, allowing them to better evaluate segment quality and guide policy learning.

Experiments

🤖

Others