Resource	Info
Paper	https://arxiv.org/abs/2310.12931
Code & Data	https://github.com/eureka-research/Eureka
Public	ICLR
Date	2025.06.30

Summary Overview

Large Language Models (LLMs) have excelled as high-level semantic planners for sequential decision-making tasks. However, harnessing them to learn complex low-level manipulation tasks, such as dexterous pen spinning, remains an open problem. We bridge this fundamental gap and present EUREKA, a human-level reward design algorithm powered by LLMs.

Evolution-driven Universal REward Kit for Agent (EUREKA), a novel reward design algorithm powered by coding LLMs with the following contributions:

1. Achieves human-level performance on reward design
2. Solves dexterous manipulation tasks that were previously not feasible by manual reward engineering
3. Enable a new gradient-free in-context learning approach to reinforcement learning from human feedback (RLHF)

Main Content

Problem Setting and Definitions

The goal of reward design is to return a shaped reward function for a ground-truth reward function that may be difficult to optimize directly (e.g., sparse rewards); this ground-truth reward function may only be accessed via queries by the designer.

Definition 2.1. (Reward Design Problem (Singh et al., 2010)) A reward design problem (RDP) is a tuple $P = \langle M, \mathcal{R}, \pi_M, F \rangle$ where $M = (S, A, T)$ is the world model with state space $S$, action space $A$, and transition function $T$. $\mathcal{R}$ is the space of reward functions. $\mathcal{A}_M(\cdot) : \mathcal{R} \to \Pi$ is a learning algorithm that outputs a policy $\pi : S \to \Delta(A)$ that optimizes reward $R \in \mathcal{R}$ in the resulting Markov Decision Process (MDP), $(M, R)$. $F : \Pi \to \mathbb{R}$ is the fitness function that produces a scalar evaluation of any policy, which may only be accessed via policy queries (i.e., evaluate the policy using the ground truth reward function).

In an RDP, the goal is to output a reward function $R \in \mathcal{R}$ such that the policy $\pi := \mathcal{A}_M(R)$ that optimizes $R$ achieves the highest fitness score $F(\pi)$.

Reward Generation Problem. In our problem setting, every component within a RDP is specified via code. Then, given a string $l$ that specifies the task, the objective of the reward generation problem is to output a reward function code $R$ such that $F(\mathcal{A}_M(R))$ is maximized.

EUREKA outperforms human rewards.

EUREKA consistently improves over time.

🤖

Others