her replay

HER (Hindsight Experience Replay) is a technique for learning from failure in goal-conditioned RL — when the agent fails to reach the intended goal, HER relabels the experience with the actually achieved state as the goal, creating a successful learning signal from every trajectory.

How HER Works

• Original: Agent tries to reach goal $g$, ends up at state $s'$ ≠ $g$ — failed trajectory, negative reward.
• Relabeling: Create a new experience with goal $g' = s'$ — the same trajectory now "succeeded" at reaching $s'$.
• Learning: The agent learns to reach many states, even though it failed at the original goal.
• Strategies: Relabel with final state, random future state, or closest achieved state.

Why It Matters

• Sparse Rewards: In goal-conditioned tasks with sparse rewards (only at goal), standard RL gets almost no learning signal — HER solves this.
• Sample Efficiency: Every failed trajectory becomes useful — dramatically improves sample efficiency.
• Robotics: HER was crucial for robotic manipulation — reaching, pushing, and grasping with sparse rewards.

HER is learning from every failure — relabeling failed goals with achieved states to extract learning from every trajectory.