BLOGS

Reinforcement Learning

Published Jul 20, 2025 - Author: Guoxiang (Shawn) Zhou

Share

Overview

Supervised learning needs a label for every input: this image is a cat, this point is ground. For many robotics problems there is no such label. Nobody can write down the correct torque for each of a quadruped’s 12 joints at every millisecond while it climbs stairs. What we can say is whether the outcome was good: the robot moved at the commanded speed, did not fall, and did not waste energy.

Reinforcement learning (RL) learns from exactly that kind of signal. An agent tries actions, observes what happens, receives a scalar reward, and gradually changes its behavior to collect more reward over time. No one tells it which action was correct; it has to find out by trial and error, and it has to figure out which of its many earlier actions deserve credit for a reward that arrives later.

This post covers:

  1. The interaction loop and the formal model behind it (the Markov decision process).
  2. The components every RL method is built from: policy, reward, return, value function, model.
  3. The main algorithm families, with PPO in some detail, since it trains most robot controllers today.
  4. How RL produces a locomotion controller for a legged robot: simulation, rewards, observations, domain randomization, teacher-student training, and deployment.

The interaction loop

Everything in RL is built on one loop. At each time step $t$ the agent observes the state $s_t$ of the environment, picks an action $a_t$, and the environment responds with a new state $s_{t+1}$ and a reward $r_t$.

flowchart LR
  rlA["Agent<br/>policy π(a | s)"] -- "action aₜ" --> rlE["Environment<br/>robot + world"]
  rlE -- "next state sₜ₊₁" --> rlA
  rlE -- "reward rₜ" --> rlA

For a walking robot:

Term Example for a quadruped
Agent The neural network controller
Environment Robot body, actuators, terrain, gravity, contacts
State $s_t$ Base orientation and velocity, joint angles and speeds, terrain around the feet
Action $a_t$ 12 joint position targets
Reward $r_t$ + tracking the commanded velocity, - falling, - energy, - jerky motion
Episode One run from a reset until a fall or a time limit, e.g. 20 s

The loop repeats for an episode. Then the environment resets and a new episode starts. Training is millions of episodes.

Markov decision process

The loop is formalized as a Markov decision process (MDP), the tuple $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$:

Markov means the next state depends only on the current state and action, not on the history before it. That is a property of what you put into the state. Joint angles alone are not Markov for a walking robot, since they do not tell you how fast the joints are moving; add joint velocities and they are much closer. When the agent sees only part of the true state (a camera does not see the friction coefficient), the problem is a partially observable MDP (POMDP), and the policy usually gets a history of observations or a recurrent memory to compensate. Every real robot is a POMDP.

Components of an RL agent

Policy

The policy $\pi$ is the agent’s behavior: a mapping from state to action. It is the thing we are ultimately learning, and the thing that runs on the robot.

In deep RL the policy is a neural network $\pi_\theta$ with weights $\theta$. For locomotion it is often a small multilayer perceptron (three layers of 128 to 512 units), sometimes with a recurrent layer, which runs in well under a millisecond on a CPU.

Reward

The reward $r_t$ is a single number per step that says how good that step was. It is the only way to tell the agent what you want. The reward hypothesis says that any goal can be expressed as maximizing the expected sum of a scalar reward. In practice, writing that scalar so that it produces the behavior you actually want is the hardest part of applied RL.

Rewards are either sparse (+1 when the robot reaches the goal, 0 otherwise) or dense (a small reward every step for moving in the right direction). Sparse rewards are easy to specify but hard to learn from, because a random policy almost never sees a non-zero reward. Dense, shaped rewards learn quickly but invite reward hacking: the agent finds a way to score well that you did not intend, such as vibrating its legs to get a velocity reward without walking.

Return and discount

The agent does not maximize the reward of the next step. It maximizes the return, the discounted sum of all future rewards:

\[G_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \dots = \sum_{k=0}^{\infty} \gamma^k r_{t+k} \tag{1}\]

The discount $\gamma$ in Eq. (1) does three things:

Value functions

A value function predicts the return. It is how the agent assigns credit to actions whose payoff comes later.

Values satisfy a recursive relation, the Bellman equation, Eq. (2). The value of a state is the immediate reward plus the discounted value of where you land:

\[V^\pi(s) = \mathbb{E}_{a \sim \pi,\ s' \sim P} \left[ r(s, a) + \gamma V^\pi(s') \right] \tag{2}\]

For the optimal policy, the expectation over actions becomes a max:

\[V^*(s) = \max_a \mathbb{E}_{s' \sim P} \left[ r(s, a) + \gamma V^*(s') \right] \tag{3}\]

If the dynamics $P$ are known and the state space is small, you can apply Eq. (3) repeatedly to every state until the values stop changing. That is value iteration, and acting greedily with respect to $V^*$ gives the optimal policy.

The figure solves a small grid world this way. The robot starts at S and wants to reach G. Entering G pays a reward of +10 and ends the episode. The bottom row in front of the goal is a cliff: entering it pays -10 and also ends the episode. Both are terminal states, so their value is 0: the +10 is the reward for the last step, and it shows up in the values of the cells next to G, not in G itself. Each step costs a small amount, and with some probability the robot slips sideways instead of moving where it intended, like a foot sliding on a wet surface.

high value V(s) low value most likely path from S
Value iteration on a cliff grid world. Each cell shows its value $V^*(s)$ and the greedy action. With slip and a cheap step, the best route leaves the cliff edge and takes a longer, safer detour. Make each step expensive and the policy accepts the risk and walks along the edge. Lower $\gamma$ and the goal’s influence fades a few cells away from it, so the start cell barely knows the goal exists.

Things to try:

The same trade-offs show up on a real robot: the discount sets how far ahead it plans, and the balance between task reward and penalties sets how cautious it is.

Model

A model is the agent’s own prediction of the environment, $\hat{P}(s’ \mid s, a)$ and $\hat{R}(s, a)$. Model-based RL learns or is given a model and plans with it (value iteration above is the extreme case: a perfect model). Model-free RL never builds a model; it learns values or a policy directly from experience. Most robot locomotion work is model-free inside a simulator, which is a model, just not one the agent can query for planning.

Exploration and exploitation

To find good actions the agent must try actions it is unsure about (exploration). To collect reward it must use what it already knows (exploitation). Doing only one fails: pure exploitation gets stuck on the first strategy that works at all, pure exploration never uses what it learned. Common mechanisms:

Algorithm families

flowchart TB
  rlRoot["RL algorithms"] --> rlMF["Model-free"]
  rlRoot --> rlMB["Model-based<br/>MuZero, Dreamer, MBPO"]
  rlMF --> rlVB["Value-based<br/>learn Q, act greedily"]
  rlMF --> rlPG["Policy gradient<br/>optimize π directly"]
  rlVB --> rlQL["Q-learning, DQN"]
  rlPG --> rlRF["REINFORCE"]
  rlPG --> rlAC["Actor-critic<br/>policy + value"]
  rlAC --> rlPPO["PPO (on-policy)"]
  rlAC --> rlSAC["SAC, TD3 (off-policy)"]

Value-based: Q-learning and DQN

Learn $Q(s, a)$ and act with $a = \arg\max_a Q(s, a)$. The update moves $Q$ toward the one-step temporal difference (TD) target:

\[Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right] \tag{4}\]

The bracket in Eq. (4) is the TD error: what actually happened (one real reward plus the estimated rest) minus what was predicted. DQN (2015) replaced the table with a neural network, plus a replay buffer and a slowly updated target network for stability, and learned to play Atari games from pixels. The weakness for robotics is the $\max_a$: with 12 continuous joint targets there is no finite list of actions to take the max over.

Policy gradient

Optimize the policy directly. Write the objective as the expected return $J(\theta) = \mathbb{E}_{\pi_\theta}[G_0]$. The policy gradient theorem gives its gradient as

\[\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta} \left[ \sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t) \, A(s_t, a_t) \right] \tag{5}\]

Read Eq. (5) as: for each action taken, increase its log-probability in proportion to how much better than average it turned out. Good actions become more likely, bad ones less likely. This works for continuous actions out of the box. REINFORCE uses the full sampled return in place of $A$, which is unbiased but very noisy.

Actor-critic

Use two networks: an actor (the policy) and a critic (a value function $V_\phi(s)$). The critic estimates the advantage $A$ in Eq. (5), which lowers the variance of the policy gradient a lot. The simplest estimate is the TD error, $A_t \approx r_t + \gamma V(s_{t+1}) - V(s_t)$. Generalized Advantage Estimation (GAE) blends TD errors over many steps with a parameter $\lambda$ that trades bias (small $\lambda$) against variance ($\lambda \to 1$).

Two main sub-families:

  On-policy (PPO) Off-policy (SAC, TD3)
Data used for an update Only fresh rollouts from the current policy Any past data, from a replay buffer
Sample efficiency Low High
Stability, tuning effort Robust, few sensitive hyperparameters More sensitive
Scales with parallel simulation Very well Less naturally
Typical use Locomotion in simulation Real-robot learning, manipulation

With thousands of parallel simulated robots, samples are cheap and stability matters more than sample efficiency. That is why PPO is the default for legged locomotion.

PPO

The naive policy gradient has a failure mode: one large update step can wreck a good policy, and since the next batch of data comes from that wrecked policy, it may never recover. Proximal Policy Optimization (PPO) keeps each update close to the policy that collected the data.

Let $r_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\theta_\text{old}}(a_t \mid s_t)$ be the ratio of new to old action probability. PPO maximizes the clipped surrogate objective

\[L^\text{CLIP}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) A_t,\ \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) A_t \right) \right] \tag{6}\]

with $\epsilon$ typically 0.2. Read it term by term.

The ratio. $r_t(\theta)$ measures how much the update has changed the probability of the action that was actually taken. $r_t = 1$ means unchanged (at the start of each update, $\theta = \theta_\text{old}$), $r_t = 1.5$ means 50% more likely, $r_t = 0.5$ means half as likely. The first term inside the min, $r_t(\theta) A_t$, is the plain policy gradient objective written with this ratio: its gradient at $r_t = 1$ equals Eq. (5). On its own it has no limit. For a good action ($A_t > 0$) it keeps rewarding a larger $r_t$, so a few gradient steps on the same batch can push the probability far beyond what the data supports.

The clip. $\text{clip}(x, l, u)$ limits a number to the interval $[l, u]$:

\[\text{clip}(r, 1 - \epsilon, 1 + \epsilon) = \begin{cases} 1 - \epsilon & r < 1 - \epsilon \\ r & 1 - \epsilon \le r \le 1 + \epsilon \\ 1 + \epsilon & r > 1 + \epsilon \end{cases} \tag{7}\]

With $\epsilon = 0.2$ the ratio is confined to $[0.8, 1.2]$. Outside that interval the clipped term is a constant, so its gradient with respect to $\theta$ is zero: the sample stops asking for more change. If an action had probability 0.5 under the old policy and turned out better than expected, the update gains nothing from pushing it beyond 0.6.

The min. Clipping alone would also flatten the cases where the update went the wrong way, and the policy would never correct them. Taking the min of the clipped and unclipped terms picks whichever is smaller, a pessimistic lower bound on the improvement. The result is that the clip removes only the incentive to move further in the helpful direction, never the penalty for moving in the harmful one:

Advantage Ratio Term chosen by the min Gradient on the action probability
$A_t > 0$ (good action) $r_t < 1 - \epsilon$ unclipped $r_t A_t$ Up: undo a decrease that went the wrong way
$A_t > 0$ inside $[1 - \epsilon, 1 + \epsilon]$ both equal Up
$A_t > 0$ $r_t > 1 + \epsilon$ clipped $(1 + \epsilon) A_t$ Zero: already increased enough
$A_t < 0$ (bad action) $r_t < 1 - \epsilon$ clipped $(1 - \epsilon) A_t$ Zero: already decreased enough
$A_t < 0$ inside $[1 - \epsilon, 1 + \epsilon]$ both equal Down
$A_t < 0$ $r_t > 1 + \epsilon$ unclipped $r_t A_t$ Down: undo an increase that went the wrong way

The figure plots these six cases.

PPO clipped objective unclipped r · A flat: zero gradient, no further push
The PPO objective, Eq. (6), for one sample. For a good action (left), increasing its probability helps only until the ratio reaches $1 + \epsilon$; beyond that the objective is flat and the gradient is zero. For a bad action (right), decreasing its probability helps only down to $1 - \epsilon$. Taking the min keeps the unclipped, pessimistic branch when the update has already moved the wrong way.

The flat regions of Eq. (6) are the point: once an action’s probability has moved by more than $\epsilon$ in the helpful direction, that sample stops pushing. This allows several epochs of minibatch gradient steps on the same batch of data without the policy running away. The clip is a soft limit, not a hard constraint: other samples can still move the policy, and parameters are shared across states, so implementations also monitor the KL divergence between the old and new policy and stop early or lower the learning rate when it grows too large.

One PPO iteration:

for iteration in range(num_iterations):
    # 1. Rollout: run the current policy in N parallel environments for T steps.
    batch = collect_rollouts(policy, envs, num_steps=T)  # obs, actions, rewards, dones, log_probs

    # 2. Advantages with GAE, using the critic's value estimates.
    values = critic(batch.obs)
    advantages, returns = gae(batch.rewards, values, batch.dones, gamma=0.99, lam=0.95)

    # 3. Several epochs of minibatch updates on the same data.
    for epoch in range(5):
        for mb in batch.minibatches(4):
            ratio = exp(policy.log_prob(mb.obs, mb.actions) - mb.log_probs)
            surrogate = min(ratio * mb.adv, clip(ratio, 1 - 0.2, 1 + 0.2) * mb.adv)
            policy_loss = -mean(surrogate) - entropy_coef * policy.entropy(mb.obs).mean()
            value_loss = mean((critic(mb.obs) - mb.returns) ** 2)
            optimize(policy_loss + value_coef * value_loss)

With a GPU simulator, $N$ is 4096 robots and $T$ is 24 steps, so each iteration collects about 100,000 transitions in well under a second of wall time.

RL for robot controllers

Why learn a controller

The classical approach to legged locomotion is model-based: a simplified dynamics model, a contact schedule, and model predictive control (MPC) that optimizes forces over a short horizon. It works well and is interpretable. It struggles where the model is wrong: slippery or deformable ground, unexpected contacts, missed steps, terrain that does not match the map.

RL moves the hard work offline. Instead of solving an optimization on the robot at every step, it trains a policy against millions of situations in simulation, including the ones that break the model. On the robot, the policy is one forward pass of a small network.

  Model-based (MPC, WBC) Learned policy (RL)
Online cost Optimization every step One network forward pass
Needs Accurate model, contact schedule, tuning Simulator, reward design, compute
Robustness to unmodeled effects Limited by the model Learned from randomized simulation
Interpretability, guarantees Good: constraints are explicit Weak: behavior is implicit in weights
Changing the behavior Edit costs or constraints Retrain

The two are not exclusive. Common hybrids use RL to produce references for a model-based controller, or a model-based controller as a safety filter around a learned policy.

The controller on the robot

learned in simulation Command target velocity operator or navigation Policy network small MLP (+ memory) ~50 Hz Joint PD control τ = Kp(q* - q) - Kd q̇ in the drives, kHz rate Robot + terrain 12 actuated joints contacts, slip Observation o_t base velocity, gravity direction, joint states, previous action, terrain heights + command State estimation base pose and twist, elevation map Sensors IMU, encoders, LiDAR, depth q* τ next step
A learned locomotion controller as deployed. Only the policy network is learned. It does not output torques directly but joint position targets $q^*$, which a fast PD loop in the drives tracks. State estimation and the elevation map are the classical perception stack; they turn sensor data into the observation the policy was trained on. The command is not a separate input: it is concatenated into the same observation vector, so the policy learns to read it like any other measurement.

Design choices that matter:

Training in simulation

Real-robot training is slow and breaks hardware. Locomotion policies are trained entirely in simulation (Isaac Gym/Isaac Lab, MuJoCo, RaiSim) and deployed zero-shot. GPU simulators step thousands of robots in parallel, which brings training of a basic flat-ground walking policy down to minutes and rough-terrain policies to hours.

flowchart TB
  rlCur["Terrain curriculum<br/>harder terrain when succeeding"] --> rlSim["Parallel simulation<br/>4096 robots, randomized physics"]
  rlSim -- "observations, rewards" --> rlPPO2["PPO update<br/>actor + critic"]
  rlPPO2 -- "new policy weights" --> rlSim
  rlPPO2 -- "after convergence" --> rlExp["Export policy<br/>ONNX / TorchScript"]
  rlExp --> rlRob["Robot<br/>zero-shot deployment"]

Reward design for locomotion

A locomotion reward is a weighted sum of many terms: a few that define the task and many that regularize the motion. Without the regularizers, the policy usually finds an odd gait that works in simulation and breaks hardware.

Term Form Purpose
Linear velocity tracking $\exp(-\lVert \mathbf{v}_{xy}^\text{cmd} - \mathbf{v}_{xy} \rVert^2 / \sigma)$ The task: move at the commanded speed
Yaw rate tracking $\exp(-(\omega_z^\text{cmd} - \omega_z)^2 / \sigma)$ The task: turn at the commanded rate
Vertical velocity, roll and pitch rate $-v_z^2$, $-\lVert \boldsymbol{\omega}_{xy} \rVert^2$ Keep the body steady
Joint torques $-\lVert \boldsymbol{\tau} \rVert^2$ Energy, actuator heating
Joint accelerations $-\lVert \ddot{\mathbf{q}} \rVert^2$ Smooth motion
Action rate $-\lVert \mathbf{a}_t - \mathbf{a}_{t-1} \rVert^2$ No jitter, which real actuators cannot follow
Feet air time positive for steps of a reasonable length Proper stepping instead of shuffling
Undesired contacts $-1$ per knee or base contact Walk on feet only
Termination large negative on fall Do not fall

The tracking terms use $\exp(-\text{error}^2)$ rather than $-\text{error}^2$ so that they are bounded and positive, which keeps the reward from being dominated by large errors early in training. Tuning the weights is iterative: train, watch the gait, find what looks wrong, add or reweight a term.

Closing the sim-to-real gap

A policy that walks in simulation can fail on the robot, because simulation differs from reality in ways the policy learned to rely on. The main tools:

Teacher-student training

In simulation we know everything: exact friction, contact states, the terrain without noise. On the robot we know much less. Teacher-student training (also called privileged learning, or asymmetric actor-critic when only the critic gets the extra data) uses this:

flowchart TB
  subgraph rlP1["Phase 1: teacher, RL"]
    rlPriv["Privileged state<br/>friction, contacts, true terrain"] --> rlT["Teacher policy<br/>trained with PPO"]
  end
  subgraph rlP2["Phase 2: student, supervised"]
    rlObs["Robot observations<br/>noisy, with history"] --> rlS["Student policy<br/>recurrent or history encoder"]
    rlS --> rlL["Imitation loss"]
    rlL -. "gradient" .-> rlS
  end
  rlT -- "target actions and latents" --> rlL
  rlS --> rlDep["Deployed on robot"]
  1. Train a teacher with RL using privileged information. With perfect knowledge, learning is fast and the resulting gait is good.
  2. Train a student that sees only what the real robot sees, including a history of observations, to reproduce the teacher’s actions (and often its internal latent representation of the environment). This is supervised learning, which is much easier than RL from noisy observations.
  3. Deploy the student. Its memory has learned to infer friction, slippage and terrain properties from the recent motion, because that was necessary to imitate the teacher.

This recipe produced some of the best-known results on ANYmal: blind locomotion over rough natural terrain, and perceptive locomotion that combines the elevation map with proprioception and falls back on proprioception when the map is wrong, for example in tall grass or snow.

Beyond locomotion

The same pipeline, with different rewards and observations, is used for:

Practical pitfalls

Summary

References