Published Jul 20, 2025 - Author: Guoxiang (Shawn) Zhou Technology
Supervised learning needs a label for every input: this image is a cat, this point is ground. For many robotics problems there is no such label. Nobody can write down the correct torque for each of a quadruped’s 12 joints at every millisecond while it climbs stairs. What we can say is whether the outcome was good: the robot moved at the commanded speed, did not fall, and did not waste energy.
Reinforcement learning (RL) learns from exactly that kind of signal. An agent tries actions, observes what happens, receives a scalar reward, and gradually changes its behavior to collect more reward over time. No one tells it which action was correct; it has to find out by trial and error, and it has to figure out which of its many earlier actions deserve credit for a reward that arrives later.
This post covers:
Everything in RL is built on one loop. At each time step $t$ the agent observes the state $s_t$ of the environment, picks an action $a_t$, and the environment responds with a new state $s_{t+1}$ and a reward $r_t$.
flowchart LR
rlA["Agent<br/>policy π(a | s)"] -- "action aₜ" --> rlE["Environment<br/>robot + world"]
rlE -- "next state sₜ₊₁" --> rlA
rlE -- "reward rₜ" --> rlA
For a walking robot:
| Term | Example for a quadruped |
|---|---|
| Agent | The neural network controller |
| Environment | Robot body, actuators, terrain, gravity, contacts |
| State $s_t$ | Base orientation and velocity, joint angles and speeds, terrain around the feet |
| Action $a_t$ | 12 joint position targets |
| Reward $r_t$ | + tracking the commanded velocity, - falling, - energy, - jerky motion |
| Episode | One run from a reset until a fall or a time limit, e.g. 20 s |
The loop repeats for an episode. Then the environment resets and a new episode starts. Training is millions of episodes.
The loop is formalized as a Markov decision process (MDP), the tuple $(\mathcal{S}, \mathcal{A}, P, R, \gamma)$:
Markov means the next state depends only on the current state and action, not on the history before it. That is a property of what you put into the state. Joint angles alone are not Markov for a walking robot, since they do not tell you how fast the joints are moving; add joint velocities and they are much closer. When the agent sees only part of the true state (a camera does not see the friction coefficient), the problem is a partially observable MDP (POMDP), and the policy usually gets a history of observations or a recurrent memory to compensate. Every real robot is a POMDP.
The policy $\pi$ is the agent’s behavior: a mapping from state to action. It is the thing we are ultimately learning, and the thing that runs on the robot.
In deep RL the policy is a neural network $\pi_\theta$ with weights $\theta$. For locomotion it is often a small multilayer perceptron (three layers of 128 to 512 units), sometimes with a recurrent layer, which runs in well under a millisecond on a CPU.
The reward $r_t$ is a single number per step that says how good that step was. It is the only way to tell the agent what you want. The reward hypothesis says that any goal can be expressed as maximizing the expected sum of a scalar reward. In practice, writing that scalar so that it produces the behavior you actually want is the hardest part of applied RL.
Rewards are either sparse (+1 when the robot reaches the goal, 0 otherwise) or dense (a small reward every step for moving in the right direction). Sparse rewards are easy to specify but hard to learn from, because a random policy almost never sees a non-zero reward. Dense, shaped rewards learn quickly but invite reward hacking: the agent finds a way to score well that you did not intend, such as vibrating its legs to get a velocity reward without walking.
The agent does not maximize the reward of the next step. It maximizes the return, the discounted sum of all future rewards:
\[G_t = r_t + \gamma r_{t+1} + \gamma^2 r_{t+2} + \dots = \sum_{k=0}^{\infty} \gamma^k r_{t+k} \tag{1}\]The discount $\gamma$ in Eq. (1) does three things:
A value function predicts the return. It is how the agent assigns credit to actions whose payoff comes later.
Values satisfy a recursive relation, the Bellman equation, Eq. (2). The value of a state is the immediate reward plus the discounted value of where you land:
\[V^\pi(s) = \mathbb{E}_{a \sim \pi,\ s' \sim P} \left[ r(s, a) + \gamma V^\pi(s') \right] \tag{2}\]For the optimal policy, the expectation over actions becomes a max:
\[V^*(s) = \max_a \mathbb{E}_{s' \sim P} \left[ r(s, a) + \gamma V^*(s') \right] \tag{3}\]If the dynamics $P$ are known and the state space is small, you can apply Eq. (3) repeatedly to every state until the values stop changing. That is value iteration, and acting greedily with respect to $V^*$ gives the optimal policy.
The figure solves a small grid world this way. The robot starts at S and wants to reach G. Entering G pays a reward of +10 and ends the episode. The bottom row in front of the goal is a cliff: entering it pays -10 and also ends the episode. Both are terminal states, so their value is 0: the +10 is the reward for the last step, and it shows up in the values of the cells next to G, not in G itself. Each step costs a small amount, and with some probability the robot slips sideways instead of moving where it intended, like a foot sliding on a wet surface.
Things to try:
The same trade-offs show up on a real robot: the discount sets how far ahead it plans, and the balance between task reward and penalties sets how cautious it is.
A model is the agent’s own prediction of the environment, $\hat{P}(s’ \mid s, a)$ and $\hat{R}(s, a)$. Model-based RL learns or is given a model and plans with it (value iteration above is the extreme case: a perfect model). Model-free RL never builds a model; it learns values or a policy directly from experience. Most robot locomotion work is model-free inside a simulator, which is a model, just not one the agent can query for planning.
To find good actions the agent must try actions it is unsure about (exploration). To collect reward it must use what it already knows (exploitation). Doing only one fails: pure exploitation gets stuck on the first strategy that works at all, pure exploration never uses what it learned. Common mechanisms:
flowchart TB
rlRoot["RL algorithms"] --> rlMF["Model-free"]
rlRoot --> rlMB["Model-based<br/>MuZero, Dreamer, MBPO"]
rlMF --> rlVB["Value-based<br/>learn Q, act greedily"]
rlMF --> rlPG["Policy gradient<br/>optimize π directly"]
rlVB --> rlQL["Q-learning, DQN"]
rlPG --> rlRF["REINFORCE"]
rlPG --> rlAC["Actor-critic<br/>policy + value"]
rlAC --> rlPPO["PPO (on-policy)"]
rlAC --> rlSAC["SAC, TD3 (off-policy)"]
Learn $Q(s, a)$ and act with $a = \arg\max_a Q(s, a)$. The update moves $Q$ toward the one-step temporal difference (TD) target:
\[Q(s, a) \leftarrow Q(s, a) + \alpha \left[ r + \gamma \max_{a'} Q(s', a') - Q(s, a) \right] \tag{4}\]The bracket in Eq. (4) is the TD error: what actually happened (one real reward plus the estimated rest) minus what was predicted. DQN (2015) replaced the table with a neural network, plus a replay buffer and a slowly updated target network for stability, and learned to play Atari games from pixels. The weakness for robotics is the $\max_a$: with 12 continuous joint targets there is no finite list of actions to take the max over.
Optimize the policy directly. Write the objective as the expected return $J(\theta) = \mathbb{E}_{\pi_\theta}[G_0]$. The policy gradient theorem gives its gradient as
\[\nabla_\theta J(\theta) = \mathbb{E}_{\pi_\theta} \left[ \sum_t \nabla_\theta \log \pi_\theta(a_t \mid s_t) \, A(s_t, a_t) \right] \tag{5}\]Read Eq. (5) as: for each action taken, increase its log-probability in proportion to how much better than average it turned out. Good actions become more likely, bad ones less likely. This works for continuous actions out of the box. REINFORCE uses the full sampled return in place of $A$, which is unbiased but very noisy.
Use two networks: an actor (the policy) and a critic (a value function $V_\phi(s)$). The critic estimates the advantage $A$ in Eq. (5), which lowers the variance of the policy gradient a lot. The simplest estimate is the TD error, $A_t \approx r_t + \gamma V(s_{t+1}) - V(s_t)$. Generalized Advantage Estimation (GAE) blends TD errors over many steps with a parameter $\lambda$ that trades bias (small $\lambda$) against variance ($\lambda \to 1$).
Two main sub-families:
| On-policy (PPO) | Off-policy (SAC, TD3) | |
|---|---|---|
| Data used for an update | Only fresh rollouts from the current policy | Any past data, from a replay buffer |
| Sample efficiency | Low | High |
| Stability, tuning effort | Robust, few sensitive hyperparameters | More sensitive |
| Scales with parallel simulation | Very well | Less naturally |
| Typical use | Locomotion in simulation | Real-robot learning, manipulation |
With thousands of parallel simulated robots, samples are cheap and stability matters more than sample efficiency. That is why PPO is the default for legged locomotion.
The naive policy gradient has a failure mode: one large update step can wreck a good policy, and since the next batch of data comes from that wrecked policy, it may never recover. Proximal Policy Optimization (PPO) keeps each update close to the policy that collected the data.
Let $r_t(\theta) = \pi_\theta(a_t \mid s_t) / \pi_{\theta_\text{old}}(a_t \mid s_t)$ be the ratio of new to old action probability. PPO maximizes the clipped surrogate objective
\[L^\text{CLIP}(\theta) = \mathbb{E}_t \left[ \min\left( r_t(\theta) A_t,\ \text{clip}(r_t(\theta), 1 - \epsilon, 1 + \epsilon) A_t \right) \right] \tag{6}\]with $\epsilon$ typically 0.2. Read it term by term.
The ratio. $r_t(\theta)$ measures how much the update has changed the probability of the action that was actually taken. $r_t = 1$ means unchanged (at the start of each update, $\theta = \theta_\text{old}$), $r_t = 1.5$ means 50% more likely, $r_t = 0.5$ means half as likely. The first term inside the min, $r_t(\theta) A_t$, is the plain policy gradient objective written with this ratio: its gradient at $r_t = 1$ equals Eq. (5). On its own it has no limit. For a good action ($A_t > 0$) it keeps rewarding a larger $r_t$, so a few gradient steps on the same batch can push the probability far beyond what the data supports.
The clip. $\text{clip}(x, l, u)$ limits a number to the interval $[l, u]$:
\[\text{clip}(r, 1 - \epsilon, 1 + \epsilon) = \begin{cases} 1 - \epsilon & r < 1 - \epsilon \\ r & 1 - \epsilon \le r \le 1 + \epsilon \\ 1 + \epsilon & r > 1 + \epsilon \end{cases} \tag{7}\]With $\epsilon = 0.2$ the ratio is confined to $[0.8, 1.2]$. Outside that interval the clipped term is a constant, so its gradient with respect to $\theta$ is zero: the sample stops asking for more change. If an action had probability 0.5 under the old policy and turned out better than expected, the update gains nothing from pushing it beyond 0.6.
The min. Clipping alone would also flatten the cases where the update went the wrong way, and the policy would never correct them. Taking the min of the clipped and unclipped terms picks whichever is smaller, a pessimistic lower bound on the improvement. The result is that the clip removes only the incentive to move further in the helpful direction, never the penalty for moving in the harmful one:
| Advantage | Ratio | Term chosen by the min | Gradient on the action probability |
|---|---|---|---|
| $A_t > 0$ (good action) | $r_t < 1 - \epsilon$ | unclipped $r_t A_t$ | Up: undo a decrease that went the wrong way |
| $A_t > 0$ | inside $[1 - \epsilon, 1 + \epsilon]$ | both equal | Up |
| $A_t > 0$ | $r_t > 1 + \epsilon$ | clipped $(1 + \epsilon) A_t$ | Zero: already increased enough |
| $A_t < 0$ (bad action) | $r_t < 1 - \epsilon$ | clipped $(1 - \epsilon) A_t$ | Zero: already decreased enough |
| $A_t < 0$ | inside $[1 - \epsilon, 1 + \epsilon]$ | both equal | Down |
| $A_t < 0$ | $r_t > 1 + \epsilon$ | unclipped $r_t A_t$ | Down: undo an increase that went the wrong way |
The figure plots these six cases.
The flat regions of Eq. (6) are the point: once an action’s probability has moved by more than $\epsilon$ in the helpful direction, that sample stops pushing. This allows several epochs of minibatch gradient steps on the same batch of data without the policy running away. The clip is a soft limit, not a hard constraint: other samples can still move the policy, and parameters are shared across states, so implementations also monitor the KL divergence between the old and new policy and stop early or lower the learning rate when it grows too large.
One PPO iteration:
for iteration in range(num_iterations):
# 1. Rollout: run the current policy in N parallel environments for T steps.
batch = collect_rollouts(policy, envs, num_steps=T) # obs, actions, rewards, dones, log_probs
# 2. Advantages with GAE, using the critic's value estimates.
values = critic(batch.obs)
advantages, returns = gae(batch.rewards, values, batch.dones, gamma=0.99, lam=0.95)
# 3. Several epochs of minibatch updates on the same data.
for epoch in range(5):
for mb in batch.minibatches(4):
ratio = exp(policy.log_prob(mb.obs, mb.actions) - mb.log_probs)
surrogate = min(ratio * mb.adv, clip(ratio, 1 - 0.2, 1 + 0.2) * mb.adv)
policy_loss = -mean(surrogate) - entropy_coef * policy.entropy(mb.obs).mean()
value_loss = mean((critic(mb.obs) - mb.returns) ** 2)
optimize(policy_loss + value_coef * value_loss)
With a GPU simulator, $N$ is 4096 robots and $T$ is 24 steps, so each iteration collects about 100,000 transitions in well under a second of wall time.
The classical approach to legged locomotion is model-based: a simplified dynamics model, a contact schedule, and model predictive control (MPC) that optimizes forces over a short horizon. It works well and is interpretable. It struggles where the model is wrong: slippery or deformable ground, unexpected contacts, missed steps, terrain that does not match the map.
RL moves the hard work offline. Instead of solving an optimization on the robot at every step, it trains a policy against millions of situations in simulation, including the ones that break the model. On the robot, the policy is one forward pass of a small network.
| Model-based (MPC, WBC) | Learned policy (RL) | |
|---|---|---|
| Online cost | Optimization every step | One network forward pass |
| Needs | Accurate model, contact schedule, tuning | Simulator, reward design, compute |
| Robustness to unmodeled effects | Limited by the model | Learned from randomized simulation |
| Interpretability, guarantees | Good: constraints are explicit | Weak: behavior is implicit in weights |
| Changing the behavior | Edit costs or constraints | Retrain |
The two are not exclusive. Common hybrids use RL to produce references for a model-based controller, or a model-based controller as a safety filter around a learned policy.
Design choices that matter:
Real-robot training is slow and breaks hardware. Locomotion policies are trained entirely in simulation (Isaac Gym/Isaac Lab, MuJoCo, RaiSim) and deployed zero-shot. GPU simulators step thousands of robots in parallel, which brings training of a basic flat-ground walking policy down to minutes and rough-terrain policies to hours.
flowchart TB
rlCur["Terrain curriculum<br/>harder terrain when succeeding"] --> rlSim["Parallel simulation<br/>4096 robots, randomized physics"]
rlSim -- "observations, rewards" --> rlPPO2["PPO update<br/>actor + critic"]
rlPPO2 -- "new policy weights" --> rlSim
rlPPO2 -- "after convergence" --> rlExp["Export policy<br/>ONNX / TorchScript"]
rlExp --> rlRob["Robot<br/>zero-shot deployment"]
A locomotion reward is a weighted sum of many terms: a few that define the task and many that regularize the motion. Without the regularizers, the policy usually finds an odd gait that works in simulation and breaks hardware.
| Term | Form | Purpose |
|---|---|---|
| Linear velocity tracking | $\exp(-\lVert \mathbf{v}_{xy}^\text{cmd} - \mathbf{v}_{xy} \rVert^2 / \sigma)$ | The task: move at the commanded speed |
| Yaw rate tracking | $\exp(-(\omega_z^\text{cmd} - \omega_z)^2 / \sigma)$ | The task: turn at the commanded rate |
| Vertical velocity, roll and pitch rate | $-v_z^2$, $-\lVert \boldsymbol{\omega}_{xy} \rVert^2$ | Keep the body steady |
| Joint torques | $-\lVert \boldsymbol{\tau} \rVert^2$ | Energy, actuator heating |
| Joint accelerations | $-\lVert \ddot{\mathbf{q}} \rVert^2$ | Smooth motion |
| Action rate | $-\lVert \mathbf{a}_t - \mathbf{a}_{t-1} \rVert^2$ | No jitter, which real actuators cannot follow |
| Feet air time | positive for steps of a reasonable length | Proper stepping instead of shuffling |
| Undesired contacts | $-1$ per knee or base contact | Walk on feet only |
| Termination | large negative on fall | Do not fall |
The tracking terms use $\exp(-\text{error}^2)$ rather than $-\text{error}^2$ so that they are bounded and positive, which keeps the reward from being dominated by large errors early in training. Tuning the weights is iterative: train, watch the gait, find what looks wrong, add or reweight a term.
A policy that walks in simulation can fail on the robot, because simulation differs from reality in ways the policy learned to rely on. The main tools:
In simulation we know everything: exact friction, contact states, the terrain without noise. On the robot we know much less. Teacher-student training (also called privileged learning, or asymmetric actor-critic when only the critic gets the extra data) uses this:
flowchart TB
subgraph rlP1["Phase 1: teacher, RL"]
rlPriv["Privileged state<br/>friction, contacts, true terrain"] --> rlT["Teacher policy<br/>trained with PPO"]
end
subgraph rlP2["Phase 2: student, supervised"]
rlObs["Robot observations<br/>noisy, with history"] --> rlS["Student policy<br/>recurrent or history encoder"]
rlS --> rlL["Imitation loss"]
rlL -. "gradient" .-> rlS
end
rlT -- "target actions and latents" --> rlL
rlS --> rlDep["Deployed on robot"]
This recipe produced some of the best-known results on ANYmal: blind locomotion over rough natural terrain, and perceptive locomotion that combines the elevation map with proprioception and falls back on proprioception when the map is wrong, for example in tall grass or snow.
The same pipeline, with different rewards and observations, is used for: