Published Jul 12, 2026 - Author: Guoxiang (Shawn) Zhou Technology
A robotics simulator answers one question: given the robot’s state, its commands and its environment, what happens next, and what would the sensors report? Everything else, the graphical user interface (GUI), the Robot Operating System (ROS) bridge, the reinforcement learning (RL) wrappers, is built around that loop.
Simulation is used for four different jobs, and they want different tools:
No simulator is best at all four. This post explains what is inside a simulator, walks through the main ones, Gazebo, MuJoCo (Multi-Joint dynamics with Contact), Isaac Sim, Isaac Lab and others, covers the sim-to-real gap and how to close it, and then the reverse direction, real-to-sim. It ends with a decision guide.
flowchart TD
A["Scene description<br/>URDF / SDF / MJCF / USD"] --> B["Physics engine<br/>dynamics, contact, joints"]
C["Controller or policy<br/>torques, PD targets"] --> B
B --> D["World state<br/>poses, velocities, contacts"]
D --> E["Sensor models<br/>IMU, encoders, camera, depth, LiDAR"]
E --> F["Interface<br/>ROS 2 bridge, Python API, Gym env"]
F --> C
Robots and worlds are loaded from description files:
| Format | Origin | Used by | Notes |
|---|---|---|---|
| URDF (Unified Robot Description Format) | ROS | almost everything, as import | Tree of links and joints only, no closed loops, no world description |
| SDF (Simulation Description Format) | Gazebo | Gazebo | Robots and worlds, sensors, plugins, nested models |
| MJCF (MuJoCo’s XML modeling format) | MuJoCo | MuJoCo, MuJoCo XLA (MJX), Brax | Compact Extensible Markup Language (XML), defaults classes, tendons, actuators with explicit gains |
| USD (Universal Scene Description) | Pixar, NVIDIA Omniverse | Isaac Sim, Isaac Lab | Layered scene graph, scales to large photoreal scenes, physics via schemas |
In practice URDF is the interchange format: the robot description lives in the ROS repository, and each simulator imports it into its native format. Every importer makes choices (default joint damping, collision mesh convexification, inertia handling), so check the imported model instead of trusting it.
Articulated rigid-body dynamics in generalized coordinates $q$:
\[M(q)\,\ddot{q} + C(q, \dot{q})\,\dot{q} + g(q) = \tau + J_c(q)^\top \lambda\]$M$ is the mass matrix, $C$ the Coriolis and centrifugal terms, $g$ gravity, $\tau$ the joint torques and $J_c^\top \lambda$ the contact forces mapped into joint space. The equation is the easy part. The hard parts are:
These choices are the main reason the same robot behaves differently in two simulators. A policy that exploits a quirk of one contact model, for example sliding feet that never slip, fails in another one and on hardware.
Proprioceptive sensors (joint encoders, inertial measurement unit (IMU), force/torque) come almost for free from the state. Exteroceptive sensors are rendering problems: a camera needs a rasterizer or ray tracer, a LiDAR needs ray casting against the scene geometry, and a depth camera needs both plus a model of its failure modes. Sensor fidelity varies far more between simulators than physics fidelity does.
Classic simulators step one world on the central processing unit (CPU), roughly in real time. GPU simulators step thousands of copies of the same world in parallel, with the state of all environments kept in GPU tensors so the policy network reads observations without copying them back. For RL this is the difference between days and minutes: Rudin et al. trained ANYmal to walk on rough terrain in about 20 minutes on a single GPU with 4096 parallel environments [3]. The price is flexibility: GPU pipelines prefer identical robots, simple sensors and batched Python application programming interfaces (APIs), not a ROS stack running per environment.
Gazebo is the default simulator of the ROS ecosystem, maintained by Open Robotics (now part of Intrinsic).
gz-sim, gz-physics, gz-sensors, gz-rendering, released as named distributions such as Fortress, Harmonic and Jetty. Each ROS 2 distribution has a recommended pairing, for example Jazzy with Harmonic.gz-physics; the Dynamic Animation and Robotics Toolkit (DART) is the default, Bullet and a trivial physics engine (TPE) are alternatives.ros_gz bridges topics, and gz_ros2_control lets the same ros2_control controllers run in simulation and on hardware.Strengths: runs the real ROS 2 stack unmodified, headless mode for CI, a large catalog of community models and worlds, lightweight enough for a laptop.
Weaknesses: one world per process and roughly real-time speed, so it is not a training engine. Contact dynamics for legged robots and manipulation are adequate for software testing, not for transferring learned policies. Rendering is functional rather than photorealistic.
Use it for testing navigation, mapping, mission logic and launch files end to end, and for regression tests in CI.
MuJoCo was written by Emo Todorov for model-based control research, acquired by DeepMind in 2021 and open-sourced under Apache 2.0 in 2022 [1].
Strengths: accurate and well-documented contact model, easy installation (pip install mujoco), runs anywhere including macOS, a large set of curated robot models in MuJoCo Menagerie.
Weaknesses: no ROS integration out of the box, limited sensor rendering, soft contacts need tuning (solref, solimp) to avoid visible penetration or overly springy feet.
Use it for control research, MPC, contact-rich manipulation studies, quick RL prototypes, and as an independent second simulator for validating policies trained elsewhere.
Isaac Sim is NVIDIA’s robotics simulator built on Omniverse. It is open source on GitHub since version 5.0.
Strengths: the highest sensor fidelity among general-purpose robotics simulators, industrial-scale scenes, synthetic data generation and digital twins in one tool.
Weaknesses: needs an NVIDIA RTX GPU with plenty of video memory (VRAM), a heavy install and long startup times, Linux and Windows only, a steep learning curve (USD, Kit extensions, Omniverse concepts). For a pure ROS software test it is far more machinery than needed.
Use it for perception development, synthetic datasets, sensor placement studies, digital twins of customer sites, and ROS 2 testing where camera or LiDAR realism matters.
Isaac Lab is an open-source robot learning framework on top of Isaac Sim. It merges ETH Zurich’s Orbit [2] and NVIDIA’s earlier Isaac Gym, which is now deprecated.
Strengths: the most complete stack for training locomotion and manipulation policies at scale, with domain randomization and sim-to-real features designed in from the start.
Weaknesses: inherits Isaac Sim’s hardware and installation requirements; the abstraction layers take time to learn; API changes between major versions.
Use it for reinforcement learning and imitation learning of locomotion and manipulation policies that will run on hardware.
Isaac Sim and Isaac Lab are often confused. Isaac Sim is the simulator and application; Isaac Lab is a Python framework that uses Isaac Sim as its engine and adds everything needed for learning. If you are training policies, start in Isaac Lab. If you are building scenes, sensors, digital twins or ROS tests, work in Isaac Sim.
| Simulator | What it is | Typical use |
|---|---|---|
| Drake | Toyota Research Institute’s toolbox: multibody dynamics, hydroelastic contact, optimization, systems framework | Rigorous manipulation research, trajectory optimization, verification |
| PyBullet | Python bindings to Bullet, the RL workhorse of 2017 to 2020 | Legacy code, teaching; largely superseded by MuJoCo |
| Webots | Open-source desktop simulator with many robot models | Education, quick prototypes |
| CoppeliaSim | Scriptable simulator with several physics engines | Education, industrial cell prototyping |
| Genesis | Python-native GPU simulator combining several solvers (rigid bodies, material point method (MPM), smoothed-particle hydrodynamics (SPH), finite element method (FEM)) | Research on multi-physics learning; check headline speed claims against your own workloads |
| RaiSim | Fast rigid-body engine from ETH’s legged robotics group | Early ANYmal RL work [4] |
| ManiSkill / SAPIEN | GPU-parallel manipulation benchmark and simulator | Manipulation learning benchmarks |
| CARLA | Unreal Engine based driving simulator with traffic and sensors | Autonomous driving |
| Gazebo | MuJoCo | Isaac Sim | Isaac Lab | |
|---|---|---|---|---|
| Primary job | ROS software testing | Control research, fast physics | Sensors, synthetic data, digital twins | Robot learning at scale |
| Physics | DART / Bullet | MuJoCo (soft convex contact) | PhysX 5 | PhysX 5 (Newton coming) |
| Parallel environments | 1 per process | 1 per thread on CPU; thousands with MJX / MuJoCo Warp | Few to hundreds | Thousands on one GPU |
| Sensor realism | Medium | Low | High (RTX) | High, but costly at scale |
| ROS 2 | Native (ros_gz) |
Community bridges | Built-in bridge | Not the focus |
| Scene format | SDF | MJCF | USD | USD |
| Hardware | Any laptop | Any laptop | NVIDIA RTX GPU | NVIDIA RTX GPU |
| License | Apache 2.0 | Apache 2.0 | Apache 2.0 (Omniverse parts have own terms) | Berkeley Software Distribution (BSD) 3-Clause |
A controller that works in simulation and fails on hardware has hit the reality gap: the difference between the simulated transition $p_{\text{sim}}(s’ \mid s, a)$ and the real one $p_{\text{real}}(s’ \mid s, a)$, plus the difference between simulated and real observations. Learned policies are especially exposed because optimization finds and exploits every modeling error that helps the reward.
Dynamics
Actuation
Latency and timing
Sensing
Environment
System identification. Measure what you can and fit what you cannot: weigh links, estimate inertias from excitation trajectories, fit friction and motor parameters from logged data. This moves the simulator’s nominal parameters toward reality.
Actuator modeling. When the actuator is the dominant error, learn it. Hwangbo et al. trained an actuator network that maps the history of position errors and velocities to output torque, using data logged on ANYmal, and used it inside the simulator; it was a key enabler of the first RL locomotion policies on that robot [4]. Isaac Lab ships actuator models of this kind.
Domain randomization. Instead of trying to get every parameter right, sample parameters from a range at each episode, so the policy has to work for all of them and the real robot becomes one more sample [5, 6]. Typical randomized quantities: friction, restitution, link masses and center of mass, motor strength, PD gains, latency, observation noise, external pushes, terrain, and for vision, textures, lighting and camera pose. Automatic domain randomization grows the ranges as the policy masters them [7].
Randomization is not free. Too wide a range makes the task harder to learn and produces a conservative policy that is mediocre everywhere. Randomize what you cannot identify, and identify what you can.
Noise, delay and estimator modeling. Add sensor noise, bias and dropouts to observations; delay actions and observations by a random number of steps; and, where possible, feed the policy the same estimated state it will see on hardware rather than ground truth.
Privileged teacher, student policy. Train a teacher policy with access to privileged information that exists only in simulation (true friction, terrain heights, contact states), then train a student that sees only real sensor history to imitate it. The student learns to infer the hidden quantities implicitly. Lee et al. used this to make ANYmal walk over challenging natural terrain [8], and Miki et al. extended it with perception and a learned belief state over noisy elevation maps [9].
Online adaptation. Policies such as Rapid Motor Adaptation (RMA) estimate an extrinsics vector from recent state-action history and adapt at run time without retraining [10].
Regularize toward realistic behavior. Penalize joint torques, action rate, foot slip and high contact forces. These rewards keep the policy away from regions where the simulator is least accurate, such as violent impacts.
Validate across simulators. Before hardware, run the trained policy in a second simulator with a different contact model, for example train in Isaac Lab and evaluate in MuJoCo. A policy that survives sim-to-sim transfer has not overfit one engine’s quirks. It is the cheapest test in the pipeline.
Fine-tune on hardware. With safe exploration and enough automation, a small amount of real-world data can correct residual errors, either by updating the policy or by learning a residual dynamics model.
For learned perception, the gap is mostly appearance and sensor artifacts. The tools are photoreal rendering (Isaac Sim RTX), randomization of textures, lighting and clutter, sensor-specific noise models, and mixing synthetic data with a smaller set of labeled real data. A useful sanity check: train on synthetic only and evaluate on real; the drop tells you how much the synthetic data is worth.
Depth and elevation-map-based policies are easier to transfer than red-green-blue (RGB) camera-based ones, because geometry is easier to simulate than appearance. That is one reason many deployed legged locomotion policies use height maps rather than images.
Sim-to-real asks how to make a simulated result hold in the real world. Real-to-sim asks the reverse: how to build the simulation from the real world, so that the robot is trained or tested in the environment it will actually meet.
flowchart TD
A["Real robot and site<br/>logs, scans, images"] --> B["Reconstruction<br/>geometry, appearance, physics parameters"]
B --> C["Digital twin<br/>in simulation"]
C --> D["Train, test,<br/>find failures"]
D --> E["Deploy on robot"]
E --> A
System identification is real-to-sim for the robot itself: collect excitation trajectories, then fit inertial parameters, friction and actuator models so that simulated trajectories match the recorded ones. Differentiable simulators (MJX, Warp-based engines, Drake) make this an optimization problem over physical parameters with gradients, instead of a manual tuning loop.
Targeted training. RialTo builds a simulated twin of a specific real scene from a phone scan, fine-tunes a policy there with RL, and transfers it back, improving robustness over imitation alone [11]. The general pattern, real-to-sim-to-real, trades generality for performance on the scenes that matter.
Policy evaluation. Real-world evaluation of robot policies is slow and noisy. SimplerEnv showed that simulated twins of real evaluation setups, with visual matching, can rank manipulation policies consistently with real-world results [12], making simulation useful as a cheap evaluation proxy.
Log replay and failure reproduction. A field failure is recorded in a rosbag. Open-loop replay feeds the recorded sensor data back into the software stack to reproduce perception or planning bugs. Closed-loop resimulation goes further: it reconstructs the scene from the recording and lets the robot act differently than it did, so that a fix can be tested in the same place where the failure happened. Neural reconstruction is what makes closed-loop resimulation of camera data feasible, since the rendered views change when the robot deviates from the recorded trajectory.
Site-specific validation. Before deploying a robot at a new facility, a twin built from the site’s scan or CAD model lets you check localization, path coverage, docking positions and sensor visibility.
| Situation | Choice | Why |
|---|---|---|
| Navigation stack regression tests in CI | Gazebo, headless | Runs the real ROS 2 nodes, cheap, no GPU needed |
| Locomotion policy for a quadruped or humanoid | Isaac Lab, validate in MuJoCo | Massive parallelism, actuator models, terrain curricula; second engine catches simulator overfitting |
| MPC or whole-body controller development | MuJoCo, or Drake | Fast, accurate dynamics, analytic derivatives |
| Synthetic training data for a detector | Isaac Sim with Replicator | Photoreal rendering and automatic labels |
| LiDAR placement study for a new robot | Isaac Sim | Configurable ray-traced LiDAR models with real scan patterns |
| Reproducing a field failure at a customer site | Isaac Sim twin from scan or splats, or log replay | Real geometry and appearance |
| Teaching, quick experiments on a laptop | MuJoCo, Webots | Easy install, runs anywhere |
Most robotics teams end up with more than one: Gazebo or Isaac Sim for system-level testing with ROS, Isaac Lab or MuJoCo for learning, and a second engine for cross-validation. That is not duplication. Each simulator has its own systematic errors, and agreement between two of them is better evidence than a perfect score in one.