How to use
- Copy the installation command and review the source package before installing it in your own agent environment.
- Check the source document for required tools, credentials and supported hosts.
- Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/google-deepmind/robotics-antigravity-plugin --skill mujoco-playgroundBefore you start
- Installation does not include a model subscription, API allowance or permission to access your files.
- Repository stars and installation counts describe activity, not an independent quality or security test.
Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.
MuJoCo Playground Sim-to-Real Composition (`playground_sim2real.py`)
Compose live MuJoCo Studio visualization, perception, IK, collision-aware planning, and visual servoing on mujoco_playground's out-of-the-box so101 and yam 50 Hz DeviceLayer stacks with identical code in mode="sim" and mode="real":
- Workspace Setup:
- Clone and sync:
git clone https://github.com/google-deepmind/mujoco_playground.git && cd mujoco_playground && make sync-cpu(ormake sync/make sync-cu12on Linux CUDA; adduv sync --extra so101or--extra yam --default-index https://pypi.org/simpleon robot hosts, and pass--no-synctouv run). - Install composed skill packages into
.venv:uv pip install --python .venv/bin/python --default-index https://pypi.org/simple ruckig opencv-python-headless scipy mink daqp. SetMUJOCO_GL=osmesaonly on headless Linux (omit on macOS).
- Unified `sim` / `real` `DeviceLayer` Execution ([`PlaygroundSim2Real`](scripts/playground_sim2real.py)):
- Wrap
SO101Follower/SO101FollowerDevice(so101_aloha_table, arm inrad, gripper in[0, 100]) orYamFollower/YamFollowerDevice(yam_aloha_table, arm inrad, gripper in[0, 1]) viawith PlaygroundSim2Real(robot="yam", mode="sim", setup_scene=...) as env:, passingsetup_scene(spec)(mujoco-spec-editing/mujoco-python) to customize objects, containers, conveyors, or camera/base poses for any scene. - Never bypass the follower
DeviceLayeror read privileged objectmj_data.xposinside controllers;PlaygroundSim2Realsteps throughfollower/goal_position/*, readsfollower/joint_position/*+ camera streams, and cosine-interpolates safely toFOLLOWER_HOMEonclose().
- Live MuJoCo Studio Visualization & Hot-Reload (`mujoco-studio-sidecar` + `mujoco-studio`):
- Load `mujoco-studio-sidecar` and `mujoco-studio` whenever interactively building scenes, controllers, or Studio UI panels.
mujoco_playground/.venv/bin/pythonalready shipsmujoco.studio:
publish build_playground_spec(robot="yam", setup_scene=...) from a SimPlugin in <workspace_dir>/sim_plugin.py (on_load_model calling self.handle.set_model(spec) and returning True, plus if self in event.plugins: self.handle.reload() on HotReloadEvent), place viewer_plugin.py / messages.py alongside it, launch studiod.py with <playground_dir>/.venv/bin/python (<mujoco-studio-sidecar>/scripts/studiod.py --workspace_dir=<dir>, IsDaemon: true), include [MuJoCo Studio](sidecar://robotics/mujoco_studio/), and verify /_studio/status after edits (or stream PlaygroundSim2Real rollouts to a Studio ViewerHandle via env.attach_studio(handle)).
- Composing `robotics` Plugin Skills Zero-Shot (`sim` -> `real`):
- `rgb-object-localization`: Call
cam = env.camera_opencv("wrist_camera") (computes K, T_world_cam, and T_ee_cam from measured q_arm via mj_kinematics + mj_camlight with MuJoCo-to-OpenCV R @ diag([1, -1, -1]) applied) and pass (mask, cam["K"], cam["T_world_cam"], z_plane_m) to ray_plane_contour_pose or two views to triangulate_matched_points.
- `i2rt-yam-ik` & Arm IK: On
yam, map world targets into the arm base frame (computed dynamically fromenv.mj_modelviaenv.t_world_base) withp_b = env.world_to_base_point(p_w)beforeYamIk(xml_path=env.robot_xml_path).make_topdown_grasp_pose(*p_b)+ik.solve(...)(or transform full poses withenv.world_to_base_pose). On 5-DOFso101(or either arm), callenv.solve_arm_ik(pos_world, approach_dir_world=(0, 0, -1), jaw_yaw_rad=...). - `motion-planning` (`motion_planner` `mp`): Transform world obstacles into the robot MJCF root frame via
env.world_to_base_point/env.world_to_base_pose(tableatz = 0.0in base frame), plan free-space detours and linear Cartesian strokes withplanner = mp.RobotArmPlanner(mjcf_path=env.robot_xml_path, joint_names=env.arm_joint_names, gripper_cmd=env.gripper_normalized)(planner.plan_arm/planner.plan_cartesian_stroke,control_dt_s=env.control_period_s), and stream viaenv.execute_trajectory(res["q_trajectory"]). - `visual-servoing` (`Ibvs`): Compute unoccluded hover height with
Ibvs.principal_ray_hover_distance(cam["T_ee_cam"], env.approach_axis_ee), refine wrist alignment at hover viaIbvs.ray_plane_contour_pose(mask, cam["fx_px"], cam["fy_px"], cam["cx_px"], cam["cy_px"], cam["T_world_cam"], z_plane_m)(or 6-DOFIbvs.compute_step), then descend, close gripper (env.set_gripper(...)), and lift.