How to use
- Copy the installation command and review the source package before installing it in your own agent environment.
- Check the source document for required tools, credentials and supported hosts.
- Describe your input and desired deliverable; review the output against the source material.
npx skills add https://github.com/google-deepmind/robotics-antigravity-plugin --skill fr3-duo-controlBefore you start
- Installation does not include a model subscription, API allowance or permission to access your files.
- Repository stars and installation counts describe activity, not an independent quality or security test.
Rendered from the source repository. Headings, examples, code, tables, links, and referenced images are preserved.
Franka Research 3 Duo Control (fr3_duo)
Operate and program a dual-arm FR3 Duo (left/right 7-DOF FR3 arms, Robotiq 2F-85 grippers, overhead ZED Mini stereo camera, and wrist RealSense D405 RGB cameras) under R1–R16 in safety_rules.md and hardware_limits.md. For offline mode="sim", canonical MjSpec scenes, and live MuJoCo Studio workflows, see mujoco_sim.md.
1. Setup Gate
- Offline sim only: Run
Fr3DuoRobot(cfg, mode="sim")directly (default_robot_config()works with no hardware orconfig/robot_config.json; setMUJOCO_GL=eglorosmesaon headless Linux forrender_sim_cameras). - Missing/incomplete `config/robot_config.json`, or bring-up/homing/diagnostics: Follow bringup_checklist.md via bringup.py (
uv run scripts/bringup.py, e.g.--dry-run,--step geometry,--up-to all,--step home --yes; orchestrates discover_hardware.py, check_cameras.py, check_robot.py). Split operator prompts cleanly across two turns during Stage 1 rather than dumping all checks at once:
- Turn 1 (connection prerequisite -> run `--step preflight --read-only`): Confirm only left/right FR3 FCI IPs (stating robot vs. operator left/right) and that joints are unlocked, FCI is enabled, and physical E-stop is within reach. Let
--step preflightauto-check tilted Desk gravity/payload, USB/RS-485 ports, cameras, and conflicting processes before asking the operator to intervene. - Turn 2 (Stage 1b geometry -> run `--step geometry`): Show/attach fr3_duo_dimensions_reference.png inline to the operator and confirm the 6 measurements in meters (
m):base_height_m,table_height_m->z_table_m,table_front_offset_m,table_width_m,table_depth_m, andtcp_offset_z_m(0.145 mstock 2F-85 silicone pads vs0.185 mextended fingers). Defer the optionaltmuxprompt (R11) until live motion stages.
Before live hardware actuation or downstream skill composition, config/robot_config.json must satisfy validate_robot_config_complete(cfg) == [] (arms.{left,right}.{ip,gripper_port}, all 3 cameras.* entries, and geometry_confirmed=True).
- Hardware already verified (`R2`, complete config): Skip bring-up motion checks; run your task with
Fr3DuoRobot(cfg, mode="hardware", dry_run=True)(orFr3DuoSafetyChecker), then wait for explicit"go"(R1) beforedry_run=False.
2. Canonical Contract, Coordinate Frames & Runtime Control API
- Base frame (`base` / `franka_fr3_duo_world`):
+Xforward across tabletop ([0.15, 0.95] m),+Yrobot-left ([-0.55, +0.55] m; operator-right facing pedestal),+Zup (z_table_mtypically[-0.03, +0.05] m).leftandrightare from the robot's perspective (not the operator's when facing the pedestal)—state both when prompting a human. - Joints, Grippers & Tool Center Point (TCP):
state_16dorder is[left_q1..q7, left_grip, right_q1..q7, right_grip]({side}/fr3_joint1..7). TCP is{side}/robotiq_gripper_{side}/pinch_site(0.145 mdefaulttcp_offset_z_m); gripper0.0 = 85 mm open,1.0 = 0 mm closed(empty pinch stalls at0.82–0.91). - 4 Canonical RGB Streams (`1280x720` USB/UVC):
head_camera_left,head_camera_right(split from2560x720ZED Mini stereo),wrist_camera_left,wrist_camera_right({side}/wrist_camera/wrist_camera_{side}D405). - Python / `uv` Environment (`pylibfranka`):
pylibfrankaships pre-built Linux x8664 wheels for Python `3.9–3.12` (`requires-python = ">=3.9,<3.13"` in [bringup.py](scripts/bringup.py)). Run `uv run scripts/bringup.py ...` directly, or pass `uv run --python 3.12 <taskscript>.py/ create.venvwith--python 3.12` for custom task scripts. - Architecture (`Fr3DuoRobot` -> `Fr3DuoRealBackend` / `Fr3DuoSimBackend` -> Hardware Controllers + Services):
- Robot & Backends:
- fr3_duo_robot.py:
Fr3DuoRobot(cfg, mode="sim"|"hardware", dry_run=False)context manager validating moves viaFr3DuoSafetyCheckerand dispatching toFr3DuoRealBackendorFr3DuoSimBackend(read_state,recover_errors,render_sim_cameras,activate_grippers,move_gripper,move_arm_waypoints,move_bimanual_waypoints).Fr3DuoRealBackendreadsstate.current_errorsand attemptspylibfranka.Robot.automatic_error_recovery()on recoverable errors/reflexes; extendfr3_arm_controller.py+fr3_duo_robot.pywhen addingpylibfrankaor real-backend features.
- Hardware Controllers:
- fr3_arm_controller.py: Core
pylibfrankawrapper (ArmWorkerProcessisolated 1 kHz FCI worker,read_pylibfranka_state,recover_pylibfranka_errors,audit_arm_desk_config,validate_arm_readiness,probe_arm_readonly,read_arm_joints_or_rest). - robotiq_gripper_controller.py:
RobotiqGripper,RobotiqStatus,resolve_serial_ports,select_gripper_ports. - fr3_camera_controller.py:
Fr3DuoCameraController(freshness<= 0.30 s, side-by-side ZED stereo split),render_contact_sheet,discover_uvc_devices,check_linux_usb_permissions.
- Services / Shared Utilities:
- fr3_config.py:
RobotConfig,ArmConfig,ArmsConfig,CamerasConfig,DEFAULT_CONFIG_PATH,default_robot_config,load_robot_config,save_robot_config,normalize_robot_config,table_box_bounds,validate_robot_config_complete,detect_preempt_rt,resolve_realtime_config. - fr3_mjcf.py:
build_fr3_duo_spec(cfg)&build_fr3_duo_mjcf(cfg)(single canonical MenagerieMjSpecbuilder + sparse GitHub cache),DEFAULT_REST_POSE_RAD. - fr3_kinematics.py:
compute_arm_fk,compute_arm_pinch_from_flange,estimate_external_wrench_base,get_camera_calibration_opencv,project_base_points_opencv. - fr3_safety.py:
FR3_HW_MAX_VEL_RAD_S,FR3_HW_MAX_ACC_RAD_S2(7.5J1–J2,9.5J3–J7),FR3_HW_MAX_JERK_RAD_S3(3500/4500),MIN_INTER_ARM_CLEARANCE_M(0.030),generate_quintic_trajectory,validate_joint_waypoints,AbortWatcher,Fr3DuoSafetyChecker.
3. Composition Map (rgb-object-localization -> motion-planning -> fr3_duo_robot -> visual-servoing / iteration)
- Perception & planning (`rgb-object-localization`, `motion-planning`, `visual-servoing`): Pass
get_camera_calibration_opencv(...)(K,T_base_cam) toray_plane_contour_pose,build_fr3_duo_mjcf(cfg)tomp.RobotArmPlanner(..., arm="left"|"right")(add_other_arm_to_scene), andwrist_camera_{side}(intrinsics_tuple,T_ee_cam) toibvs.Ibvs. - Iteration & Studio / Sim (`iteration`, `mujoco-studio-sidecar`, `sim2real-gauntlet`): Stage scripts with 4-cam MP4 capture; build interactive Studio scenes via
fr3_mjcf.build_fr3_duo_spec(mujoco_sim.md) or grade offline.
Import sibling robotics scripts relative to skills/fr3-duo-control/ (../rgb-object-localization/scripts, ../motion-planning/scripts, ../visual-servoing/scripts; include pillow and scipy via --with pillow --with scipy or .venv when composing motion-planning):
from pathlib import Path
import sys
import tempfile
_SKILLS_DIR = Path(__file__).resolve().parents[2]
for rel in ("fr3-duo-control/scripts", "motion-planning/scripts", "rgb-object-localization/scripts", "visual-servoing/scripts"):
sys.path.insert(0, str(_SKILLS_DIR / rel))
from fr3_config import load_robot_config
from fr3_mjcf import build_fr3_duo_mjcf
from fr3_kinematics import get_camera_calibration_opencv
from fr3_duo_robot import Fr3DuoRobot
from fr3_safety import clamp_q_to_padded_limits
import motion_planner as mp
import rgb_localization
import ibvs
cfg = load_robot_config("config/robot_config.json")
mj_model, mjcf_xml = build_fr3_duo_mjcf(cfg)
tmp_xml = Path(tempfile.mkdtemp(prefix="fr3_mp_")) / "fr3_duo.xml"
tmp_xml.write_text(mjcf_xml, encoding="utf-8")
left_planner = mp.RobotArmPlanner(mjcf_path=str(tmp_xml), arm="left", gripper_cmd=0.0)
scene = mp.Scene3D()
table_center, table_half_extents = cfg.table_box_bounds()
scene.add_aligned_box("table", center=table_center, half_extents=table_half_extents, group="table")
with Fr3DuoRobot(cfg, mode="sim") as robot: # pass dry_run=True to validate real moves offline first
state = robot.read_state()
left_planner.add_other_arm_to_scene(scene, other_arm_q=state["right"]["q_rad"], other_gripper_cmd=state["right"]["gripper_ratio"])
cal = get_camera_calibration_opencv(mj_model, state["left"]["q_rad"], state["right"]["q_rad"])
# 1. Localize target (OpenCV T_base_cam) or triangulate (MuJoCo mj_pose for mp.*)
pose_3d = rgb_localization.ray_plane_contour_pose(mask_u8, cal["head_camera_left"]["K"], cal["head_camera_left"]["T_base_cam"], z_plane_m=float(cfg.z_table_m))
table_pt = mp.intersect_ray_plane(640.0, 360.0, cal["head_camera_left"]["mj_pose"], cal["head_camera_left"]["mp_intrinsics"], plane_z=float(cfg.z_table_m))
# 2. Plan & execute sparse waypoints (clamping J4 safety margin to [−2.9421, −0.2518])
plan = left_planner.plan_arm(scene, state["left"]["q_rad"], q_pregrasp)
safe_wps = [clamp_q_to_padded_limits(q).tolist() for q in plan["waypoints"]]
robot.move_arm_waypoints("left", safe_wps)
# 3. Close-range eye-in-hand IBVS alignment before grasp
ibvs_left = ibvs.Ibvs(*cal["wrist_camera_left"]["intrinsics_tuple"])
step = ibvs_left.compute_step(u, v, z_depth, u_goal, v_goal, cal["wrist_camera_left"]["T_ee_cam"])