google-deepmind/robotics-antigravity-plugin

visual-servoing

Aligns an eye-in-hand physical or simulated robot arm end-effector (SO-101, I2RT YAM, or MuJoCo robot grippers) and grasp orientation to sub-millimeter precision using robot IBVS (analytical hover-plane ray alignment or CPU OpenCV feature-matching IBVS).

소스 보기

사용 방법

  1. 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
  2. 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
  3. 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/google-deepmind/robotics-antigravity-plugin --skill visual-servoing

시작 전 확인

  • 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
  • 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

Eye-in-Hand Visual Servoing (`Ibvs`)

Use bundled `Ibvs` (opencv-python numpy, CPU only, no GPU or torch) with calibrated camera intrinsics (fx_px, fy_px, cx_px, cy_px) and hand-eye extrinsics T_ee_cam:

  1. Choose the Visual Frontend by Target Texture:
  • Smooth / Low-Texture Objects: Segment the target foreground mask (passing an optional roi_mask to exclude any robot/gripper geoms visible in the FOV) and extract pose features via Ibvs.ray_plane_contour_pose or Ibvs.pca_contour_target(..., min_aspect=1.0, roi_mask=roi_mask, prev_theta_rad=...). Use det["anisotropy"] (eigval_max / eigval_min) to decide whether wrist yaw alignment is required vs. keeping neutral wrist roll on symmetric objects.
  • Textured Objects (CPU `match_keypoints`): Call m = match_keypoints(img_cur, img_goal, detector="sift", mask_cur=target_mask) ("orb" is faster; pass masks to exclude gripper fingers). It keeps Lowe-ratio matches with RANSAC homography inliers and returns m["u_px"], m["v_px"] -> m["u_goal_px"], m["v_goal_px"] for compute_step, or None when fewer than min_inliers survive (re-detect the target instead of servoing on bad matches).
  1. Analytical Hover-Plane Alignment (Known Support Plane `z_plane_m`, 2 Steps to `< 1 mm`):
  • Compute the optimal unoccluded hover clearance analytically from T_ee_cam via Ibvs.principal_ray_hover_distance(T_ee_cam, approach_axis_ee)["hover_distance_m"] (clamped to safe workspace clearance h_hover).
  • At hover, compute T_world_cam = T_world_ee(q_actual) @ T_ee_cam from measured joint states and call target = Ibvs.ray_plane_contour_pose(mask, fx_px, fy_px, cx_px, cy_px, T_world_cam, z_plane_m, roi_mask=roi_mask, prev_yaw_rad=...) (which back-projects the contour onto z_plane_m first with arc-length weighting so tilted-camera perspective never biases the 3D midpoint).
  • Command the robot's IK solver to (target["x_m"], target["y_m"]) (aligning jaw yaw to target["yaw_rad"] when target["anisotropy"] >= 2.0) until metric XY error ||(x_ee, y_ee) - (target["x_m"], target["y_m"])|| < pos_tol_m (or pixel error against servo.project_ee_point(h_actual * approach_axis_ee, T_ee_cam) converges).
  1. Full 6-DOF `IBVS` Velocity Step (Unknown Plane or Non-Planar Approach):
  • Pair each frame timestamp t_img with q_actual(t_img), instantiate servo = Ibvs(fx_px, fy_px, cx_px, cy_px) with depth z_depth_m (from aligned RGB-D or two-view triangulation), call step = servo.compute_step(u_px, v_px, z_depth_m, u_goal_px, v_goal_px, T_ee_cam), and integrate T_target = T_world_ee @ Ibvs.se3_exp(step["v_ee"], dt_s) into differential IK until step["converged"].
  1. Loop Protocol:
  • Fresh frame per step: Discard buffered frames before each capture, recompute T_world_cam from q_actual at capture time, and wait for each commanded step to finish before the next capture.
  • Iteration cap: Stop after ~10 hover-plane corrections or ~200 IBVS velocity steps without convergence; re-localize and re-plan instead of continuing.
  • Target lost: If detection returns None or the contour touches the image border, never extrapolate the last target. Retract 3–5 cm along -approach_axis_ee to widen the view and re-detect; if still lost, return to the look pose and re-localize with an object-localization skill.
  • Camera occlusion near the goal: If the gripper, tool, or held object will block the camera before the goal, stop servoing at the last pose where the target is fully visible and finish with a short, task-appropriate open-loop move. If the camera stays clear, servo all the way to the goal.
같은 저장소의 Skills

더 많은 Skills

모든 Skills
google-deepmind
커뮤니티

blender-creation

Creates physically calibrated, articulated, PBR-textured 3D MuJoCo assets (MJCF model.xml, .mjz, mjSpec) from scratch using headless Blender (bpy, bmesh, blendermeshutils, assetbuildutils), convex collision synthesis, 10-scenario auditing, and Studio preview. Use only when asked to create, model, or build a 3D MuJoCo object from a URL, image, or text description, or when blender-creation is named. Don't use for non-MuJoCo Blender/Cycles/CAD modeling.

1시간 설치 수
2
GitHub Stars
80
업데이트
11월 17일
google-deepmind
커뮤니티

fr3-duo-control

Franka Duo robot only (Franka Research 3 / FR3 Duo dual-arm cell with Robotiq 2F-85 grippers, ZED Mini and RealSense D405 cameras). Brings up, homes, and controls it via pylibfranka on hardware or in MuJoCo sim. Use only when the user names Franka Duo, FR3 Duo, or fr3duo.

1시간 설치 수
2
GitHub Stars
80
업데이트
11월 17일
google-deepmind
커뮤니티

i2rt-yam-bringup

Checks and installs the I2RT YAM SDK (i2rt), configures Linux socketcan or macOS gsusb CAN bringup for I2RT YAM robot arms, and provides a wave smoke test (wave.py). Use only before bringing up or programming a physical or simulated I2RT YAM robot arm, or when asked to run a wave gesture on an I2RT YAM robot. Don't use for non-YAM CAN bus debugging (e.g. automotive OBD-II or industrial PLC CAN networks) or non-robotics tasks.

1시간 설치 수
2
GitHub Stars
80
업데이트
11월 17일
google-deepmind
커뮤니티

i2rt-yam-ik

Solves 6-DOF inverse kinematics for the I2RT YAM and YAM Pro robot arm using Mink + DAQP on combined YAM + gripper MJCF. Use only when computing joint targets or local Cartesian moves on an I2RT YAM or YAM Pro robot arm. Don't use for non-YAM inverse kinematics, general quadratic programming (QP) solvers, or non-robotics tasks.

1시간 설치 수
2
GitHub Stars
80
업데이트
11월 17일