google-deepmind/robotics-antigravity-plugin

rgb-object-localization

Localizes metric 3D object coordinates and principal orientation axes of manipulation targets for robot arms (SO-101, I2RT YAM, ALOHA, or MuJoCo tabletop scenes) from RGB-only workspace cameras using gemini-4-hydrogen (or VLM + segmentation fallback), plane…

소스 보기

사용 방법

  1. 설치 명령을 복사하고 자신의 에이전트 환경에 설치하기 전에 소스 패키지를 확인하세요.
  2. 필요한 도구, 인증 정보, 지원 환경은 원본 문서에서 확인하세요.
  3. 입력과 원하는 산출물을 설명하고 출처와 결과를 대조하세요.
npx skills add https://github.com/google-deepmind/robotics-antigravity-plugin --skill rgb-object-localization

시작 전 확인

  • 설치에는 모델 구독, API 한도, 파일 접근 권한이 포함되지 않습니다.
  • 별점 수와 설치 수는 활동 지표이며 독립적인 품질·보안 검증이 아닙니다.
원본 Skill 문서

원본 저장소의 제목, 예시, 코드, 표, 링크, 이미지를 유지해 표시합니다.

RGB-Only 3D Object Localization (`rgb_localization.py`)

Localize metric 3D coordinates and principal axes from RGB-only cameras using bundled `rgb_localization.py` (opencv-python numpy, plus google-genai and kornia torch when applicable):

  1. Camera Verification & 2D Target Detection (`gemini-4-hydrogen` or Agent VLM + Local Mask Fallback):
  • Flush stale OS buffer frames before static captures and visually verify camera stream roles once on startup (never assume OS device index order).
  • With `GEMINI_API_KEY`: Query client.models.generate_content(model="gemini-4-hydrogen", ...) (thinking_level="low", "high" in clutter) for 0–1000 JSON [{"label": ..., "point": [y, x], "box_2d": [ymin, xmin, ymax, xmax]}] (u = x * W / 1000, v = y * H / 1000).
  • Fallback when `GEMINI_API_KEY` is unset: Save one snapshot per camera, inspect once with native VLM (read_file) to confirm camera roles and target appearance, and segment the target foreground mask locally with OpenCV color/contrast thresholding (passing an optional roi_mask to exclude any robot/gripper geoms visible in the FOV).
  1. Metric 3D Mapping (Choose by Camera & Scene Geometry):
  • Calibrated Camera + Known Support Plane (`ray_plane_contour_pose`): Given calibrated intrinsics K, camera pose T_world_cam = T_world_ee(q_actual) @ T_ee_cam, and target plane height z_plane_m (accounting for measured support height and visible object face/centroid height), call ray_plane_contour_pose(mask, K, T_world_cam, z_plane_m, roi_mask=roi_mask). Only align wrist yaw to yaw_rad when anisotropy >= 2.0; keep neutral wrist orientation on symmetric objects.
  • Uncalibrated Static Camera + Planar Workspace (`fit_plane_homography` & `map_homography_xy_and_yaw`): Move the end-effector to >= 4 non-collinear (x_fk, y_fk) points on z_plane_m, detect fingertip motion centroids via cv2.absdiff(img_open, img_closed), fit H = fit_plane_homography(pts_uv, pts_xy), and map target (u_px, v_px) + tangent (vx_px, vy_px) via map_homography_xy_and_yaw.
  • Arbitrary 3D / Unknown-Plane Targets (`triangulate_matched_points`): Capture 2 calibrated views across a sufficient baseline (>= 8 cm) using measured q_actual FK. Match textured crops with kornia >= 0.8 KF.DISK.from_pretrained("depth") + KF.LightGlueMatcher("disk") via K.image.image_to_tensor (KF.SuperPoint was removed in Kornia 0.8) — or pass matched 2D centroids directly for smooth/textureless targets — into triangulate_matched_points(K1, T1, pts1_uv, K2, T2, pts2_uv).
같은 저장소의 Skills

더 많은 Skills

모든 Skills
google-deepmind
커뮤니티

blender-creation

Creates physically calibrated, articulated, PBR-textured 3D MuJoCo assets (MJCF model.xml, .mjz, mjSpec) from scratch using headless Blender (bpy, bmesh, blendermeshutils, assetbuildutils), convex collision synthesis, 10-scenario auditing, and Studio preview. Use only when asked to create, model, or build a 3D MuJoCo object from a URL, image, or text description, or when blender-creation is named. Don't use for non-MuJoCo Blender/Cycles/CAD modeling.

1시간 설치 수
2
GitHub Stars
80
업데이트
11월 17일
google-deepmind
커뮤니티

fr3-duo-control

Franka Duo robot only (Franka Research 3 / FR3 Duo dual-arm cell with Robotiq 2F-85 grippers, ZED Mini and RealSense D405 cameras). Brings up, homes, and controls it via pylibfranka on hardware or in MuJoCo sim. Use only when the user names Franka Duo, FR3 Duo, or fr3duo.

1시간 설치 수
2
GitHub Stars
80
업데이트
11월 17일
google-deepmind
커뮤니티

i2rt-yam-bringup

Checks and installs the I2RT YAM SDK (i2rt), configures Linux socketcan or macOS gsusb CAN bringup for I2RT YAM robot arms, and provides a wave smoke test (wave.py). Use only before bringing up or programming a physical or simulated I2RT YAM robot arm, or when asked to run a wave gesture on an I2RT YAM robot. Don't use for non-YAM CAN bus debugging (e.g. automotive OBD-II or industrial PLC CAN networks) or non-robotics tasks.

1시간 설치 수
2
GitHub Stars
80
업데이트
11월 17일
google-deepmind
커뮤니티

i2rt-yam-ik

Solves 6-DOF inverse kinematics for the I2RT YAM and YAM Pro robot arm using Mink + DAQP on combined YAM + gripper MJCF. Use only when computing joint targets or local Cartesian moves on an I2RT YAM or YAM Pro robot arm. Don't use for non-YAM inverse kinematics, general quadratic programming (QP) solvers, or non-robotics tasks.

1시간 설치 수
2
GitHub Stars
80
업데이트
11월 17일