Gemini 4 Argon 将物理推理接入机器人开发

Gemini 4 Argon 在物理理解、代码生成和视觉语义推理方面实现了跨越式提升，为机器人技术栈的各个环节带来了新的工作流程，并加速了迈向物理 AGI 的进程。

这种能力上的飞跃体现在具身推理（ERQA）基准测试中。该基准测试考察模型理解并解决三维环境中物理与空间问题的能力。仅凭推理，Argon 就取得了 87% 的得分，超过了其他所有前沿模型。

作为通用物理推理模型，Argon 可以应用于机器人的整个生命周期。在本文中，我们将分享 Argon 在四个关键领域的代表性提示词：

机器人控制 → 借助 MuJoCo 练习实现移动操作、零样本机器人控制，以及对视频世界模型的闭环引导。

Real2Sim → 从机器人摄像头画面、第一人称人类视频或纸上的手绘草图中，重建经过物理校准的仿真孪生体。

物理理解 → 由智能体执行的多模态事后标注，包括对灵巧操作的细致理解、稠密价值函数和进度估计。

研究与自我改进 → 通过仿真改进行为，并在硬件上迭代实现自我改进。

借助 Argon 实现机器人控制

1. Spot 移动操作：通过 MuJoCo 练习将物品放入垃圾桶

Argon 控制一台 Boston Dynamics Spot® 四足机器人（腿部 12 个自由度 + 机械臂 6 个自由度），学习一项新技能：踩下脚踏式垃圾桶的踏板，将汽水罐扔进去。Argon 为脚踏式垃圾桶创建了一个带关节机构的 MuJoCo 数字孪生体，先在其中练习动作，再踩下真实的踏板，把汽水罐放进打开的垃圾桶。

抱歉，你的浏览器不支持播放此视频。

There's a Diet Coke can behind you and another on the black table on your left.
Use the spot-mobile-manipulation skill to pick them up and trash them in the black foot-pedal bin.
Before stepping down on the pedal, verify in the front cameras that your front-right foot target is centered on the pedal, and use a 2-agent verification swarm (foot-verifier and hand-verifier) to verify both the foot-pedal step and the 3D hand-in-bin alignment before releasing.

Argon 控制一条简单的机械臂，将倒下的马克杯扶正，并把揉皱的餐巾纸放进杯子里。Argon 借助视觉坐标辅助框架和动作原语进行推理，经过多轮操作完成这项任务。

抱歉，你的浏览器不支持播放此视频。

[Attached: wrist_cam.png, obs.json]

Task: Get the robot arm to reorient the mug upright and put the napkin inside the mug.

## Execution Discipline
- Simply output 1 sentence observation followed immediately by your `curl` command (with `# Plan:` on Turn 1 and `# Thought:` on every turn). Everything needed is in `obs.json` and the calibrated reticle/grid drawn on `wrist_cam.png`.

## Visual Coordinate Reference (Annotated on wrist_cam.png)
- `+X` (UP in image): Away from robot base | `-X` (DOWN): Towards robot base
- `+Y` (LEFT in image): World left         | `-Y` (RIGHT): World right
- `+Z`: Height above table                | `Yellow numbers`: World (X, Y) in centimetres (2 cm grid at `grid_z`)
- `Cyan + crosshair`: Current 3D pinch center
- `Cyan [ + ] brackets`: Current jaw opening and closing line
- `Green line`: Current object long-axis orientation (`yaw_deg`: 0° = vertical along +X, 90° = horizontal along +Y)
- `pitch_deg = 90°`: Straight down (-Z) top-down grasp/place; `pitch_deg < 60°`: Tilted coaxial approach along finger axis

## Closed-Loop Turn Workflow
1. Inspect current state & view `wrist_cam.png`:
   `curl -s http://127.0.0.1:8081/obs` (returns `ee_pos`, `holding`, `status`, `workspace`, `wrist_cam_path`).
2. Select VLM call granularity upfront on Turn 1:
   - Coarse (2–4 calls): `look_at` -> `grasp` -> `place` -> `done`
   - Pre-Grasp Hover Verify (4–7 calls): `look_at` -> `goto(z=0.04)` (verify cyan `[ + ]` brackets straddle target) -> `grasp` -> `place_until_contact` -> `done`
   - Fine Visual Servoing (8–20 calls): `look_at` -> `grasp` -> pre-pose `goto` -> closed-loop `move_by` / `move_until_contact` -> `done`
3. Send motion primitives (`POST /plan`) and verify visually after every turn in the updated `wrist_cam.png`. If `status == "grasp_failed"`, call `robot.look_at(...)` to re-observe and recover closed-loop.

## Code-as-Policies Primitives (`POST http://127.0.0.1:8081/plan`)
robot.look_at(x=0.35, y=0.13, z=0.14, grid_z=0.015)
x, y = robot.deproject(u=0.52, v=0.48, z_plane=0.015)
robot.grasp(x=x, y=y, z=0.015, yaw_deg=-35, pitch_deg=90)
robot.place(x=0.24, y=0.06, z=0.075)
robot.place_until_contact(x=0.24, y=0.06, z=0.015, max_lag=0.012, relief=0.002)
robot.goto(x=0.28, y=0.05, z=0.04, arc_z=0.14, pitch_deg=25, azimuth_deg=90, roll_deg=0, speed=0.10)
robot.move_by(forward=0.025, dz=-0.002, dpitch_deg=-3, droll_deg=5, speed=0.04)
robot.move_until_contact(dz=-0.06, speed=0.03, max_lag=0.012, relief=0.002)
robot.open_gripper() / robot.close_gripper() / robot.home() / robot.done()

Argon 引导 Gemini Omni World Model 中的机器人，将五块散落的 Scrabble 字母块（G、R、A、N、O）重新排列成整齐的一行，拼出 A - R - G - O - N。Argon 并非只给出一条静态提示词，而是在视觉闭环中持续引导。它编写带时间戳的动作提示词，检查生成过程中的中间关键帧，并不断调整提示词。

抱歉，你的浏览器不支持播放此视频。

Goal: {task_goal}

You are steering a video world model (Gemini Omni) in an iterative visual feedback loop. Inspect the attached frames from the latest rollout and write the next timestamped action prompt for the video world model:
- Break complex motions into one simple, single-direction action at a time.
- Keep inactive hands retracted and avoid blocking the camera's view of objects in the workspace.
- Explicitly state which areas of the workspace become empty after an object moves.
- Use [MM:SS.mmm] timestamps for each action beat and wrap your output in <omni_prompt> tags.

Example format (sample timestamps only, not the full prompt):
<omni_prompt>
[00:00.000] Character A's left hand rests at the side while the right hand reaches toward the target object.
[00:02.500] Character A's right fingertips push the object smoothly into the center of the table, leaving its starting area empty.
[00:05.000] Character A's right hand opens and retracts to the side so all placed objects remain in clear view of the camera.
</omni_prompt>

借助 Argon 实现 Real2Sim

1. 从机器人视角到仿真

Argon 观看一段真实机器人的遥操作记录，并构建数字孪生体；它获取或创建仿真资产，使其外观与真实物体一致，并在与灵巧手接触时表现出与真实物体相同的行为。Argon 在迭代式视觉推理闭环中调整资产和场景，使仿真中的视觉效果与接触事件和真实记录数据中的对应内容相匹配。

抱歉，你的浏览器不支持播放此视频。

[Attached: robot_rollout_episode]

Build a MuJoCo scene of the robot and reconstructed task objects. When each episode is replayed frame by frame, by driving the robot joints toward the logged observation.state with joint-space PD.

Iterate using two verification objectives:
(1) Visual match: MuJoCo renders from the head_left, left_wrist, and right_wrist cameras line up with, and look like, those real observations.
(2) Physical match: the simulated contact force/torque on each Sharpa fingertip, in the sensor frame, matches the logged tactile_force in timing, magnitude and direction.

What you can tune happens before simulation starts:
(1) Static object assets: realistic geometry, collision shapes, mass, inertia, friction and joints, with appearance close to the real objects, plus the initial object poses at t = 0.
(2) Offline-planned robot commands, changed only in frames where the real fingertips are near an object. Each command must stay within 0.55 rad (arm joints) and 0.65 rad (hand joints) of the logged teleop trajectory. Everywhere else, replay the logged trajectory unchanged.

No sim cheating: task objects may move only through real MuJoCo contact with the robot hands and the environment (mj_step).
Not allowed after t = 0: changing MjModel parameters; applying external forces or torques to objects; welds or equality constraints; feedforward torques beyond joint-space PD; setting or rolling back object qpos/qvel; disabling or excluding collisions; reporting contact forces that don't come from the MuJoCo solver (for example, copying the logged tactile readings).

立即将重建场景和轨迹执行结果呈现为交互式 HTML 会话查看器，每个回合对应一个页面，展示：

• 底部：同步时间轴（播放、拖动定位、速度控制），用于驱动所有面板。
• 左侧面板：交互式 3D Blender 场景，展示仿真机器人和物体，并随着回放推进显示接触点和力箭头。
• 右侧面板：全部 3 台摄像机的真实画面与仿真画面并排对比，以及真实与仿真的手指受力可视化。

### 2. 从人类视角到仿真

给定一段操作任务的第一人称人类视频，Argon 会识别与任务相关的物体，用经过凸分解的复合网格替换凹形容器的粗略包围盒，并在 MuJoCo 接触物理仿真中，将人类手部轨迹重定向到一个具有 76 个自由度的双臂人形机器人上，其五指手各有 22 个自由度。

抱歉，你的浏览器不支持播放此视频

```plaintext
[Attached: egocentric_human_video.mp4, candidate_sim_views.png, hand_3d_keypoints.json]

You are building a physically solvable 500 Hz MuJoCo manipulation environment (`scene.xml`) and studio Blender render that faithfully reproduces the tabletop task shown in `egocentric_human_video.mp4` on a bimanual humanoid robot with 22-DoF dexterous five-fingered hands:

1. Scene Reconstruction & Camera Calibration:
   - Identify only the manipulation-relevant objects (the grasped target object and the receiving container or landmark) and triangulate their 3D tabletop coordinates from the egocentric camera intrinsics/extrinsics.
   - Replace crude bounding boxes with compound convex-decomposed 3D meshes so concave containers (nested bowls, recessed wooden trays, open bins) support physical interior placement.

2. 500 Hz Dexterous Motion-Fitting Stress Test:
   - Retarget the 3D human wrist and finger keypoints to the humanoid's 7-DoF arm and 22-DoF hand joints, and step the trajectory in MuJoCo at 500 Hz (dt = 0.002s) with zero kinematic weld constraints.
   - Verify opposing thumb-and-finger force closure, clean object lift (> 2 cm), and stable final placement inside or beside the target landmark.

3. 从草图到仿真

借助 Argon，开发者可以拍下一张粗略的纸笔草图：图中，一个 Apollo 2 人形机器人沿抛物线轨迹投出一颗球，击倒桌上按 3-2-1 堆叠的箱子。随后，Argon 会自动构建 GPU 加速的 mujoco_warp 场景，合成保持平衡的 76 自由度全身及五指投掷轨迹，在闭环物理仿真中验证 6 个目标全部被击倒，并导出 Blender 4.5 Cycles 渲染结果。

抱歉，你的浏览器不支持播放此视频

[Attached: hand_drawn_throw_sketch.jpg]

Translate this hand-drawn sketch into a physically verified humanoid simulation in `mujoco_warp` and a raytraced Blender Cycles video:

1. Scene & Kinematic Setup from Sketch:
   - Parse the spatial layout from `hand_drawn_throw_sketch.jpg`: place the 76-DoF humanoid (32-DoF body equipped with two 22-DoF five-finger dexterous hands) at the origin facing a table holding a 3-2-1 pyramid of 6 rigid target boxes, with a projectile sphere grasped in the right five-finger hand.

2. Whole-Body Balance & Dexterous Throw Synthesis:
   - Synthesize a 50 Hz whole-body throwing trajectory (140 steps, 2.80 s) across four phases: (1) stable stance & 5-finger power grasp closure, (2) torso/shoulder wind-up with ankle-hip Center-of-Mass compensation, (3) forward whipping acceleration, and (4) synchronized 22-DoF five-finger opening at peak hand velocity so the ball launches on a ballistic arc into the pyramid's second-tier center of mass.
   - Verify in `mujoco_warp` contact physics that the humanoid maintains upright balance (< 3 cm pelvis drift) and knocks down all 6 target boxes.

借助 Argon 解锁物理理解能力

1. 事后标注与数据质量

Argon 对灵巧操作具有精确的物理理解能力，能够根据视频提供细致的分段和文本标注。Argon 可以识别质量较差的操作片段，例如停顿、不稳定的抓握和操作失误。给定一段俯视视频，其中双臂 ALOHA 2 机器人在 136 秒内将一张正方形纸折成折纸狐狸，Argon 会生成与时间对齐的四层事后标注体系：回合目标（G1）、6 个连续的语义子任务（G2）、17 个接触层面的原子动作（G3），以及自动质量控制（QC）权重调整，在操作失误和重试期间保持任务进度不变。

抱歉，你的浏览器不支持播放此视频

[Attached: Aloha2_Origami_20250309.mp4]

Produce a time-aligned multi-granularity hindsight annotation JSON with four levels:
1. G1_episode_instruction: Concise imperative sentence describing the overall manipulation goal.
2. G2_subtasks: Contiguous semantic subtask phases specifying which arm(s) anchor vs. fold/rotate the object.
3. G3_atomic_primitives: Fine-grained contact-level primitives (3–15s) aligned with grasp closure, crease sweeps, release, or retry attempts.
4. quality_control_and_reweighting: Hold task_progress_pct flat during non-progressing slip/delamination retry loops and downweight them (0.15x–0.25x) for Behavior Cloning.

为了用机器人重现经典的 12 球天平谜题，Argon 构建了一个双臂 Franka FR3 MuJoCo 环境，其中包含 12 颗外观相同的台球（一颗球被暗中设为更重或更轻）和一台带关节机构的 Roberval 双盘天平。随后，它编写了一个仅依赖视觉的闭环求解器，通过双目 RGB 摄像机对球进行三角定位，在每次称量后通过视觉读取天平的倾斜状态，并在 3 次称量内找出异常球，将其放入重球（红色）或轻球（蓝色）托盘。

抱歉，你的浏览器不支持播放此视频

# Prompt 1 — Physics-Accurate MuJoCo Environment & Benchmark Generation
Make a MuJoCo environment where a dual-arm robot (two 7-DoF Franka FR3 arms with parallel-jaw grippers on a shared tabletop) is tasked to solve the classic 12-ball balance scale weight puzzle from https://www.mathsisfun.com/pool_balls_solution.html.
1. Create 12 visually indistinguishable pool balls in a 2x6 starting rack (11 standard 150 g, 1 odd ball either 270 g or 30 g).
2. Build a functional two-pan Roberval balance scale at the center of the table with parallelogram equality constraints (`mjEQ_JOINT`) so the Left (Gold) and Right (Teal) pans stay horizontal, plus Red (Heavy) and Blue (Light) submission trays.
3. Track distinct settled weighings (`weighing_count`) and reward `1.00` for isolating the true odd ball in the correct Heavy/Light tray within <= 3 weighings.

# Prompt 2 — Closed-Loop Vision-Only Solver
Implement an independent closed-loop policy relying strictly on stereo RGB camera inputs (`head_camera_left_rgb_image`, `head_camera_right_rgb_image`) and robot proprioception (`add_privileged_task_observations=False`):
1. Segment the 12 balls, Gold/Teal pans, and Red/Blue trays in both stereo views and triangulate their 3D coordinates.
2. Visually read the balance scale tilt after each weighing from stereo vertical pixel disparity (+1 = left heavier, -1 = right heavier, 0 = balanced).
3. Rotate the gripper wrist yaw by 90 degrees when grasping/placing inside the narrow scale pans to clear the raised rims, and isolate the odd ball into the Red or Blue tray within <= 3 weighings.

Argon 可作为机器人任务的通用价值函数。在对人类 UMI 数据集中的一段 60 秒双手圆环分类操作回合进行评估时，Argon 会围绕接触状态转换，反复检查同步的头部、左夹爪和右夹爪摄像机画面，生成密集的价值函数／任务进度预测，并将轨迹分解为 28 个片段，分别具有正、零和负的进度斜率。

抱歉，你的浏览器不支持播放此视频

[Attached: umi_head.mp4, umi_left_gripper.mp4, umi_right_gripper.mp4]

Look at 3 videos of human wearing a gripper performing activities from head, left, right gripper views. Look at the videos thoroughly and give a detailed plot of progress at the task (with positive, neutral, negative slope), identifying important segments in the video and labeling the activities. Look at individual frames or subsegments of the video if necessary to get a better understanding.

借助 Argon 解锁研究与自我改进能力

1. 通过仿真改进行动

Argon 可以在仿真中生成新数据，以提升通用视觉语言动作（VLA）模型的性能。仅从一张真实桌面的 RGB 照片出发，Argon 就能自主构建经过标定的 MuJoCo 仿真孪生环境，定义新任务，并合成闭环运动生成器，在仿真中采集超过 4,500 个经过域随机化的回合，使真实机器人在现实环境中的成功率从 56% 提升至 75%，并将完成时间从 35 秒缩短至 24 秒。

抱歉，你的浏览器不支持播放此视频

[Attached: real_tabletop_photo.jpg]

You are given a single overhead RGB camera photo (`real_tabletop_photo.jpg`) of a bimanual robot workspace (two 7-DoF Franka FR3 arms with parallel-jaw grippers and 4 RGB cameras) facing a table arranged with photobooth party props.
1. Reconstruct procedural 3D meshes and compound MuJoCo collision primitives for all 9 party props matching their real-world dimensions, colors, and resting poses, including spring-hinged deformable crown panels for the soft plush cap.
2. Define a 77-task single-arm and bimanual curriculum (pick-and-place, uprighting, directional pushing, mid-air handovers, bimanual scoop-and-push, and pipelined table clearing).
3. Synthesize a closed-loop IK trajectory generator across ±10 cm pose randomization, verify `reward == 1.0` in MuJoCo physics, and export 4,500+ episodes for VLA fine-tuning.

2. 视觉抓取与放置任务中的自我改进

在一台简单的六自由度机械臂上，Argon 通过反复尝试和自我反思，持续提高抓取与放置任务的执行速度和放置精度。

抱歉，你的浏览器不支持播放此视频。

# Stage 1 — Measure Objects & Dry-Run Plan (No Hardware Motion)
Stage 1: measure objects — find the objects and plan the whole motion. The arm does not move in this stage.

A. I put a LEGO brick and a box on the table in front of the arm, and the wooden box to one side. Find the brick and the wooden box in the image. And also in the real world. Show this in an annotated image.

B. A dry-run plan (no hardware). Write the planner to pick and place the lego block. Print the plan table, draw all poses onto the overhead image, and stop. Do not move the robot.
Report with evidence.

# Stage 2 — Pre-Pick Hover & Wrist-Camera Verification
Stage 2: Put the gripper above the brick without touching it, and verify with the wrist camera that the arm and the vision agree. Please record executions with both cameras (compressed video is OK).

# Stage 3 (Autonomous Pick-and-Place → Self-Improvement → Optimization):
1. Execute autonomously: Turn the Stage 1 dry run into a real grasp, lift, and place into the wooden box. Run it yourself subject to safety rules from AGENTS.md.

2. Self-improve: When an attempt fails, diagnose it yourself. Review your own logs and wrist/overhead footage, find the root cause, check the fix in a dry run, and retry.

3. Optimize on feedback: Once it succeeds, make it at least 2× faster within safety limits (rule R5) unless I authorize more, or give feedback.

3. 用单条提示词在 MuJoCo WebAssembly 中复现机器人学论文

每篇论文只需一条提示词，Argon 就能阅读机器人控制论文及其公开的模型检查点，将物理模型和闭环控制器移植到独立运行的 MuJoCo WebAssembly 浏览器实验环境中，并在无头 Chromium 中进行验证。这些控制器涵盖 DeepLoco 中由 7 个网络组成的分层运动控制系统、HUSKY 中以 50 Hz 运行的 PPO 策略网络，以及 Whole-Body MPC 中通过 Web Worker 运行的 iLQR。

抱歉，你的浏览器不支持播放此视频。

[Attached: paper.pdf, optional author_checkpoints/ or urdf_assets/]

Reproduce the paper "<Paper Title & arXiv ID>" from scratch in MuJoCo WebAssembly and deploy a self-contained browser lab + visual essay:
1. Match the paper's exact physics timestep, actuator gains (kp/kv), torque limits, and kinematic constraints in MuJoCo XML, and run real neural inference or Web Worker iLQR in JavaScript at the paper's exact control rates.
2. Build `index.html` (Spatial UI simulator with camera presets, live telemetry plots, and disturbance injection) and `scrollytelling.html` (interactive visual essay).
3. Run a headless Chromium (Playwright) verification loop asserting 0 console errors and non-zero task progress over 300+ frames.

用 Argon 推进具身智能

Argon 将编程、视觉感知、物理仿真和具身机器人控制连接起来。通过原生地对多视角视频、三维几何、接触力学和本体感知遥测数据进行推理，Argon 不仅能理解场景中有哪些物体，还能理解这些物体如何与智能体发生物理交互，从而完成需要全身协调的灵巧操作任务。Gemini 4 Argon 是加速迈向物理 AGI 的重要里程碑。通过构建这一核心智能，我们希望让机器人能够与人类协同工作，解决复杂挑战。

从今天起，你就可以在 Google 的开发者平台上使用 Gemini 4 Argon 进行开发：

**Google AI Studio：**面向 Google AI Ultra 订阅用户开放。

**Gemini API：**通过 AI Studio 中的 Gemini API 付费层级提供。在 Interactions API 中使用 gemini-4-argon，各个 google-genai SDK 从发布首日起即提供支持。

**Google Antigravity：**面向 Google AI Ultra 订阅用户开放。开始开发时，请在 Google Antigravity 中将 Gemini 4 Argon（gemini-4-argon）选为当前使用的模型，用于复杂规划、/goal 运行和多智能体编排。