LEGO stacking, full deployment run at 4× speed. The controller the agent wrote localizes both assemblies from their studs, places the yellow one on the green one, presses it down, and lifts to check the studs interlocked. No model calls during the run. Left, right and wrist cameras.

RAX (Robot Agent eXploration) lets a general-purpose coding agent learn robot tasks by exploring on the real hardware, then compile what worked into executable programs, with no demonstrations, teleoperation or policy training. I brought it up on a UFactory xArm 6 with three RGB-D cameras at Robotics Center.

Protocol

  1. Solve it by hand. The agent drives the arm one tool call at a time, looking at fresh camera images between moves, until the task succeeds once.
  2. Write and repair a controller. It turns what worked into a Python controller, runs it on scenes the operator resets, reads each failure, and patches the code.
  3. Deploy. The controller runs on fresh scenes through the same tools, with no model queried during an episode.

Two tasks: picking a brick into a tray, and stacking one LEGO assembly onto another until the studs fully interlock. Across all 18 deployment runs the arm never misplaced a grasp, hit anything, or dropped the brick. Every failed run was stopped by a safety check the agent had written for itself, most of them stricter than necessary.

The xArm 6 workcell: arm, table with tray and brick, and a fixed RealSense camera on a stand The G2 parallel gripper with the wrist RGB-D camera mounted above the fingers
Workcell. Left: the xArm 6 with the pick-and-place scene and one of the fixed cameras. Right: the G2 gripper and wrist camera.

Porting from Franka to xArm

RAX was built for a Franka FR3 with ZED cameras. I added an xArm backend behind the same five-method hardware interface (tools(), validate(), call(), status(), close()), so the shared robot service, audit log, operator channel and agent workflow carry over unchanged and the Franka backend still works.

  • Motion. The Franka’s impedance controller needed RAX’s own servo loop to remove steady-state error. The xArm is stiff position-control hardware that runs its own trajectories, so that loop is gone; an adapter converts the agent’s meters and quaternions into the SDK’s millimeters and roll/pitch/yaw.
  • Safety. Arming, the workspace box, step limits, timeouts and stop commands carry over. Joint moves are checked at 31 points along the path.
  • Gripper. The G2 driver always reports is_grasped: false, so a grasp can only be confirmed visually. The controller learned to check a grasp by watching the brick move.
  • Cameras and calibration. A RealSense service aligns depth to color, and a ChArUco routine solves for the wrist-camera mount and locates the fixed cameras against the same board.