phys-0

• One door for every action. The agent, the Lab Console, any MCP client and the CLI all call the same 84 typed tools, so a move made by a person and a move made by a model are the same kind of record
• Worlds you can see. Virtual benches are built in a 3D editor with the real SO-101 meshes, glassware, cameras and Rapier physics, and the agent can build them too
• A camera that measures. On the real bench, OpenCV finds each vial and reads its indicator dye into an estimated pH

Problem

Chem-0 asked whether an agent could run a cheap robot arm on a chemistry bench, and its answer was that the hard parts are not the model: calibration, feedback speed, servo overshoot, and keeping a trustworthy record of what happened. Those are platform problems. Every experiment re-solved them in scripts, and nothing a person did by hand was recorded the same way as what the agent did.

Solution

phys-0 turns Chem-0 into a platform. A single TypeScript backend owns a registry of typed tools: arm control and calibration, cameras, LeRobot recording and policy training, virtual worlds and simulation, assets, experiment records. Everything that can act on the world goes through it, and everything that goes through it is validated and written down.

12345

1/5 Four ways in, one door: the agent, the Lab Console, any MCP client and the CLI all call the same typed tools.

How

• Console: Electron, with experiments, worlds, assets, arms and artifacts; an agent chat that streams tool calls; camera feeds; metrics; and status-bar entry points for recording, training and replay
• Virtual worlds: a three.js editor with Rapier physics, the SO-101's URDF meshes, about thirty-five glassware, sensor and lighting items, drag-in placement, transform gizmos and collision checks
• Hardware: a Python bridge for LeRobot (Feetech servos), placo for inverse kinematics, a six-endpoint calibration wizard with a ghost setpoint, and leader/follower teleop for recording datasets
• Vision: OpenCV vial detection and bromothymol-blue colour classification into an estimated pH, logged per vial
• Interfaces: an MCP server, a phys0 CLI and the console, all over the same backend and the same SQLite store

The 3D world editor with an SO-101 arm, a vial rack, a beaker and a flask dragged onto the bench

Results

In the recording, the agent was asked to add a virtual SO-101 to a world and list what was in it. It made three tool calls (list_worlds, create_virtual_arm, list_virtual_entities) and reported the new arm with its ID, pose and collision settings alongside the bench's thirteen other entities. On the real bench in May, the camera located the vials and estimated each one's pH from its dye:

A real camera frame from the bench: the SO-101 arm, a rack of vials each boxed and labelled with an estimated pH, and a multimeter

Only the SO-101 is verified hardware. The other robots in the asset catalogue (Franka Panda, UR5e, xArm 6, a PX4 drone) are unvalidated records, and the project's own ledger says not to present them as working. Making the video also turned up two bugs worth knowing about: the Titration-station template's bench falls through the floor under physics, which leaves its arm in collision, and the CLI's sim start does not see a world the same CLI just created.

Lessons

Putting every action behind a typed tool was the right abstraction, and it paid off in an unexpected place: it made the agent cheap to trust, because its moves are the same records as a person's. The weak point is everything that touches the real world (calibration and servo compliance carried over from Chem-0), and the next version should spend its effort there rather than on more tools.