Building a Harness for Physical AI: Any Model, Any Machine

We gave an agent an API key and one instruction: pick up the red block and place it on the pad. It knew nothing else about the machine, a simulated UR5e cell.

At 14 seconds it asked the machine what resources it had. At 41 seconds it asked where everything was mounted. Then it pulled camera properties, detections, and point clouds, grabbed the block, looked again before placing it, and let go. Total: 13.8 minutes. The block landed 2.4mm from center, measured by a wrist-camera check, not by the agent.

The agent didn’t succeed because it knew the robot. It succeeded because the robot could answer its questions. That’s what we mean by a harness.

Why robots need a harness

Coding agents became dependable for a similar reason. Software already answers questions a model would otherwise guess at. The file system says what exists, tests say whether it works, and a bad change costs a revert. A coding harness puts those answers within the model’s reach.

A robot has none of that by default. Camera placement lives in a calibration script. The arm’s capabilities live in a vendor SDK. Whether a move is safe lives with whoever is standing near the e-stop. And there’s no revert: a wrong guess means real motion near real objects.

Robot foundation models are getting better fast. The machine underneath hasn’t changed, and it still needs four answers: What can I do? Where am I? What am I allowed to do? Did that work?

When we started on this, our working assumption was that models couldn’t supply those answers. We had to correct that. Policies trained across many robots can learn to drive a new one, and perception models can estimate where things are. The more accurate framing is that models estimate, and machines must know.

That distinction matters in three places. First, known facts: where an arm is mounted and its joint limits can be written down once, so estimating them only adds error. Second, guarantees: a limit must hold precisely when the model is wrong, so it can’t depend on the model being right. Third, judgment: whatever checks the result must be independent of whatever produced it.

Four failures, four deterministic fixes

Over the last year, we watched every kind of model meet a machine: frontier models used as components, engineers building with agents, cold agents like the one above, learned policies on real robots, and machines opened to any agent through our MCP server. Different models, different tasks, the same four failures. Each mapped to one of the four questions, and each fix turned out to be deterministic.

They couldn’t find commands that existed. Our run notes put it best: the platform wasn’t missing a capability, it was missing the sentence that tells an agent the capability exists. One added line in the CLI help cut first contact from about 40 seconds to 11. Now the machine describes itself, JSON in and JSON out, and every tool is tiered by consequence: read-only, live read, writes behind approval, and live destructive calls with no preview or undo. The tiers are explicit, so the agent doesn’t have to guess.

They planned against geometry nobody declared. In one run, a camera with no declared orientation let an agent localize a block to half a millimeter in an image that meant nothing. The fix wasn’t a better model. It was a declaration: mounting, intrinsics, and fixed geometry, written once in a versioned config and never re-estimated.

They kept moving after the session ended. In one incident, a gripper released its load when the agent’s session lapsed. Limits can’t live in the agent’s code: that’s the thing that’s wrong when something goes wrong. On a robot, wrong means a dropped part, a collision, or an arm moving with nobody in control.

So limits sit below the caller, beyond its reach. A heartbeat covers every actuation path: 10ms interval, 2-second window, and if the caller is gone, actuation stops. Every motion is checked against the declared world and refused if it would collide. Every operation is cancellable, and one call stops everything. A limit that holds only when the model is right isn’t a limit.

They said “done.” They weren’t. Agents reported success on tasks they hadn’t actually completed. A success report is just another estimate, and the agent has every reason to be optimistic. So the system checks the result, not the agent’s claim. The rule is simple: verify by observation, not return value. Move, look again with the camera, and compare the result against the declared goal. In simulation, an oracle checks every verdict. In our runs, a wrist-camera check scored every placement.

Seats, not model families

The names in this space keep changing: VLA, world model, large behavior model, foundation model. Designing around any one means redesigning every quarter. Because of this, a harness is built around roles, not model families. Each role is defined by its output, and each output fails in its own way:

• Perceive outputs estimates about the world: boxes, masks, poses, grasps. It needs camera position in the robot’s frame and confirmation that the image is still current when the arm moves.
• Predict outputs expected futures: next frames, outcomes. It needs an anchor to what actually happened and a clock shared with the machine.
• Control outputs actions: joint targets, end-effector deltas. It needs the machine’s limits and a stop it can’t override.
• Orchestrate decides what to do next. It needs an honest list of tools and clarity about who’s in control.
• Construct outputs code, configuration, and modules. It needs the lifecycle contract of what it’s building and proof that it deploys.
• Judge outputs verdicts. It needs ground truth and independence from what it’s judging.

A seat is a typed contract for a role’s output, and any model can be wrapped to fit it. A learned policy, for example, is one more component in a machine’s config.

{
  "name": "policy",
  "model": "viam-labs:vla:controller",
  "attributes": {
    "model_path": "lerobot/smolvla_base",
    "arm": "xarm",
    "camera": "wrist-cam",
    "rate_hz": 10
  }
}

‍

Its output passes through the same limits as everything else: bad values are dropped, each step is capped, joint and speed limits enforced, collisions refused. Swap it for a motion planner and the arm API, limits, and record underneath don’t change. The line between computed and learned can move in both directions without a rebuild.

Why better models make the harness matter more

The harness isn’t scaffolding you remove once models are good enough. However good a policy gets, it can’t establish what machine it’s on, where it is, what it’s allowed to do, or whether it worked. Those answers must be built for every machine it runs on. Better models get deployed on more machines, so that work grows with model quality. It doesn’t shrink.

The fair objection is that a human-maintained deterministic layer will fall behind models improving every quarter. Software’s deterministic layer kept up because agents help build it, and this one is designed the same way. When an agent figures out a new arm, it implements the arm API once and ships it as a module. Every other agent and caller can then use it unchanged. DoCommand is the escape valve for anything the typed APIs don’t cover yet; extensions that prove out get promoted into typed contracts. Versioned config, scoped keys, and approval gates govern who can change what.

A working definition

A harness for physical AI is a small set of answers the machine must know rather than estimate, seats for the models that estimate everything else, a boundary between the two that moves in both directions, and the governance that keeps it trustworthy as it grows.

That’s our current thinking, and I expect it to keep changing. If you’re working on the same problems, I’d like to hear how you’re approaching them.