Observations and actions
Prepare the instruction, camera image, and robot state for an inference request.
An observation combines a task instruction, a camera image, and the robot's state. Capture the image and state together so they describe the same positions of the robot and objects.
Select a model
Call GET /models with your API key and select an id from the response's data array as the request's model. Model availability depends on account access. Use the identifier returned by the API. See the Cloud API for the request.
When integrating or switching models, read GET /models/{model_id}/schema to check the input and output constraints:
| Schema field | What to check |
|---|---|
inputs.properties.state | Array length and numeric ranges |
image | File format, RGB color mode, pixel count, and aspect ratio limits |
text | Non-empty input and max_chars when present |
output, session.output | Shapes of raw model output and delivered session actions |
Task instruction
Describe the goal in concise natural language, such as “Unfold the blue shirt.” Keep the instruction consistent while collecting new observations for the same task.
Place it in instruction and encode it together with state as the message's JSON text:
const text = JSON.stringify({ instruction, state });Camera image
Provide one RGB PNG or JPEG. Encode the original file as a base64 data URL:
data:image/png;base64,<BASE64_IMAGE>Use the data:image/jpeg;base64, prefix for JPEG. The MIME type must match the file format.
Place the data URL in image_url.url. Keep the pixel count and aspect ratio within the schema's max_pixels and max_aspect_ratio limits; follow that model's image field for size and preprocessing requirements. The camera view and image orientation must match its input requirements. When using the local console, images come from its configured camera.
Robot state
state is an array of 8 finite numbers. The RoboAPI model interface uses this order:
[x, y, z, qx, qy, qz, qw, gripper]| Field | Meaning |
|---|---|
x, y, z | Control-point position in the model's task frame, in meters |
qx, qy, qz, qw | Control-point orientation as an xyzw quaternion |
gripper | Normalized gripper opening: 0 closed, 1 open |
Use a current captured state, a normalized quaternion, and valid finite numbers. When reading a robot vendor's interface directly, convert its coordinate frame, quaternion order, and gripper opening first.
The RoboAPI local runtime uses the flange center as its control point. Coordinate calibration maps robot and model coordinates.
The model interface uses xyzw quaternion order. Robot vendors and individual local interfaces may use different pose representations; convert according to each interface's definition. The RoboAPI runtime adapts supported devices. Custom integrations must also match the selected model's coordinate frame and tool orientation.
Read the output
The returned output contains multiple action rows in the same 8-value order. Validate the row count against the model schema and the current response instead of fixing it in client code.
In session responses, metadata.action_shape is a JSON string describing the delivered actions. Parse it to validate the current array. inference_usage.output_shape describes the raw model output, not the number of executed steps. Check numeric validity, robot workspace, and motion limits before execution. See Tasks and sessions for the control loop and receipts, and the Cloud API for request fields.