JavaScript and curl
Submit a camera image and robot state, then read the returned action sequence.
These Node.js and curl examples use the same inputs to request and print actions without commanding a robot. For a custom integration, your controller validates and executes these actions. With RoboAPI, submit a task instruction to the local HTTP API; its runtime handles observation, inference and execution.
Prepare
Configure PLATFORM_BASE_URL and PLATFORM_API_KEY as described in Account and API keys, then list available models:
curl --fail-with-body "$PLATFORM_BASE_URL/models" \
-H "Authorization: Bearer $PLATFORM_API_KEY"Set PLATFORM_MODEL to an id from that response and replace the instruction with your task:
export PLATFORM_MODEL="<MODEL_ID>"
export ROBOT_INSTRUCTION="Unfold the blue shirt"Prepare two files in your working directory:
input.png: a current camera observation in RGB PNG format.state.json: the corresponding robot state as a JSON array of 8 numbers. See Observations and actions for field order and coordinates.
JavaScript
Use Node.js 22 or later. Save the following as infer.mjs and run it.
import { readFileSync } from "node:fs";
function env(name) {
const value = process.env[name];
if (!value) throw new Error(`Set ${name} first`);
return value;
}
const baseUrl = env("PLATFORM_BASE_URL").replace(/\/$/, "");
const state = JSON.parse(readFileSync("state.json", "utf8"));
if (!Array.isArray(state) || state.length !== 8 || !state.every(Number.isFinite)) {
throw new Error("state.json must contain an array of 8 finite numbers");
}
const image = readFileSync("input.png").toString("base64");
const response = await fetch(`${baseUrl}/chat/completions`, {
method: "POST",
signal: AbortSignal.timeout(60_000),
headers: {
Authorization: `Bearer ${env("PLATFORM_API_KEY")}`,
"Content-Type": "application/json"
},
body: JSON.stringify({
model: env("PLATFORM_MODEL"),
messages: [{
role: "user",
content: [
{ type: "text", text: JSON.stringify({
instruction: env("ROBOT_INSTRUCTION"), state
}) },
{ type: "image_url", image_url: {
url: `data:image/png;base64,${image}`
} }
]
}]
})
});
const requestId = response.headers.get("x-request-id");
console.log("Request ID:", requestId);
const result = await response.json();
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${result.error?.message ?? "Request failed"}`);
}
const call = result.choices?.[0]?.message?.tool_calls?.find(
(item) => item.function?.name === "execute_actions"
);
if (!call) throw new Error("Missing execute_actions result");
const { output } = JSON.parse(call.function.arguments);
if (!Array.isArray(output) || output.length === 0 || !output.every(
(row) => Array.isArray(row) && row.length === 8 && row.every(Number.isFinite)
)) {
throw new Error("Expected an action array with 8 finite numbers per row");
}
console.log(output);node infer.mjscurl
These commands require Bash or Zsh and jq. Use the environment variables and two input files above to create a request and save its response:
jq -n \
--arg model "$PLATFORM_MODEL" \
--arg instruction "$ROBOT_INSTRUCTION" \
--rawfile image <(base64 < input.png | tr -d '\r\n') \
--slurpfile state state.json \
'{model: $model, messages: [{role: "user", content: [
{type: "text", text: ({instruction: $instruction, state: $state[0]} | tojson)},
{type: "image_url", image_url: {url: ("data:image/png;base64," + $image)}}
]}]}' > request.json
curl --fail-with-body --max-time 60 \
"$PLATFORM_BASE_URL/chat/completions" \
-H "Authorization: Bearer $PLATFORM_API_KEY" \
-H "Content-Type: application/json" \
--data-binary @request.json \
--dump-header response.headers \
--output response.jsonAfter a successful request, read the actions from execute_actions:
jq -e '
[.choices[0].message.tool_calls[]?
| select(.function.name == "execute_actions")][0]
// error("Missing execute_actions result")
| .function.arguments | fromjson | .output
' response.jsonResponse headers are saved in response.headers. Read X-Request-ID to query request status and usage:
REQUEST_ID=$(awk 'tolower($1) == "x-request-id:" {gsub("\r", "", $2); print $2}' response.headers | tail -n 1)
printf 'Request ID: %s\n' "$REQUEST_ID"Read the action row count from the returned array. If a request fails, follow Cloud API errors to handle its HTTP status before sending another request.