Your first API call

This Python program calls the cloud model and prints the returned actions. You need an API key, a camera image, and the matching robot state. Running it does not command a device.

1. Set up your environmentCopy Anchor Link

Create a key in the Developer Console. Install the OpenAI Python SDK, then set PLATFORM_BASE_URL and PLATFORM_API_KEY as shown in Account and API keys.

python3 -m venv .venv
source .venv/bin/activate
python -m pip install openai

List the models available to your key:

curl --fail-with-body "$PLATFORM_BASE_URL/models" \
  -H "Authorization: Bearer $PLATFORM_API_KEY"

Copy the id of the model you want to use into PLATFORM_MODEL. Replace the placeholder below:

export PLATFORM_MODEL='<MODEL_ID>'

2. Prepare an observationCopy Anchor Link

Save an RGB PNG as input.png. Save the eight robot-state values measured at capture time as state.json. These values describe position, a rotation quaternion, and the gripper. See Observations and actions for their order and coordinate requirements.

Use real state values that match your controller's adaptation. The image and state should describe the same observation. Replace the example instruction with a concrete task visible in your image.

3. Request actionsCopy Anchor Link

Save this as first_action.py:

import base64
import json
import os
from pathlib import Path
from openai import OpenAI

image = base64.b64encode(Path("input.png").read_bytes()).decode("ascii")
state = json.loads(Path("state.json").read_text())

with OpenAI(
    base_url=os.environ["PLATFORM_BASE_URL"],
    api_key=os.environ["PLATFORM_API_KEY"],
    timeout=60,
    max_retries=0,
) as client:
    response = client.chat.completions.create(
        model=os.environ["PLATFORM_MODEL"],
        messages=[{
            "role": "user",
            "content": [
                {"type": "text", "text": json.dumps({
                    "instruction": "Unfold the blue shirt",
                    "state": state,
                })},
                {"type": "image_url", "image_url": {
                    "url": f"data:image/png;base64,{image}",
                }},
            ],
        }],
    )
    print("Request ID:", response._request_id)
    calls = response.choices[0].message.tool_calls or []
    action_call = next(
        call for call in calls
        if call.type == "function" and call.function.name == "execute_actions"
    )
    actions = json.loads(action_call.function.arguments)["output"]
    print("Actions:", len(actions))
    print("First action:", actions[0])

python first_action.py

A successful call prints the request ID, action count and first row. response._request_id reads the X-Request-ID response header for request queries; do not substitute the completion's id. The model determines the number of rows. See Observations and actions for the meaning of each row.

Build your control loopCopy Anchor Link

Each request carries one observation. Your application repeats these steps:

1. Read the latest camera image and robot state.
2. Request actions with the task instruction.
3. Validate the action format and execute within your controller's coordinate, speed, and motion limits.
4. Capture a fresh observation for the next request.

For continuous tasks, use sessions and execution receipts to link observations and actions across turns. After a stop or communication failure, discard unexecuted stale actions and check the robot's state again. A network retry must not cause an earlier action sequence to execute twice.

For a ready-made device acquisition and execution flow, continue with RoboAPI installation. For another language, see JavaScript and curl. Consult the API reference for fields and API errors for failures.

To connect another robot or develop a complete hardware adapter, continue with Connect new hardware and Hardware adapter interfaces.