Camera and Audio

The robot publishes its camera and microphone as tracks in its LiveKit room, and plays back an audio track you publish. The SDK carries them on the hybrid and livekit lanes; the udp lane carries drive and state only.

uv add --prerelease allow "menlo-sdk[livekit]"

Ask First

Capabilities are per-transport, so test rather than assume:

if robot.has("camera"):
    frame = robot.camera.photo(timeout=5.0)

robot.require("drive", "camera")     # raises UnsupportedError if missing
robot.info.capabilities              # frozenset({'drive', 'state'}) over UDP

The full set is drive, state, battery, camera, microphone, speaker.

One Frame

frame = robot.camera.photo(timeout=5.0)     # ONE fresh rgb8 Frame
frame.width, frame.height, frame.encoding
frame.to_jpeg(quality=85)                   # bytes, for a vision model or an upload
frame.to_numpy()                            # (h, w, 3) uint8

to_jpeg needs Pillow and to_numpy needs numpy; each names the package in its ImportError. The SDK itself depends on neither.

A Stream

robot.camera.latest()                       # newest frame or None, never blocks
for frame in robot.camera.frames(timeout=5.0): ...   # latest-wins iterator
robot.camera.subscribe(on_frame)            # callback per Frame, on the transport thread

for chunk in robot.microphone.chunks(timeout=5.0): chunk.data   # pcm_s16le, in order
robot.microphone.dropped                    # chunks a slow consumer lost

robot.speaker.play_pcm(pcm_s16le, sample_rate_hz=16_000, channels=1)

A vision loop, in the shape it should take:

with Robot().connect() as robot:
    robot.require("camera", "drive")
    if robot.state.mode is Mode.DAMP:
        robot.stand(); robot.wait_for(Mode.STAND, timeout=10.0)
    while running:
        frame = robot.camera.latest()
        if frame is None:
            robot.stop(); continue
        vx, vyaw = steer(frame.to_numpy())       # your model here
        robot.set_velocity(vx, 0.0, vyaw)         # held; a slow model does not stop the walk
    robot.stop()

A Clip

clip = robot.camera.capture_clip(5.0, audio=True)   # Clip(frames, audio, started_at)
clip.duration_s, clip.fps
clip.save_wav("clip.wav")                           # standard library
clip.save_frames("out/")                            # JPEGs; needs Pillow
clip.save_mp4("clip.mp4")                           # needs opencv-python
clip.frames_as_numpy()                              # (n, h, w, 3); needs numpy

Media Before the Firmware

connect("livekit", require_state=False) opens the room without waiting for the robot to report. Camera, microphone and speaker work whether or not the firmware is running; state and the motion verbs unlock when the first state frame arrives.

Agents in the same room

A voice or vision agent framework subscribes to the robot's tracks the way it subscribes to a human's webcam, and the SDK joins the same room as one more participant to do the driving. The agent framework's LiveKit credentials belong to that process; the SDK never sees them.

Connect — the lanes that carry media

Reference