Commands & Actions

A node implements 37 command types. Twenty-eight of them are exposed to an agent as actions it can put in a Control Run; the rest are driven by other parts of the platform.

Action names and command types are not always the same word. mouse.click travels as input.mouse.click. Actions are what an agent sends; command types are what the node receives, what a scope gates, and what the audit log records. See Actions and command types.

Action and command-type map

Risk decides what gets asked. None and Low run inside the scope you granted; Medium is visible in the run; High always raises an approval; Critical raises one that no run-level consent can pre-cover. See Approval Gates.

Command types with no action

Nine command types exist that an agent cannot put in a run. They are driven by other parts of the platform instead:

All four activity.* command types are gated by the single activity.read scope. There is no activity.read command.

Screen

screen.capture

Captures the full primary display.

Scope: screen.capture · Risk: Low

Response: { image: "<base64-jpeg>", sizeBytes, format: "jpeg" }, plus the origin and size of the screen region it came from so image coordinates can be scaled back to real pixels. That geometry is emitted on Windows only. See Platform Support & Limits.

screen.region

Captures a rectangular region.

Scope: screen.capture · Risk: Low

Response: { image, sizeBytes, format, region: { x, y, width, height } }

screen.monitors

Lists the attached displays and their geometry.

Scope: screen.capture · Risk: None

Response: { count, monitors: [ { id, isPrimary, x, y, width, height, scaleFactor, name } ] }

screen.queryElements

Returns on-screen UI elements, for automation that names controls instead of guessing at pixels.

Scope: screen.capture · Risk: None

Response: { elements: [...], count, focusedWindowTitle, warning? }

Capped at 300 elements; a truncated result comes back with a warning suggesting a narrower filter or a smaller depth. On Windows, password fields are dropped at the source, so the element never leaves the machine at all. Coverage varies sharply by platform: full element access inside windows on Windows, shallower on macOS, top-level windows only on Linux under X11, and nothing under Wayland.

screen.context

Reports what the desktop is doing right now, without taking a picture of it.

Scope: screen.capture · Risk: None

Response: { foregroundApp, activeWindow, focusedElement: { role, name, value }, monitorCount, capturedAt, warning? }

Best-effort: if something cannot be determined it returns a result with a warning rather than failing.

This is on demand. The agent asks for screen.context as an action inside a run when it needs to know what is in front of the user. It is not handed a desktop context block on every turn: that is on the roadmap and does not ship in this release. See Desktop Awareness.

Mouse

input.mouse.move

Scope: input.control · Risk: Low

input.mouse.click

Scope: input.control · Risk: Medium

input.mouse.scroll

Scope: input.control · Risk: Medium

Keyboard

input.keyboard.type

Types a string character by character.

Scope: input.control · Risk: Medium

input.keyboard.press

Presses a single named key.

Scope: input.control · Risk: Medium

input.keyboard.hotkey

Modifier combinations.

Scope: input.control · Risk: Medium

The parameter is keys, not key, for both of the above.

Applications

app.launch

Scope: app.manage · Risk: High

Response: { pid, app }

app.close

Closes gracefully, then forces after a few seconds.

Scope: app.manage · Risk: High

app.focus

Brings a window to the foreground.

Scope: app.manage · Risk: High

app.list

Scope: app.manage · Risk: None

Response: { count, apps: [ { name, pid, windowTitle } ] }

Windows

window.list

Scope: window.manage · Risk: None

{
  "count": 3,
  "windows": [
    { "id": "0x00010234", "title": "Untitled - Notepad",
      "x": 100, "y": 100, "width": 800, "height": 600,
      "processName": "notepad.exe" }
  ]
}

window.move

Scope: window.manage · Risk: Medium

window.minimize, window.maximize

Scope: window.manage · Risk: Medium

Clipboard

clipboard.read

Scope: clipboard.access · Risk: Low

Response: { text, length }

clipboard.write

Scope: clipboard.access · Risk: Medium

Files

filesystem.list, filesystem.read, and filesystem.write are documented with their containment rules and size caps in File Access, because the parameters make little sense without the roots that bound them.

The scope is not granted by default, and an empty roots list denies everything.

System

system.info

Scope: system.info · Risk: None

{
  "osDescription": "Microsoft Windows 11.0.26200",
  "machineName": "DESKTOP-ABC",
  "processorCount": 8,
  "totalMemoryMb": 16384,
  "uptime": 3600,
  "platform": "windows",
  "architecture": "x64"
}

system.processes

Scope: system.info · Risk: None

Response: { count, processes: [ { pid, name, cpuPercent, memoryMb } ] }

system.execute

Runs a shell command. Always approval-gated, whatever the scope configuration says.

Scope: system.execute · Risk: Critical

Runs through the platform shell: cmd.exe /c on Windows, /bin/zsh -c on macOS, /bin/bash -c on Linux. Response: { exitCode, stdout, stderr, durationMs }, with each stream truncated at 1 MB.

On Windows a timed-out or cancelled command has its entire process tree killed, so a hung command does not leave children behind.

Notifications

notify.show

Scope: notify.send · Risk: Medium

Canvas

canvas.present opens the session's live Canvas on the node's own screen. It is consent-gated on the device by default and is documented, with its parameters, in Canvas on a Node.

Activity

Four command types serve the node's local activity history. All four are gated by the activity.read scope, which is not granted on pairing, and none of them returns anything useful until recording has been enabled for that device.

• activity.query reads back entries within a bounded time window. An entry that has a saved screenshot is reported as having one, and that image can then be viewed: desktop.activity's screenshotFor parameter fetches a single moment through activity.screenshot. The image is put in front of the model itself only on Claude models reached through the Anthropic API or a connected Claude subscription. On every other provider path — OpenAI-compatible, Gemini, Bedrock, Copilot — the model receives the text result alone.
• activity.settings reads the recorder's configuration and, with an update, changes it: on or off, paused, sample interval, retention, both exclusion lists, the screenshot settings, and erasing the history outright. It reports live recorder state alongside the settings, so the Dashboard shows what is actually running rather than what was configured. Changes apply without restarting the node.
• activity.screenshot returns the saved image for one entry, by id, one entry per call.
• activity.delete removes named entries from the local history and deletes their screenshots with them.

See Desktop Awareness.

Voice runtime

Four command types let a node speak: voice.session.start, voice.session.stop, voice.session.interrupt, and voice.output.speak. They are gated by the voice.runtime scope, which is not granted by default, and are driven by the voice runtime and the CLI rather than by an agent run.

voice.output.speak reads text aloud through the operating system's own speech synthesiser. Note that these are output only. The node does not listen, and there is no wake word or microphone capture on the node.

The Gateway's node-token voice routes are off until an operator turns them on. Set Sophon:Voice:NodeWake:Enabled, which defaults to off; until it is set, every one of those routes answers as if it did not exist. Nothing shipped calls them today.

A node's reported voice state follows what actually happened rather than what was asked for: speaking reverts to armed or idle when the speech process finishes, is interrupted, or the session is stopped.

On macOS, setting a language used to silence every spoken reply. The node now speaks through the system voice, and a reply is no longer passed to the speech tool as a command-line argument, so text beginning with a dash can neither fail to speak nor be read as an option.

Breaking changes in this release

Errors

Command failures return a structured error with a code. The wire-level list, including busy, cancelled, and node_offline, is in the node protocol. For why a command was refused before it ever ran, see audit outcomes.

Where to go next

• Control Runs — how these actions batch into one approved unit
• Permissions & Scopes — which scopes gate what
• Gates, Limits & Audit — rate limits and consent
• Platform Support & Limits — where each command behaves differently