Remote Control & Desktop Automation
Let an agent control a paired desktop — mouse, keyboard, apps, windows — through a Sophon Node.
The remote.control tool is the agent's bridge to a paired desktop. Through a Sophon Node, an agent can take screenshots, read the on-screen UI tree, move and click the mouse, type and press hotkeys, launch or focus applications, manage windows, read and write files and the clipboard, and, with explicit approval, run a shell command. The reasoning happens on the Gateway; the node executes and returns the result.
One call carries an ordered batch of actions, not a single action. The batch runs in order on one pinned machine, stops at the first failure, and ends with a fresh screenshot, all under one approval. The mechanics are in Control Runs.
If you have exactly one paired Node, the tool selects it automatically. With multiple Nodes, the agent passes a nodeId to target one specifically.
When to use it
- Driving a desktop app with no API — a legacy tool, a vendor portal, an Excel macro.
- Hands-on confirmation flows — clicking through a dialog on your home PC while you're away.
- Visual tasks — capture the screen, query the UI, then act on what's there.
For pure web work, prefer Browser Automation — it's faster and more reliable than driving a real desktop browser.
Command categories
Every action maps to a node command type, which is gated by a permission scope. The tool covers these categories:
| Category | Scope |
|---|---|
| Screen | screen.capture |
| Mouse and keyboard | input.control |
| Clipboard | clipboard.access |
| Applications | app.manage |
| Windows | window.manage |
| Files | filesystem.access |
| Notifications | notify.send |
| Shell | system.execute |
screen.queryElements reads on-screen UI elements, so the agent can target controls by name instead of guessing pixel coordinates, though how deeply it can see varies by platform.
Action names are not always the same as the command types the node receives: mouse.click travels as input.mouse.click. The full mapping, with every action's scope and risk, is in Commands & Actions.
Permission model
Scopes are enforced Gateway-side, before a command ever reaches the Node. If the target Node doesn't hold the required scope, the Gateway returns a permission-denied result — the action never runs. A Node only allows the command scopes you granted it in the Dashboard, so a Node configured for screenshots and clicks simply cannot launch an app or run a shell command.
Configure these per-Node under Settings → Devices → <node> → Permissions. The wildcard scope node.command grants every scope. It is a scope name, not a tool (why). For shared or sensitive machines, grant specific scopes instead. Full details are in Permissions & Scopes.
A run's risk is the highest risk among its actions. Input and window actions sit at Medium; launching, closing or focusing an app and writing a file require High approval; and shell.execute is Critical — every invocation routes through the approval gate with the full command shown to you, on top of a built-in denylist. There is no "trust this tool" override for shell execution, and no run-level approval covers it.
Showing a canvas on the desktop
A related tool, node.present_canvas, opens the agent Canvas for the current session on the Node's screen. The Node launches the Gateway's Dashboard canvas page in its default browser, so everything the agent pushes via canvas rendering shows up live on the desktop rather than only in the Dashboard. This command requires the canvas.control scope and is consent-gated on the Node by default. See Canvas on a Node for the full flow.
Example agent flow
User: "Open Notepad on my office PC and paste my address."
Agent:
remote.control({
actions: [
{ action: "app.launch", params: { name: "notepad.exe" } },
{ action: "keyboard.type", params: { text: "123 Example St" } }
],
observe: "screenshot"
})
Response: "Done — Notepad is open with your address typed."That is one approval, not two, because it is one run. It halts if Notepad fails to launch, so the address is never typed into whatever happened to be focused instead. The closing screenshot is how the agent confirms what it did.
When the agent captures the screen, the result is attached inline in chat so you can see exactly what the node saw, and it reaches the model itself, not just the transcript. While a run is in flight the chat shows a live panel logging each step, with a Stop button that cancels both the command and the turn.
Where to go next
- Sophon Node — architecture, pairing, lifecycle
- Control Runs — batching, halting, and what one approval covers
- Commands & Actions — full parameter reference
- Gates, Limits & Audit — the checks every action passes
- Permissions & Scopes — what each scope grants