dsh-computer-use

LIMITATIONS

Known limitations

- **A text-only model cannot see pixels.** It works only through `uiSummary` outlines, `ui_element` paths and window metadata. Anything canvas-drawn (games, image editors, custom-rendered apps) is effectively invisible to it. - **DPI and multiple monitors.** The host is per-monitor DPI aware and works in virtual-screen physical pixels, so mappings stay correct, but: the overlay draws one border around the whole virtual screen rather than one per display, `screenshot` captures a single display or rectangle at a time, and moving a window between displays with different scale factors invalidates the previous `lastView` — retake a screenshot. - **Elevation.** A normal-integrity agent cannot drive an elevated window; it will click "through" it or appear to do nothing. For the same reason the overlay is invisible on the UAC secure desktop. - **Full-screen exclusive content** (DirectX exclusive fullscreen, some anti-cheat protected games) hides the overlay and may swallow injected input; the Esc kill switch is a user-level poller and can be starved too. Windowed/borderless fullscreen is fine. - **Every screenshot costs a round trip and money.** A capture is a real GDI screen grab, a JPEG encode, an attachment write and then image tokens on the wire. Prefer `observe: true` on the action you were going to verify anyway, and prefer `ui_element` when a control has a name. - **One desktop, one model.** There is one runtime per plugin instance and one host process; concurrent sessions in the same profile share it (and share `lastView`). - **Clipboard is plain text only.** Images and file lists are neither read nor written. - **`actionDelayMs` only covers the settle window.** It is a pause between an action and its `observe` screenshot; it does not make a slow application finish faster. - **`resume` is a model-callable action.** After an Esc the model is expected to wait until the human explicitly asks it to continue; the 5 second cooldown is a speed bump, not a guarantee, and nothing stops a model that ignores the instruction. For a hard stop, close the DSH session, or disable the plugin (`- id: computer-use` with `disabled: true`) or uninstall it. - **Skills are plain markdown on your disk, and the model can write them.** That is the point — a successful run becomes a recipe — but it also means a wrong or over-general recipe can be saved and then quietly applied to later work. The user layer is just files under `$DSH_HOME\computer-use-skills`: delete one, or call `app_skill` with `remove`, and the built-in (or the previous behaviour) comes back.