Understudy is a teachable desktop agent. It operates your computer like a human colleague across GUI, browser, shell, files, and messaging. You demonstrate a task once, it learns the intent, remembers successful paths, and gradually upgrades to faster execution routes.
See Understudy in action — operating desktop apps, browsers, and terminal in a single agent session.
The video above has been sped up. Full-length original: Google Drive
The published skill artifact generated from this showcase demo is available at examples/published-skills/taught-person-photo-cutout-bc88ec/SKILL.md.
Understudy is not interesting because it can press buttons. It is interesting because native desktop execution, teach-by-demonstration, memory crystallization, and route upgrades live inside one agent runtime.
A single task can search the web, run a local command, switch to a native app, click through a workflow, and send the result through a messaging channel. This is not stitched-together automation. It is one routed agent loop.
Understudy records video and semantic events, then learns workflow intent rather than a coordinate trace. Scene detection, event clustering, keyframe extraction, evidence pack analysis, and teach drafts are already part of the product model.
GUI is the universal fallback, not the permanent destination. Understudy can begin with GUI execution and later promote the same task toward browser, CLI, or API routes after repeated verified success.
This section stays intentionally strict: it reflects capabilities explicitly described in the docs today, without presenting future layers as already shipped.
From screenshot capture and HiDPI normalization to target resolution, click-point stabilization, complex-mode overlay validation, and post-action verification, the GUI pipeline is already a disciplined execution system rather than a fragile click bot.
`/teach start`, `/teach stop`, `/teach confirm`, `/teach validate`, and `/teach publish` already define a concrete interface for turning one desktop demonstration into a task card and publishable skill output.
The repository already includes daemon, chat, agent, dashboard, webchat, channels, and scheduling surfaces. That matters because the project is aiming at a persistent local agent system, not a one-shot automation wrapper.
The product narrative is unusually coherent: first make the agent capable, then teachable, then reliable, then faster, and only then proactive.
See, click, type, drag, scroll, and verify across real macOS applications.
Capture evidence, extract workflow intent, refine task cards, and publish reusable skills.
Repeated day-to-day usage is automatically crystallized into reusable workspace skills — no explicit teaching required.
Teach prompts and route preferences already push tasks toward faster routes, with a simpler but real upgrade path in place today.
Understand routines, suggest next actions, and act without monopolizing the user’s screen.
This is one of the strongest product ideas in Understudy: GUI is the universal start, but it is not the ideal final route.
If a capability already has a direct tool or API surface, the agent should not route back through GUI.
Local commands, scripts, and filesystem operations often avoid repeated interface work entirely.
When the natural interface is a website, managed browser control can beat GUI while preserving real workflows.
Anything with a visible interface can be operated. This is what gives Understudy broad real-world coverage.
The planner is not choosing a favorite tool up front. It is routing step-by-step toward the objective. That means the same task can inspect the web, read local files, operate a native app, and send the result out through a channel inside one flow.
The current README recommends starting the daemon first, then using terminal or dashboard entry points. The project is still Alpha, with development and testing centered on macOS today.