protologue

Computer-Use Agent

Also called computer use, GUI agent, browser agent, browser use.

A computer-use agent operates a graphical computer interface the way a person does, by looking at screenshots and issuing mouse clicks, keystrokes, and scrolls, so it can work with software that has no API.

Description

The agent loop alternates between a screenshot and an action until the task is done. Benchmarks such as OSWorld measure performance on real desktop tasks. Because these agents read arbitrary on-screen content, they are exposed to prompt injection from web pages and documents, so they are run in isolated environments with human confirmation for sensitive actions.

Sources

  1. Xie et al. (2024). OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments.
  2. Anthropic (2024). Computer use tool.

Cite this entry

Protologue. (2026). Computer-Use Agent. In Protologue: A Taxonomy of Prompting and LLM Techniques (v1.4.0, PTL-0144). https://protologue.com/t/computer-use-agent/

BibTeX
@misc{protologue_computer_use_agent,
  title = {Computer-Use Agent},
  author = {{Protologue}},
  year = {2026},
  howpublished = {Protologue: A Taxonomy of Prompting and LLM Techniques, v1.4.0},
  note = {Entry PTL-0144},
  url = {https://protologue.com/t/computer-use-agent/}
}

Markdown JSON