Gemini Computer Use
Gemini Computer Use is a tool in Google's Gemini APIs for building computer-use agents. A supported model receives screenshots and proposes interface actions. The application supplies the browser, mobile device or desktop environment and executes those actions.[1][2] Google made the tool generally available across the Interactions and GenerateContent APIs on October 30, 2026.[3]
The tool is distinct from a finished consumer assistant: enabling it supplies an action-planning interface, while application code runs the interaction loop and handles user approval.[2] Google's November 4 announcement said the transition from preview introduced no breaking changes. That announcement concerned the tool's availability, rather than guaranteeing compatibility with every earlier model or application.[4]
History and scope
Google introduced the Gemini 2.5 Computer Use model in preview on October 7, 2025. It was a specialized model based on Gemini 2.5 Pro's visual understanding and reasoning, available through the Gemini API and Vertex AI. The launch described interface control as a way to automate workflows across websites and applications, including software without a suitable direct integration.[5]
The current tool supports several general Gemini models rather than requiring the original specialized model. Google's October 30, 2026 changelog records general availability for both Gemini API surfaces. The subsequent Google AI Studio announcement reiterated availability for browser, mobile and desktop agents.[1][3][4]
Supported models and environments
The Gemini Developer API guide listed the following models on November 5, 2026. This is that guide's support list, not a claim that every model in the wider Gemini family accepts the tool.[1]
| Model | API identifier | Status in the guide |
|---|---|---|
| Gemini 4 Argon | gemini-4-argon | Supported |
| Gemini 3.8 Flash | gemini-3.8-flash | Recommended for computer use |
| Gemini 3.5 Flash-Lite | gemini-3.5-flash-lite | Supported |
| Gemini 3 Flash Preview | gemini-3-flash-preview | Preview model supported |
| Environment | Interface capabilities documented by Google |
|---|---|
| Browser | Pointer and keyboard actions, scrolling, screenshots and browser navigation |
| Mobile | Android-oriented actions including opening and listing applications, tapping and long pressing |
| Desktop | OS-level pointer and keyboard actions, dragging, scrolling and screenshots |
Actions include an intent field describing their purpose. The Developer API guide uses browser, mobile and desktop in Interactions tool configuration; GenerateContent examples use corresponding ENVIRONMENT_* values.[1][2]
Gemini 3.8 Flash's model page separately lists computer use as supported. It accepts text, image, video, audio and PDF inputs and produces text. Its documented limits are 1,048,576 input tokens and 65,536 output tokens. Those are model limits, not a guarantee about the length or success of a UI workflow.[6]
How the interaction loop works
The process resembles function calling, with the execution environment maintained by the developer:[2]
- Send the task, tool configuration and current screenshot to the model.
- Read its proposed function calls and any safety decision.
- Validate and, when permitted, execute the actions through the client environment.
- Capture the updated screenshot and current URL, return the action results, and request the next step.
Coordinates are normalized rather than direct viewport pixels. The client converts them to its actual screen dimensions. For example, Google's conversion formula maps a horizontal coordinate to int(x / 1000 * screen_width).[2] A new screenshot allows the model to observe the result of the previous action, rather than assuming that a click or keystroke achieved the intended change.
The API can request human confirmation through safety_decision. The client must present that request before executing the action and return the appropriate acknowledgement if the user approves. A denied or blocked action must not be treated as executed.[2]
Implementation and reference code
Google publishes an Apache-2.0-licensed reference application in the google-gemini/computer-use-preview repository. Its name retains the word "preview" even though the current API tool has reached general availability. The sample uses a local Playwright browser or a Browserbase session and accepts a task through its command-line interface.[7]
The repository distinguishes Gemini Developer API authentication from a Google Cloud configuration. The former uses a Gemini API key; the latter requires project and location settings. Its browser backends are sample implementation choices, not the complete set of environments exposed by the API.[7]
The sample also documents a practical screenshot limitation: native <select> menus may be rendered by the operating system and fail to appear in Playwright captures on some systems. This illustrates why a browser driver's behavior matters in addition to the model's capabilities.[7] Google recommends a sandboxed execution environment and constrained permissions for applications using the tool.[1][2]
Safety controls
The Google Cloud documentation describes policy categories for financial transactions, sensitive-record changes, communications, account creation, general data modification, consent management and legal agreements. Organization administrators can prevent developers from disabling selected human-confirmation policies using the vertexai.configurableComputerUseToolSafetyPolicies constraint. Those organization policies override developer settings.[2]
Prompt injection detection is opt-in and defaults to false. It scans screenshots for adversarial instructions; enabling it does not establish that all attacks will be detected. Google also warns that computer use can make errors and contain security vulnerabilities.[1]
These controls operate alongside the application's own authorization boundaries. A model-generated explanation does not establish user approval, and the client remains responsible for acting on confirmation results.[2]
Evaluation and limitations
Google's 2025 launch reported results for the specialized Gemini 2.5 Computer Use model on web and mobile benchmarks. Those historical comparisons should not be presented as measurements of the current tool on every supported Gemini model.[5]
Academic work separates successful task execution from safe behavior. OSWorld, introduced in 2024, provides real operating-system environments, task-specific initial states and execution-based evaluation. Its original benchmark contains 369 tasks involving web and desktop applications, file operations and workflows spanning applications.[8]
OS-Harm builds on OSWorld to assess deliberate misuse, prompt injection and unsafe behavior. Its 2025 study introduced 150 tasks and evaluated then-current models, including Gemini 2.5 Pro. It provides context for evaluating computer-use systems, not an independent safety score for the generally available Gemini tool or later models.[9]
Google's present documentation continues to recommend supervision for important tasks and avoiding situations involving critical decisions, sensitive data or serious irreversible errors.[1] General availability therefore records a product-release status; it does not establish autonomous reliability across arbitrary software.
Billing
The Gemini Developer API pricing page lists computer use as unavailable on the free tier and bills it at the selected model's regular token rates. A model offering a free text allowance does not make its computer-use tool free.[10]
For example, on November 5, 2026, Gemini 3.8 Flash standard pricing was $0.75 per million input tokens and $3.75 per million output tokens, including thinking tokens, through December 31, 2026. The listed rates from January 1, 2027 were $1.50 and $7.50 respectively.[10] These are token prices, not a fixed fee per completed task. Repeated screenshots, conversation context and generated actions contribute to an agent loop's usage.
The client environment is a separate implementation dependency. Google's reference code can use either a local browser or an external browser service; API token prices alone do not describe all the resources used by such a deployment.[7]
Continue reading
- Computer useComputer use is the capability of artificial intelligence models to control computers by viewing screens and performing mouse and keyboard actions, much like a human user would.Mentioned 7 times above
- Google AI StudioGoogle AI Studio is Google's free, web-based developer tool for prototyping with the Gemini family of models and getting a Gemini API key in seconds.Listed in See also
- Anthropic Computer UseAnthropic Computer Use is a feature of Anthropic's Claude family of large language models that lets the model operate a real computer the way a human user would: by looking at screenshots, deciding what to do…Listed in See also
- Human-in-the-loopHuman-in-the-loop (HITL) describes any arrangement in which a person is a required participant in an automated system's operating cycle rather than a bystander to it.Listed in See also
- Gemini (language model)Gemini is a family of natively multimodal large language models developed by Google DeepMind, first announced on December 6, 2023, that can reason across text, images, audio, video, and code within a single…Mentioned 27 times above
- Computer-use agentA computer-use agent (CUA) is a category of AI agent in artificial intelligence that performs tasks by directly operating a general-purpose computer's graphical user interface (GUI) the way a human does, by…Linked in the introduction
More to read
References
- ^1 ^2 ^3 ^4 ^5 ^6 ^7Google. Computer use. Gemini Developer API documentation, including supported environments, model versions and safety guidance. Accessed November 5, 2026.
- ^1 ^2 ^3 ^4 ^5 ^6 ^7 ^8 ^9Google Cloud. Computer use. Gemini Enterprise Agent Platform documentation. Accessed November 5, 2026.
- ^1 ^2Google. Gemini API release notes, October 30, 2026. General availability across Interactions and GenerateContent.
- ^1 ^2Google AI Studio (@GoogleAIStudio). Announcement of computer-use general availability. X, November 4, 2026 UTC; rendered post checked November 5, 2026.
- ^1 ^2Google DeepMind. Introducing the Gemini 2.5 Computer Use model. October 7, 2025.
- ^Google. Gemini 3.8 Flash. Model documentation, accessed November 5, 2026.
- ^1 ^2 ^3 ^4Google Gemini. Computer Use Preview. Reference implementation, README and Apache-2.0 license, accessed November 5, 2026.
- ^Tianbao Xie et al. OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. arXiv:2404.07972, submitted April 11, 2024; revised May 30, 2024.
- ^Thomas Kuntz et al. OS-Harm: A Benchmark for Measuring Safety of Computer Use Agents. arXiv:2506.14866, submitted June 17, 2025; revised October 29, 2025. NeurIPS 2025 Datasets and Benchmarks Track.
- ^1 ^2Google. Gemini Developer API pricing, computer-use and Gemini 3.8 Flash entries. Accessed November 5, 2026.
Improve this article
Add missing citations, update stale details, or write a clearer explanation. Anyone can edit: changes go live when you publish, and every edit is checked by an editor.
1 revision · v1 · 1,378 words · full history
Fact-checks are independent of edits: a reviewer re-verifies the article against its sources and stamps the date. How we verify
Research and drafting on this wiki are AI-assisted, under named human editorial standards. How AI is used here
Cite this page: AI Wiki. "Gemini Computer Use." aiwiki.ai, updated 5 Nov 2026. CC BY 4.0. https://aiwiki.ai/wiki/gemini_computer_use