;
};
export const ZoneContent = ({group, value, options = [], values = [], defaultValue, children}) => {
const optionKey = options.map(option => `${option.id}:${option.title}`).join("|");
const [activePivot, setActivePivot] = useState(defaultValue || values[0]);
const slugify = value => value.toLowerCase().replace(/[^a-z0-9]+/g, "-").replace(/^-|-$/g, "");
const resolvePivot = () => {
if (typeof window === "undefined") return defaultValue || values[0];
const params = new URLSearchParams(window.location.search);
const requested = params.get("pivots");
if (requested) {
const requestedIds = requested.split(",").map(value => value.trim()).filter(Boolean);
const match = requestedIds.find(id => values.includes(id));
if (match) return match;
}
const hash = window.location.hash.replace(/^#/, "");
if (hash) {
const match = options.find(option => option.id === hash || slugify(option.title) === hash);
if (match) return match.id;
}
try {
const stored = window.localStorage.getItem(`foundry-zone-pivot:${group}`);
if (values.includes(stored)) return stored;
} catch {
return defaultValue || values[0];
}
return defaultValue || values[0];
};
useEffect(() => {
setActivePivot(resolvePivot());
}, [group, defaultValue, values.join("|"), optionKey]);
useEffect(() => {
const onPivotChange = event => {
if (event.detail?.group === group && values.includes(event.detail.value)) {
setActivePivot(event.detail.value);
}
};
window.addEventListener("foundry-zone-pivot-change", onPivotChange);
return () => window.removeEventListener("foundry-zone-pivot-change", onPivotChange);
}, [group, values.join("|")]);
if (activePivot !== value) return null;
return <>{children};
};
Items marked (preview) in this article are currently in public preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
The computer use tool comes with significant security and privacy risks, including prompt injection attacks. For more information about intended uses, capabilities, limitations, risks, and considerations when choosing a use case, see the [Azure OpenAI transparency note](../../../responsible-ai/openai/transparency-note#risk-and-limitations-of-computer-use-preview).
Create agents that interpret screenshots and automate UI interactions like clicking, typing, and scrolling. The computer use tool uses the `computer-use-preview` Foundry model to propose actions based on visual content, enabling agents to interact with desktop and browser applications through their user interfaces.
This guide shows how to integrate the computer use tool into an application loop (screenshot → action → screenshot) by using the latest SDKs.
### Usage support
The following table shows SDK and setup support.
| Microsoft Foundry support | Python SDK | C# SDK | JavaScript SDK | Java SDK | REST API | Basic agent setup | Standard agent setup |
| ------------------------- | ---------- | ------ | -------------- | -------- | -------- | ----------------- | -------------------- |
| ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ | ✔️ |
## Prerequisites
* An Azure subscription. [Create one for free](https://azure.microsoft.com/pricing/purchase-options/azure-account?cid=msft_learn).
* A [basic or standard agent environment](../../../agents/environment-setup).
* The latest SDK package:
* **Python**: `azure-ai-projects`
* **C#/.NET**: `Azure.AI.Extensions.OpenAI`
* **TypeScript**: `@azure/ai-projects`
* **Java**: `azure-ai-agents`
* Access to the `computer-use-preview` model. See [Request access](#request-access) below.
* A virtual machine or sandboxed environment for safe testing. Don't run on machines with access to sensitive data.
## Run the maintained SDK samples (recommended)
The code snippets in this article focus on the agent and Responses API integration. For an end-to-end runnable sample that includes helper code and sample screenshots, use the SDK samples on GitHub.
* Python: [https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/ai/azure-ai-projects/samples/agents/tools](https://github.com/Azure/azure-sdk-for-python/tree/main/sdk/ai/azure-ai-projects/samples/agents/tools)
* TypeScript: [https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v2-beta/javascript/agents/tools/agentComputerUse.js](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v2-beta/javascript/agents/tools/agentComputerUse.js)
* .NET (computer use tool sample): [https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/ai/Azure.AI.Agents.Persistent/samples/Sample33\_Computer\_Use.md](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/ai/Azure.AI.Agents.Persistent/samples/Sample33_Computer_Use.md)
The SDK samples include helper utilities for screenshot capture, action execution, and image encoding. Clone the repository or copy these files to your project before running the samples.
## Request access
To access the `computer-use-preview` model, you need to register. Microsoft grants access based on eligibility criteria. If you have access to other limited access models, you still need to request access for this model.
To request access, see the [application form](https://aka.ms/oai/cuaaccess).
After Microsoft grants access, you need to create a deployment for the model.
## Code samples
Use the computer use tool on virtual machines with no access to sensitive data or critical resources. For more information about the intended uses, capabilities, limitations, risks, and considerations when choosing a use case, see the [Azure OpenAI transparency note](../../../responsible-ai/openai/transparency-note#risk-and-limitations-of-computer-use-preview).
You need the latest SDK package. The .NET SDK is currently in preview.
### Screenshot initialization for computer use tool execution
The following code sample demonstrates how to create an agent version with the computer use tool, send an initial request with a screenshot, and perform multiple iterations to complete a task. Select **Prompt Agents** to use the Azure AI Projects SDK to create a server-side prompt agent, or **Hosted Agents** to use the Agent Framework [`FoundryChatClient`](/agents/responses-api) to build an ephemeral, in-process agent.
```python theme={null}
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import PromptAgentDefinition, ComputerUsePreviewTool
# Import shared helper functions
from computer_use_util import (
SearchState,
load_screenshot_assets,
handle_computer_action_and_take_screenshot,
print_final_output,
)
"""Main function to demonstrate Computer Use Agent functionality."""
# Initialize state machine
current_state = SearchState.INITIAL
# Load screenshot assets
try:
screenshots = load_screenshot_assets()
print("Successfully loaded screenshot assets")
except FileNotFoundError:
print("Failed to load required screenshot assets. Use the maintained SDK sample on GitHub to get the helper file and images.")
exit(1)
```
### Create an agent version with the tool
```python theme={null}
# Format: "https://resource_name.ai.azure.com/api/projects/project_name"
PROJECT_ENDPOINT = "your_project_endpoint"
project = AIProjectClient(
endpoint=PROJECT_ENDPOINT,
credential=DefaultAzureCredential(),
)
computer_use_tool = ComputerUsePreviewTool(display_width=1026, display_height=769, environment="windows")
agent = project.agents.create_version(
agent_name="ComputerUseAgent",
definition=PromptAgentDefinition(
model="computer-use-preview",
instructions="""
You are a computer automation assistant.
Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.
""",
tools=[computer_use_tool],
),
description="Computer automation agent with screen interaction capabilities.",
)
print(f"Agent created (id: {agent.id}, name: {agent.name})")
```
### One iteration for the tool to process the screenshot and take the next step
```python theme={null}
openai = project.get_openai_client()
# Initial request with screenshot - start with Bing search page
response = openai.responses.create(
input=[
{
"role": "user",
"content": [
{
"type": "input_text",
"text": "I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete.",
},
{
"type": "input_image",
"image_url": screenshots["browser_search"]["url"],
"detail": "high",
}, # Start with Bing search page
],
}
],
extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}},
truncation="auto",
)
print(f"Initial response received (ID: {response.id})")
```
### Perform multiple iterations
Make sure you review each iteration and action. The following code sample shows a basic API request. After you send the initial API request, perform a loop where your application code carries out the specified action. Send a screenshot with each turn so the model can evaluate the updated state of the environment. The sample includes a maximum iteration count to prevent infinite loops, but you can adjust this as needed.
```python theme={null}
max_iterations = 10 # Allow enough iterations for completion
iteration = 0
while True:
if iteration >= max_iterations:
print(f"\nReached maximum iterations ({max_iterations}). Stopping.")
break
iteration += 1
print(f"\n--- Iteration {iteration} ---")
# Check for computer calls in the response
computer_calls = [item for item in response.output if item.type == "computer_call"]
if not computer_calls:
print_final_output(response)
break
# Process the first computer call
computer_call = computer_calls[0]
action = computer_call.action
call_id = computer_call.call_id
# Handle the action and get the screenshot info
screenshot_info, current_state = handle_computer_action_and_take_screenshot(action, current_state, screenshots)
# Regular response with just the screenshot
response = openai.responses.create(
previous_response_id=response.id,
input=[
{
"call_id": call_id,
"type": "computer_call_output",
"output": {
"type": "computer_screenshot",
"image_url": screenshot_info["url"],
},
}
],
extra_body={"agent_reference": {"name": agent.name, "type": "agent_reference"}},
truncation="auto",
)
print(f"Iteration {iteration}: response received (ID: {response.id})")
```
### Clean up
```python theme={null}
project.agents.delete_version(agent_name=agent.name, agent_version=agent.version)
print("Agent deleted")
```
### Expected output
The following example shows the expected output when running the previous code sample:
```console theme={null}
Successfully loaded screenshot assets
Agent created (id: ..., name: ComputerUseAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
Typing text "OpenAI news" - Simulating keyboard input
-> Action processed: type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
Click at (512, 384) - Simulating click on UI element
-> Assuming click on Search button when search field was populated, displaying results.
-> Action processed: click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Agent deleted
```
This sample uses [`FoundryChatClient`](/agents/responses-api) from the Microsoft Agent Framework and calls `get_computer_use_tool()` to attach the computer use preview tool. Install the package with `pip install agent-framework-foundry aiohttp`, set the `FOUNDRY_PROJECT_ENDPOINT` (point `FOUNDRY_MODEL` at a `computer-use-preview` deployment), and sign in with `az login`. The screenshot-capture loop is application-specific; see the upstream sample helper file referenced below.
```python theme={null}
import asyncio
from agent_framework import Agent
from agent_framework.foundry import FoundryChatClient
from azure.identity import AzureCliCredential
async def main() -> None:
agent = Agent(
client=FoundryChatClient(credential=AzureCliCredential()),
instructions=(
"You are a computer automation assistant. Be direct and efficient. "
"When you reach the search results page, describe the actual result titles you can see."
),
tools=[
FoundryChatClient.get_computer_use_tool(
environment="windows",
display_width=1026,
display_height=769,
)
],
)
# Replace this with your screenshot capture + action handler loop.
# See the upstream samples folder for a reference implementation.
result = await agent.run(
"Help me search for 'OpenAI news'. Type the query and submit the search."
)
print(f"Agent: {result.text}")
if __name__ == "__main__":
asyncio.run(main())
```
### Expected output
The agent issues computer-use actions (clicks, keystrokes, screenshots) until the task completes, then describes the page it reached:
```console theme={null}
Agent: I searched for "OpenAI news" in the address bar. The top results include articles from OpenAI's blog, TechCrunch, and The Verge ...
```
For a full screenshot-loop implementation, see the [Foundry provider samples](https://github.com/microsoft/agent-framework/tree/main/python/samples/02-agents/providers/foundry).
## Sample for use of an Agent with Computer Use tool
The following C# code sample demonstrates how to create an agent with the computer use tool, send an initial request with a screenshot, and perform multiple iterations to complete a task. Select **Prompt Agents** to use the Azure AI Projects SDK to create a server-side prompt agent, or **Hosted Agents** to use the Microsoft Agent Framework to build an ephemeral, in-process agent.
To enable your agent to use the computer use tool, use `ResponseTool.CreateComputerTool()` when configuring the agent's tools. This example uses synchronous code. For asynchronous usage, see the [sample code](https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/ai/Azure.AI.Extensions.OpenAI/samples/Sample10_ComputerUse.md) example in the Azure SDK for .NET repository on GitHub.
```csharp theme={null}
using System;
using System.Runtime.CompilerServices;
using Azure.AI.Projects;
using Azure.AI.Extensions.OpenAI;
using Azure.Identity;
class ComputerUseDemo
{
// Format: "https://resource_name.ai.azure.com/api/projects/project_name"
private const string ProjectEndpoint = "your_project_endpoint";
// Read image files using `ReadImageFile` method.
private static BinaryData ReadImageFile(string name, [CallerFilePath] string pth = "")
{
var dirName = Path.GetDirectoryName(pth) ?? "";
return new BinaryData(File.ReadAllBytes(Path.Combine(dirName, name)));
}
// Create a helper method to parse the ComputerTool outputs and to respond
// to Agents queries with new screenshots. Note that throughout
// this sample the media type for image is set. Agents support `image/jpeg`,
// `image/png`, `image/gif` and `image/webp` media types.
private static string ProcessComputerUseCall(ComputerCallResponseItem item, string oldScreenshot)
{
string currentScreenshot = "browser_search";
switch (item.Action.Kind)
{
case ComputerCallActionKind.Type:
Console.WriteLine($" Typing text \"{item.Action.TypeText}\" - Simulating keyboard input");
currentScreenshot = "search_typed";
break;
case ComputerCallActionKind.KeyPress:
HashSet codes = new(item.Action.KeyPressKeyCodes);
if (codes.Contains("Return") || codes.Contains("ENTER"))
{
// If we have typed the value to the search field, go to search results.
if (string.Equals(oldScreenshot, "search_typed"))
{
Console.WriteLine(" -> Detected ENTER key press, when search field was populated, displaying results.");
currentScreenshot = "search_results";
}
else
{
Console.WriteLine(" -> Detected ENTER key press, on results or unpopulated search, do nothing.");
currentScreenshot = oldScreenshot;
}
}
else
{
Console.WriteLine($" Key press: {item.Action.KeyPressKeyCodes.Aggregate("", (agg, next) => agg + "+" + next)} - Simulating key combination");
}
break;
case ComputerCallActionKind.Click:
Console.WriteLine($" Click at ({item.Action.ClickCoordinates.Value.X}, {item.Action.ClickCoordinates.Value.Y}) - Simulating click on UI element");
if (string.Equals(oldScreenshot, "search_typed"))
{
Console.WriteLine(" -> Assuming click on Search button when search field was populated, displaying results.");
currentScreenshot = "search_results";
}
else
{
Console.WriteLine(" -> Assuming click on Search on results or when search was not populated, do nothing.");
currentScreenshot = oldScreenshot;
}
break;
case ComputerCallActionKind.Drag:
string pathStr = item.Action.DragPath.ToArray().Select(p => $"{p.X}, {p.Y}").Aggregate("", (agg, next) => $"{agg} -> {next}");
Console.WriteLine($" Drag path: {pathStr} - Simulating drag operation");
break;
case ComputerCallActionKind.Scroll:
Console.WriteLine($" Scroll at ({item.Action.ScrollCoordinates.Value.X}, {item.Action.ScrollCoordinates.Value.Y}) - Simulating scroll action");
break;
case ComputerCallActionKind.Screenshot:
Console.WriteLine(" Taking screenshot - Capturing current screen state");
break;
default:
break;
}
Console.WriteLine($" -> Action processed: {item.Action.Kind}");
return currentScreenshot;
}
public static void Main()
{
// Create project client
AIProjectClient projectClient = new(endpoint: new Uri(ProjectEndpoint), tokenProvider: new DefaultAzureCredential());
// Read in three example screenshots and place them into a dictionary.
Dictionary screenshots = new() {
{ "browser_search", ReadImageFile("Assets/cua_browser_search.png")},
{ "search_typed", ReadImageFile("Assets/cua_search_typed.png")},
{ "search_results", ReadImageFile("Assets/cua_search_results.png")},
};
// Create a PromptAgentDefinition with ComputerTool.
DeclarativeAgentDefinition agentDefinition = new(model: "computer-use-preview")
{
Instructions = "You are a computer automation assistant.\n\n" +
"Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.",
Tools = {
ResponseTool.CreateComputerTool(
environment: new ComputerToolEnvironment("windows"),
displayWidth: 1026,
displayHeight: 769
),
}
};
AgentVersion agentVersion = projectClient.AgentAdministrationClient.CreateAgentVersion(
agentName: "myAgent",
options: new(agentDefinition)
);
// Create an `ResponseResult` using `ResponseItem`, containing two `ResponseContentPart`:
// one with the image and another with the text. In the loop, request Agent
// while it is continuing to browse web. Finally, print the tool output message.
ProjectResponsesClient responseClient = projectClient.ProjectOpenAIClient.GetProjectResponsesClientForAgent(agentVersion.Name);
CreateResponseOptions responseOptions = new()
{
TruncationMode = ResponseTruncationMode.Auto,
InputItems =
{
ResponseItem.CreateUserMessageItem(
[
ResponseContentPart.CreateInputTextPart("I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete."),
ResponseContentPart.CreateInputImagePart(imageBytes: screenshots["browser_search"], imageBytesMediaType: "image/png", imageDetailLevel: ResponseImageDetailLevel.High)
]),
},
};
bool computerUseCalled = false;
string currentScreenshot = "browser_search";
int limitIteration = 10;
ResponseResult response;
do
{
response = responseClient.CreateResponse(responseOptions);
computerUseCalled = false;
responseOptions.InputItems.Clear();
responseOptions.PreviousResponseId = response.Id;
foreach (ResponseItem responseItem in response.OutputItems)
{
responseOptions.InputItems.Add(responseItem);
if (responseItem is ComputerCallResponseItem computerCall)
{
currentScreenshot = ProcessComputerUseCall(computerCall, currentScreenshot);
responseOptions.InputItems.Add(ResponseItem.CreateComputerCallOutputItem(callId: computerCall.CallId, output: ComputerCallOutput.CreateScreenshotOutput(screenshotImageBytes: screenshots[currentScreenshot], screenshotImageBytesMediaType: "image/png")));
computerUseCalled = true;
}
}
limitIteration--;
} while (computerUseCalled && limitIteration > 0);
Console.WriteLine(response.GetOutputText());
// Clean up resources by deleting Agent.
projectClient.AgentAdministrationClient.DeleteAgentVersion(agentName: agentVersion.Name, agentVersion: agentVersion.Version);
}
}
```
### Expected output
The following example shows the expected output when running the previous code sample:
```console theme={null}
Agent created (id: ..., name: myAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
Typing text "OpenAI news" - Simulating keyboard input
-> Action processed: Type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
Click at (512, 384) - Simulating click on UI element
-> Assuming click on Search button when search field was populated, displaying results.
-> Action processed: Click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Agent deleted
```
This sample uses the Microsoft Agent Framework and calls `AsAIAgent(...)` on `AIProjectClient` together with `FoundryAITool.CreateComputerTool(...)` from `Microsoft.Agents.AI.Foundry` to give the agent the computer use tool. Install the `Microsoft.Agents.AI.Foundry` and `Azure.AI.Projects` packages, set the `AZURE_AI_PROJECT_ENDPOINT` and `AZURE_AI_COMPUTER_USE_DEPLOYMENT_NAME` environment variables, and sign in with `az login`. This sample omits the screenshot helpers — see the full sample for the action loop and asset utilities.
```csharp theme={null}
using Azure.AI.Projects;
using Azure.Identity;
using Microsoft.Agents.AI;
using Microsoft.Agents.AI.Foundry;
using Microsoft.Extensions.AI;
using OpenAI.Responses;
string endpoint = Environment.GetEnvironmentVariable("AZURE_AI_PROJECT_ENDPOINT")
?? throw new InvalidOperationException("AZURE_AI_PROJECT_ENDPOINT is not set.");
string deploymentName = Environment.GetEnvironmentVariable("AZURE_AI_COMPUTER_USE_DEPLOYMENT_NAME") ?? "computer-use-preview";
AIProjectClient projectClient = new(new Uri(endpoint), new DefaultAzureCredential());
using IHostedFileClient fileClient = projectClient.GetProjectOpenAIClient().AsIHostedFileClient();
AIAgent agent = projectClient.AsAIAgent(
model: deploymentName,
name: "ComputerAgent",
instructions: "You are a computer automation assistant.",
tools: [FoundryAITool.CreateComputerTool(ComputerToolEnvironment.Browser, 1026, 769)]);
// Upload pre-captured screenshots that simulate browser state transitions.
// (See the full sample for ComputerUseUtil implementation.)
Dictionary screenshots = await ComputerUseUtil.UploadScreenshotAssetsAsync(fileClient);
ChatClientAgentRunOptions runOptions = new()
{
ChatOptions = new ChatOptions
{
RawRepresentationFactory = (_) => new CreateResponseOptions { TruncationMode = ResponseTruncationMode.Auto },
}
};
ChatMessage message = new(ChatRole.User,
[
new TextContent("Search for 'OpenAI news'. Type it and submit. Once you see results, the task is complete."),
new AIContent { RawRepresentation = ResponseContentPart.CreateInputImagePart(imageFileId: screenshots["browser_search"], imageDetailLevel: ResponseImageDetailLevel.High) }
]);
AgentSession session = await agent.CreateSessionAsync();
AgentResponse response = await agent.RunAsync(message, session: session, options: runOptions);
// Loop: parse computer call actions from response, simulate them, return new screenshots.
for (int i = 0; i < 10; i++)
{
ComputerCallResponseItem? computerCall = response.Messages
.SelectMany(m => m.Contents)
.Select(c => c.RawRepresentation as ComputerCallResponseItem)
.FirstOrDefault(item => item is not null);
if (computerCall is null) break;
(_, string fileId) = await ComputerUseUtil.GetScreenshotAsync(computerCall.Action, default, screenshots);
AIContent callOutput = new()
{
RawRepresentation = new ComputerCallOutputResponseItem(
computerCall.CallId,
output: ComputerCallOutput.CreateScreenshotOutput(screenshotImageFileId: fileId))
};
response = await agent.RunAsync([new ChatMessage(ChatRole.User, [callOutput])], session: session, options: runOptions);
}
await ComputerUseUtil.EnsureDeleteScreenshotAssetsAsync(fileClient, screenshots);
Console.WriteLine($"Response: {response.Text}");
```
### Expected output
After the action loop completes, the final agent reply describes the page it reached:
```console theme={null}
Response: I searched for "OpenAI news" in the address bar. The top results include articles from OpenAI's blog, TechCrunch, and The Verge ...
```
For the full screenshot helper implementation and end-to-end action loop, see [Agent\_Step15\_ComputerUse](https://github.com/microsoft/agent-framework/tree/main/dotnet/samples/02-agents/AgentProviders/foundry/Agent_Step15_ComputerUse).
## Sample for use of an Agent with Computer Use tool
The following TypeScript code sample demonstrates how to create an agent version with the computer use tool, send an initial request with a screenshot, and perform multiple iterations to complete a task. For a JavaScript example, see the [sample code](https://github.com/Azure/azure-sdk-for-js/blob/main/sdk/ai/ai-projects/samples/v2-beta/javascript/agents/tools/agentComputerUse.js) in the Azure SDK for JavaScript repository on GitHub.
```typescript theme={null}
import { DefaultAzureCredential } from "@azure/identity";
import { AIProjectClient } from "@azure/ai-projects";
import {
SearchState,
loadScreenshotAssets,
handleComputerActionAndTakeScreenshot,
printFinalOutput,
type ComputerAction,
} from "./computerUseUtil.js";
// Format: "https://resource_name.ai.azure.com/api/projects/project_name"
const PROJECT_ENDPOINT = "your_project_endpoint";
export async function main(): Promise {
// Initialize state machine
let currentState = SearchState.INITIAL;
// Load screenshot assets
const screenshots = loadScreenshotAssets();
console.log("Successfully loaded screenshot assets");
// Create AI Project client
const project = new AIProjectClient(PROJECT_ENDPOINT, new DefaultAzureCredential());
const openai = project.getOpenAIClient();
console.log("Creating Computer Use Agent...");
const agent = await project.agents.createVersion("ComputerUseAgent", {
kind: "prompt" as const,
model: "computer-use-preview",
instructions: `
You are a computer automation assistant.
Be direct and efficient. When you reach the search results page, read and describe the actual search result titles and descriptions you can see.
`.trim(),
tools: [
{
type: "computer_use_preview",
display_width: 1026,
display_height: 769,
environment: "windows" as const,
},
],
});
console.log(`Agent created (id: ${agent.id}, name: ${agent.name}, version: ${agent.version})`);
// Initial request with screenshot - start with Bing search page
console.log(
"Starting computer automation session (initial screenshot: cua_browser_search.png)...",
);
let response = await openai.responses.create(
{
input: [
{
role: "user" as const,
content: [
{
type: "input_text",
text: "I need you to help me search for 'OpenAI news'. Please type 'OpenAI news' and submit the search. Once you see search results, the task is complete.",
},
{
type: "input_image",
image_url: screenshots.browser_search.url,
detail: "high",
},
],
},
],
truncation: "auto",
},
{
body: { agent: { name: agent.name, type: "agent_reference" } },
},
);
console.log(`Initial response received (ID: ${response.id})`);
// Main interaction loop with deterministic completion
const maxIterations = 10; // Allow enough iterations for completion
let iteration = 0;
while (iteration < maxIterations) {
iteration++;
console.log(`\n--- Iteration ${iteration} ---`);
// Check for computer calls in the response
const computerCalls = response.output.filter((item) => item.type === "computer_call");
if (computerCalls.length === 0) {
printFinalOutput({
output: response.output,
status: response.status ?? "",
});
break;
}
// Process the first computer call
const computerCall = computerCalls[0];
const action: ComputerAction = computerCall.action;
const callId: string = computerCall.call_id;
console.log(`Processing computer call (ID: ${callId})`);
// Handle the action and get the screenshot info
const [screenshotInfo, updatedState] = handleComputerActionAndTakeScreenshot(
action,
currentState,
screenshots,
);
currentState = updatedState;
console.log(`Sending action result back to agent (using ${screenshotInfo.filename})...`);
// Regular response with just the screenshot
response = await openai.responses.create(
{
previous_response_id: response.id,
input: [
{
call_id: callId,
type: "computer_call_output",
output: {
type: "computer_screenshot",
image_url: screenshotInfo.url,
},
},
],
truncation: "auto",
},
{
body: { agent: { name: agent.name, type: "agent_reference" } },
},
);
console.log(`Follow-up response received (ID: ${response.id})`);
}
if (iteration >= maxIterations) {
console.log(`\nReached maximum iterations (${maxIterations}). Stopping.`);
}
// Clean up resources
console.log("\nCleaning up...");
await project.agents.deleteVersion(agent.name, agent.version);
console.log("Agent deleted");
console.log("\nComputer Use Agent sample completed!");
}
main().catch((err) => {
console.error("The sample encountered an error:", err);
});
```
### Expected output
The following example shows the expected output when running the previous code sample:
```console theme={null}
Successfully loaded screenshot assets
Creating Computer Use Agent...
Agent created (id: ..., name: ComputerUseAgent, version: 1)
Starting computer automation session (initial screenshot: cua_browser_search.png)...
Initial response received (ID: ...)
--- Iteration 1 ---
Processing computer call (ID: ...)
Typing text "OpenAI news" - Simulating keyboard input
-> Action processed: type
Sending action result back to agent (using cua_search_typed.png)...
Follow-up response received (ID: ...)
--- Iteration 2 ---
Processing computer call (ID: ...)
Click at (512, 384) - Simulating click on UI element
-> Assuming click on Search button when search field was populated, displaying results.
-> Action processed: click
Sending action result back to agent (using cua_search_results.png)...
Follow-up response received (ID: ...)
OpenAI news - Latest Updates
Cleaning up...
Agent deleted
Computer Use Agent sample completed!
```
## Use computer use in a Java agent
Add the dependency to your `pom.xml`:
```xml theme={null}
com.azureazure-ai-agents2.2.0
```
### Create a computer use agent
```java theme={null}
import com.azure.ai.agents.AgentsClient;
import com.azure.ai.agents.AgentsClientBuilder;
import com.azure.ai.agents.ResponsesClient;
import com.azure.ai.agents.models.*;
import com.azure.identity.DefaultAzureCredentialBuilder;
import com.openai.models.responses.Response;
import com.openai.models.responses.ResponseCreateParams;
import java.util.Collections;
public class ComputerUseExample {
// Format: "https://resource_name.ai.azure.com/api/projects/project_name"
private static final String PROJECT_ENDPOINT = "your_project_endpoint";
public static void main(String[] args) {
AgentsClientBuilder builder = new AgentsClientBuilder()
.credential(new DefaultAzureCredentialBuilder().build())
.endpoint(PROJECT_ENDPOINT);
AgentsClient agentsClient = builder.buildAgentsClient();
ResponsesClient responsesClient = builder.buildResponsesClient();
// Create computer use tool
ComputerUsePreviewTool tool = new ComputerUsePreviewTool(
ComputerEnvironment.WINDOWS,
1024,
768
);
// Create agent with computer use tool
PromptAgentDefinition agentDefinition = new PromptAgentDefinition("computer-use-preview")
.setInstructions("You are a computer automation assistant.")
.setTools(Collections.singletonList(tool));
AgentVersionDetails agent = agentsClient.createAgentVersion("computer-use-agent", agentDefinition);
System.out.printf("Agent created: %s (version %s)%n", agent.getName(), agent.getVersion());
// Create a response with initial screenshot
AgentReference agentReference = new AgentReference(agent.getName())
.setVersion(agent.getVersion());
Response response = responsesClient.createAzureResponse(
new AzureCreateResponseOptions().setAgentReference(agentReference),
ResponseCreateParams.builder()
.input("Open the browser and navigate to microsoft.com"));
System.out.println("Response: " + response.output());
// The response will contain computer_call items with actions
// to execute. Process each action, take screenshots, and
// send results back using responsesClient.createAzureResponse()
// with the previousResponseId and computer call output.
// Clean up
agentsClient.deleteAgentVersion(agent.getName(), agent.getVersion());
}
}
```
For a complete computer use loop with screenshot handling, see the [ComputerUseSync.java sample](https://github.com/Azure/azure-sdk-for-java/tree/main/sdk/ai/azure-ai-agents/src/samples/java/com/azure/ai/agents/tools/ComputerUseSync.java).
## Use computer use with the REST API
Get an access token:
```bash theme={null}
export AGENT_TOKEN=$(az account get-access-token --scope "https://ai.azure.com/.default" --query accessToken -o tsv)
```
### Create an agent with computer use
```bash theme={null}
curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/agents?api-version=v1" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AGENT_TOKEN" \
-d '{
"name": "computer-use-agent",
"definition": {
"kind": "prompt",
"model": "computer-use-preview",
"instructions": "You are a computer automation assistant.",
"tools": [
{
"type": "computer_use_preview",
"environment": "windows",
"display_width": 1024,
"display_height": 768
}
]
}
}'
```
### Generate a response
```bash theme={null}
curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/responses" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AGENT_TOKEN" \
-d '{
"agent_reference": {"type": "agent_reference", "name": "computer-use-agent"},
"input": "Open the browser and navigate to microsoft.com"
}'
```
The response includes `computer_call` output items with actions to execute. Process each action, capture screenshots, and send results back using the responses endpoint with `previous_response_id`.
### Submit action results with screenshot
After executing the computer action (for example, click or type), capture a screenshot and send it back:
```bash theme={null}
curl -X POST "$FOUNDRY_PROJECT_ENDPOINT/openai/v1/responses" \
-H "Content-Type: application/json" \
-H "Authorization: Bearer $AGENT_TOKEN" \
-d '{
"agent_reference": {"type": "agent_reference", "name": "computer-use-agent"},
"previous_response_id": "",
"input": [
{
"type": "computer_call_output",
"call_id": "",
"output": {
"type": "computer_screenshot",
"image_url": "data:image/png;base64,"
}
}
]
}'
```
Replace ``, ``, and `` with values from the previous response. Repeat this cycle until the model returns a text response instead of a `computer_call`.
### Clean up
```bash theme={null}
curl -X DELETE "$FOUNDRY_PROJECT_ENDPOINT/agents/computer-use-agent?api-version=v1" \
-H "Authorization: Bearer $AGENT_TOKEN"
```
## What you can do with the computer use tool
After you integrate the request-and-response loop (screenshot -> action -> screenshot), the computer use tool can help an agent:
* Propose UI actions such as clicking, typing, scrolling, and requesting a new screenshot.
* Adapt to UI changes by re-evaluating the latest screenshot after each action.
* Work across browser and desktop UI, depending on how you host your sandboxed environment.
The tool doesn't directly control a device. Your application executes each requested action and returns an updated screenshot.
## Differences between browser automation and computer use
The following table lists some of the differences between the computer use tool and [browser automation](/tools-and-knowledge/browser-automation) tool.
| Feature | Browser Automation | Computer use tool |
| ----------------------------------- | ------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------- |
| Model support | All GPT models | `Computer-use-preview` model only |
| Can I visualize what's happening? | No | Yes |
| How it understands the screen | Parses the HTML or XML pages into DOM documents | Raw pixel data from screenshots |
| How it acts | A list of actions provided by the model | Virtual keyboard and mouse |
| Is it multistep? | Yes | Yes |
| Interfaces | Browser | Computer and browser |
| Do I need to bring my own resource? | Your own Playwright resource with the keys stored as a connection. | No additional resource required but we highly recommend running this tool in a sandboxed environment. |
### When to use each tool
**Choose computer use when you need to**:
* Interact with desktop applications beyond the browser
* Visualize what the agent sees through screenshots
* Work in environments where DOM parsing isn't available
**Choose browser automation when you need to**:
* Perform web-only interactions without limited access requirements
* Use any GPT model (not limited to `computer-use-preview`)
* Avoid managing screenshot capture and action execution loops
## Regional support
To use the computer use tool, you need a [computer use model](/models/models-sold-directly-by-azure#computer-use-preview) deployment. The computer use model is available in the following regions:
| Region | Status |
| --------------- | --------- |
| `eastus2` | Available |
| `swedencentral` | Available |
| `southindia` | Available |
## Understanding the computer use integration
When working with the computer use tool, integrate it into your application by performing the following steps:
1. Send a request to the model that includes a call to the computer use tool, the display size, and the environment. You can also include a screenshot of the initial state of the environment in the first API request.
2. Receive a response from the model. If the response has action items, those items contain suggested actions to make progress toward the specified goal. For example, an action might be `screenshot` so the model can assess the current state with an updated screenshot, or `click` with X/Y coordinates indicating where the mouse should be moved.
3. Execute the action by using your application code on your computer or browser environment.
4. After executing the action, capture the updated state of the environment as a screenshot.
5. Send a new request with the updated state as a `tool_call_output`, and repeat this loop until the model stops requesting actions or you decide to stop.
Before using the tool, set up an environment that can capture screenshots and execute the recommended actions by the agent. For safety reasons, use a sandboxed environment, such as Playwright.
## Manage conversation history
Use the `previous_response_id` parameter to link the current request to the previous response. Use this parameter when you don't want to send the full conversation history with each call.
If you don't use this parameter, make sure to include all the items returned in the response output of the previous request in your inputs array. This requirement includes reasoning items if present.
## Safety checks and security considerations
Computer use carries substantial security and privacy risks and user responsibility. Both errors in judgment by the AI and the presence of malicious or confusing instructions on web pages, desktops, or other operating environments that the AI encounters might cause it to execute commands you or others don't intend. These risks could compromise the security of your or other users’ browsers, computers, and any accounts to which AI has access, including personal, financial, or enterprise systems.
Use the computer use tool on virtual machines with no access to sensitive data or critical resources. For more information about the intended uses, capabilities, limitations, risks, and considerations when choosing a use case, see the [Azure OpenAI transparency note](../../../responsible-ai/openai/transparency-note#risk-and-limitations-of-computer-use-preview).
The API has safety checks to help protect against prompt injection and model mistakes. These checks include:
**Malicious instruction detection**: The system evaluates the screenshot image and checks if it contains adversarial content that might change the model's behavior.
**Irrelevant domain detection**: The system evaluates the `current_url` parameter (if provided) and checks if the current domain is relevant given the conversation history.
**Sensitive domain detection**: The system checks the `current_url` parameter (if provided) and raises a warning when it detects the user is on a sensitive domain.
If one or more of the preceding checks are triggered, the model raises a safety check when it returns the next `computer_call` by using the `pending_safety_checks` parameter.
```json theme={null}
"output": [
{
"type": "reasoning",
"id": "rs_67cb...",
"summary": [
{
"type": "summary_text",
"text": "Exploring 'File' menu option."
}
]
},
{
"type": "computer_call",
"id": "cu_67cb...",
"call_id": "call_nEJ...",
"action": {
"type": "click",
"button": "left",
"x": 135,
"y": 193
},
"pending_safety_checks": [
{
"id": "cu_sc_67cb...",
"code": "malicious_instructions",
"message": "We've detected instructions that may cause your application to perform malicious or unauthorized actions. Please acknowledge this warning if you'd like to proceed."
}
],
"status": "completed"
}
]
```
You need to pass the safety checks back as `acknowledged_safety_checks` in the next request to proceed.
```json theme={null}
"input":[
{
"type": "computer_call_output",
"call_id": "",
"acknowledged_safety_checks": [
{
"id": "",
"code": "malicious_instructions",
"message": "We've detected instructions that may cause your application to perform malicious or unauthorized actions. Please acknowledge this warning if you'd like to proceed."
}
],
"output": {
"type": "computer_screenshot",
"image_url": ""
}
}
]
```
## Safety check handling
In all cases where `pending_safety_checks` are returned, hand over actions to the end user to confirm proper model behavior and accuracy.
`malicious_instructions` and `irrelevant_domain`: End users should review model actions and confirm that the model behaves as intended.
`sensitive_domain`: Ensure an end user actively monitors the model actions on these sites. The exact implementation of this "watch mode" can vary by application, but a potential example could be collecting user impression data on the site to make sure there's active end user engagement with the application.
## Troubleshooting
| Issue | Cause | Resolution |
| --------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| You don't see a `computer_call` in the response. | The agent isn't configured with the computer use tool, the deployment isn't a computer use model, or the prompt doesn't require UI interaction. | Confirm the agent has a `computer_use_preview` tool, your deployment is the `computer-use-preview` model, and your prompt requires a UI action (type, click, or screenshot). |
| The sample code fails with missing helper files or screenshots. | The snippets reference helper utilities and sample images that aren't part of this documentation repo. | Run the maintained SDK samples in the "Run the maintained SDK samples" section, or copy the helper file and sample images from the SDK repo into your project. |
| The loop stops at the iteration limit. | The task needs more turns, or the app isn't applying the actions the model requests. | Increase the iteration limit, and verify that your code executes the requested action and sends a new screenshot after each turn. |
| You receive `pending_safety_checks`. | The service detected a potential security risk (for example, prompt injection or a sensitive domain). | Pause automation, require an end user to review the request, and only continue after you send `acknowledged_safety_checks` with the next `computer_call_output`. |
| The model repeats "take a screenshot" without making progress. | The screenshot isn't updating, is low quality, or doesn't show the relevant UI state. | Send a fresh screenshot after each action and use a higher-detail image when needed. Ensure the screenshot includes the relevant UI. |
| Access denied when requesting `computer-use-preview` model. | You haven't registered for access or access hasn't been granted. | Submit the [application form](https://aka.ms/oai/cuaaccess) and wait for approval. Check your email for confirmation. |
| Screenshot encoding errors. | Image format not supported or base64 encoding issue. | Use PNG or JPEG format. Ensure proper base64 encoding without corruption. Check image dimensions match `display_width` and `display_height`. |
| Actions execute on wrong coordinates. | Screen resolution mismatch between screenshot and actual display. | Ensure `display_width` and `display_height` in `ComputerUsePreviewTool` match your actual screen resolution. |
| Model hallucinates UI elements. | Screenshot quality too low or UI changed between turns. | Use higher resolution screenshots. Send fresh screenshots immediately after each action. Reduce delay between action and screenshot. |
## Related content
* [Tool best practices for agents](/agents/tool-best-practice)
* [Browser automation tool](/tools-and-knowledge/browser-automation)
* [Set up your agent environment](../../../agents/environment-setup)
* [Get started with Foundry agents](/get-started/get-started-code)
* [Computer use risks and limitations](../../../responsible-ai/openai/transparency-note#risk-and-limitations-of-computer-use-preview)