Quickstart: Use vision-enabled chats with the Azure OpenAI in Microsoft Foundry Models

Use this article to get started using Azure OpenAI to deploy and use the GPT-4 Turbo with Vision model or other vision-enabled models.

Quickstart: Use images in your AI chats

Get started using images in your chats with Azure OpenAI in Microsoft Foundry Models.

This article refers to the Microsoft Foundry (new) portal.

• Extra usage fees might apply when using chat completion models with vision functionality.
• When uploading images, there’s a limit of 10 images per chat request.
• Always set max_tokens in your API requests, or responses might be truncated.

Use this article to get started using Microsoft Foundry to deploy and test a chat completion model with image understanding.

Prerequisites

• An Azure subscription. Create one for free.
• Once you have your Azure subscription, create an Azure OpenAI resource . For more information about resource creation, see the resource deployment guide.
• A Foundry project with your Azure OpenAI resource added as a connection.

Prepare your media

You need an image to complete this quickstart. You can use this sample image or any other image you have available.

Go to Foundry

If you already have a vision-capable model deployed, skip to Start a chat session to analyze images.

1. Browse to Foundry and sign in with the credentials associated with your Azure OpenAI resource. During or after the sign-in workflow, select the appropriate directory, Azure subscription, and Azure OpenAI resource.
2. Select the project you’d like to work in.
3. On the left nav menu, select Models + endpoints and select + Deploy model.
4. Choose an image-capable deployment by selecting model name: gpt-4o or gpt-4o-mini. In the window that appears, select a name and deployment type. Make sure your Azure OpenAI resource is connected. For more information about model deployment, see the resource deployment guide.
5. Select Deploy.
6. Next, select your new model and select Open in playground. In the chat playground, the deployment you created should be selected in the Deployment dropdown.

Playground

In this chat session, you instruct the assistant to aid you in understanding images that you input.

For general help with assistant setup, chat sessions, settings, and panels, refer to the Chat quickstart.

Start a chat session to analyze images

In this chat session, you’re instructing the assistant to aid in understanding images that you input.

1. To start, make sure your image-capable deployment is selected in the Deployment dropdown.
2. In the context text box on the Setup panel, provide this prompt to guide the assistant: "You're an AI assistant that helps people find information." Or, you can tailor the prompt to your image or scenario.

We recommend you update the System Message to be specific to the task in order to avoid unhelpful responses from the model.

1. Select Apply changes to save your changes.
2. In the chat session pane, select the attachment button and then Upload image. Choose your image.
3. Add the following prompt in the chat field: Describe this image, and then select the send icon to submit it.
4. The send icon is replaced by a stop button. If you select it, the assistant stops processing your request. For this quickstart, let the assistant finish its reply.
5. The assistant replies with a description of the image.
6. Ask a follow-up question related to the analysis of your image. You could enter, "What should I highlight about this image to my insurance company?".
7. You should receive a relevant response similar to what’s shown here:
   When reporting the incident to your insurance company, you should highlight the following key points from the image:  
      
   1. **Location of Damage**: Clearly state that the front end of the car, particularly the driver's side, is damaged. Mention the crumpled hood, broken front bumper, and the damaged left headlight.  
      
   2. **Point of Impact**: Indicate that the car has collided with a guardrail, which may suggest that no other vehicles were involved in the accident.  
      
   3. **Condition of the Car**: Note that the damage seems to be concentrated on the front end, and there is no visible damage to the windshield or rear of the car from this perspective.  
      
   4. **License Plate Visibility**: Mention that the license plate is intact and can be used for identification purposes.  
      
   5. **Environment**: Report that the accident occurred near a roadside with a guardrail, possibly in a rural or semi-rural area, which might help in establishing the accident location and context.  
      
   6. **Other Observations**: If there were any other circumstances or details not visible in the image that may have contributed to the accident, such as weather conditions, road conditions, or any other relevant information, be sure to include those as well.  
      
   Remember to be factual and descriptive, avoiding speculation about the cause of the accident, as the insurance company will conduct its own investigation.
   

View and export code

At any point in the chat session, you can enable the Show raw JSON switch at the top of the chat window to see the conversation formatted as JSON. Heres’ what it looks like at the beginning of the quickstart chat session:

[
	{
		"role": "system",
		"content": [
			"You are an AI assistant that helps people find information."
		]
	},
]

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

• Azure portal
• Azure CLI

Use this article to get started using the Azure OpenAI REST APIs to deploy and use vision-enabled chat models.

Prerequisites

• An Azure subscription. Create one for free.
• Python 3.8 or later version.
• The following Python libraries: requests, json.
• An Azure OpenAI in Microsoft Foundry Models resource with a vision-enabled model deployed. See Model availability for available regions. For more information about resource creation, see the resource deployment guide.

It is currently not supported to turn off content filtering for the GPT-4 Turbo with Vision model.

Retrieve key and endpoint

To successfully call the Azure OpenAI APIs, you need the following information about your Azure OpenAI resource:

Variable	Name	Value
Endpoint	api_base	The endpoint value is located under Keys and Endpoint for your resource in the Azure portal. You can also find the endpoint via the Deployments page in Foundry portal. An example endpoint is: https://docs-test-001.openai.azure.com/.
Key	api_key	The key value is also located under Keys and Endpoint for your resource in the Azure portal. Azure generates two keys for your resource. You can use either value.

Go to your resource in the Azure portal. On the navigation pane, select Keys and Endpoint under Resource Management. Copy the Endpoint value and an access key value. You can use either the KEY 1 or KEY 2 value. Having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Create a new Python application

Create a new Python file named quickstart.py. Open the new file in your preferred editor or IDE.

1. Replace the contents of quickstart.py with the following code.

   # Packages required:
   import requests 
   import json 
   
   api_base = '<your_azure_openai_endpoint>' 
   deployment_name = '<your_deployment_name>'
   API_KEY = '<your_azure_openai_key>'
   
   base_url = f"{api_base}openai/deployments/{deployment_name}" 
   headers = {   
       "Content-Type": "application/json",   
       "api-key": API_KEY 
   } 
   
   # Prepare endpoint, headers, and request body 
   endpoint = f"{base_url}/chat/completions?api-version=2023-12-01-preview" 
   data = { 
       "messages": [ 
           { "role": "system", "content": "You are a helpful assistant." }, 
           { "role": "user", "content": [  
               { 
                   "type": "text", 
                   "text": "Describe this picture:" 
               },
               { 
                   "type": "image_url",
                   "image_url": {
                       "url": "<image URL>"
                   }
               }
           ] } 
       ], 
       "max_tokens": 2000 
   }   
   
   # Make the API call   
   response = requests.post(endpoint, headers=headers, data=json.dumps(data))   
   
   print(f"Status Code: {response.status_code}")   
   print(response.text)
   

2. Make the following changes:

   1. Enter your endpoint URL and key in the appropriate fields.
   2. Enter your model deployment name in the appropriate field.
   3. Change the value of the "image" field to the publicly accessible URL of your image.

You can also use a base 64 encoded image data instead of a URL. For more information, see the Vision chats how-to guide.

1. Run the application with the python command:

   python quickstart.py
   

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

• Azure portal
• Azure CLI

Use this article to get started using the Azure OpenAI Python SDK to deploy and use a vision-enabled chat model.

Library source code | Package (PyPi) |

Prerequisites

• An Azure subscription. Create one for free.
• Python 3.8 or later version.
• An Azure OpenAI in Microsoft Foundry Models resource with a vision-enabled chat model deployed. See Model availability for available regions. For more information about resource creation, see the resource deployment guide.

Microsoft Entra ID prerequisites

For the recommended keyless authentication with Microsoft Entra ID, you need to:

• Install the Azure CLI used for keyless authentication with Microsoft Entra ID.
• Assign the Cognitive Services User role to your user account. You can assign roles in the Azure portal under Access control (IAM) > Add role assignment.

Set up

Install the OpenAI Python client library with:

pip install openai

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Retrieve key and endpoint

To successfully make a call against Azure OpenAI, you need an endpoint and a key.

Variable name	Value
ENDPOINT	The service endpoint can be found in the Keys & Endpoint section when examining your resource from the Azure portal. Alternatively, you can find the endpoint via the Deployments page in Microsoft Foundry portal. An example endpoint is: https://docs-test-001.openai.azure.com/.
API-KEY	This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.

Go to your resource in the Azure portal. The Keys & Endpoint section can be found in the Resource Management section. Copy your endpoint and access key as you’ll need both for authenticating your API calls. You can use either KEY1 or KEY2. Always having two keys allows you to securely rotate and regenerate keys without causing a service disruption.

Environment variables

Create and assign persistent environment variables for your key and endpoint.

Command Line

    setx AZURE_OPENAI_API_KEY "REPLACE_WITH_YOUR_KEY_VALUE_HERE" 
    setx AZURE_OPENAI_ENDPOINT "REPLACE_WITH_YOUR_ENDPOINT_HERE" 

Create a new Python application

Create a new Python file named quickstart.py. Open the new file in your preferred editor or IDE.

1. Replace the contents of quickstart.py with the following code.

   import os
   from openai import AzureOpenAI
   
   api_base = os.getenv("AZURE_OPENAI_ENDPOINT")
   api_key= os.getenv("AZURE_OPENAI_API_KEY")
   deployment_name = '<your_deployment_name>'
   api_version = '2023-12-01-preview' # this might change in the future
   
   client = AzureOpenAI(
       api_key=api_key,  
       api_version=api_version,
       base_url=f"{api_base}/openai/deployments/{deployment_name}"
   )
   
   response = client.chat.completions.create(
       model=deployment_name,
       messages=[
           { "role": "system", "content": "You are a helpful assistant." },
           { "role": "user", "content": [  
               { 
                   "type": "text", 
                   "text": "Describe this picture:" 
               },
               { 
                   "type": "image_url",
                   "image_url": {
                       "url": "<image URL>"
                   }
               }
           ] } 
       ],
       max_tokens=2000 
   )
   
   print(response)
   

2. Make the following changes:

   1. Make sure the AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_API_KEY environment variables are set.
   2. Enter the name of your model deployment in the deployment_name variable.
   3. Change the value of the "url" field to the publicly accessible URL of your image.

You can also use a base 64 encoded image data instead of a URL. For more information, see the Vision chats how-to guide.

1. Run the application with the python command:

   python quickstart.py
   

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

• Azure portal
• Azure CLI

Use this article to get started using the OpenAI JavaScript SDK to deploy and use a vision-enabled chat model.

This SDK is provided by OpenAI with Azure specific types provided by Azure.

Reference documentation | Library source code | Package (npm) | Samples

Prerequisites

• An Azure subscription - Create one for free
• LTS versions of Node.js
• Azure CLI used for passwordless authentication in a local development environment, create the necessary context by signing in with the Azure CLI.
• An Azure OpenAI resource created in a supported region (see Region availability). For more information, see Create a resource and deploy a model with Azure OpenAI.

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Microsoft Entra ID prerequisites

For the recommended keyless authentication with Microsoft Entra ID, you need to:

• Install the Azure CLI used for keyless authentication with Microsoft Entra ID.
• Assign the Cognitive Services User role to your user account. You can assign roles in the Azure portal under Access control (IAM) > Add role assignment.

Set up

1. Create a new folder vision-quickstart and go to the quickstart folder with the following command:

   mkdir vision-quickstart && cd vision-quickstart
   

2. Create the package.json with the following command:

   npm init -y
   

3. Install the OpenAI client library for JavaScript with:

   npm install openai
   

4. For the recommended passwordless authentication:

   npm install @azure/identity
   

Retrieve resource information

You need to retrieve the following information to authenticate your application with your Azure OpenAI resource:

Microsoft Entra ID

Variable name	Value
AZURE_OPENAI_ENDPOINT	This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal.
AZURE_OPENAI_DEPLOYMENT_NAME	This value will correspond to the custom name you chose for your deployment when you deployed a model. This value can be found under Resource Management > Model Deployments in the Azure portal.

Learn more about keyless authentication and setting environment variables.

API key

Variable name	Value
AZURE_OPENAI_ENDPOINT	This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal.
AZURE_OPENAI_API_KEY	This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.
AZURE_OPENAI_DEPLOYMENT_NAME	This value will correspond to the custom name you chose for your deployment when you deployed a model. This value can be found under Resource Management > Model Deployments in the Azure portal.

Learn more about finding API keys and setting environment variables.

To use the recommended keyless authentication with the SDK, make sure that the AZURE_OPENAI_API_KEY environment variable isn’t set.

Create a new JavaScript application for image prompts

Select an image from the azure-samples/cognitive-services-sample-data-files. Enter your publicly accessible image URL in the code below or set the IMAGE_URL environment variable to it.

If you use a SAS URL to an image stored in Azure blob storage, you need to enable Managed Identity and assign the Storage Blob Reader role to your Azure OpenAI resource (do this in the Azure portal). This allows the model to access the image in blob storage.

You can also use a base 64 encoded image data instead of a URL. For more information, see the Vision chats how-to guide.

Microsoft Entra ID

1. Create the index.js file with the following code:

   const AzureOpenAI = require('openai').AzureOpenAI;
   const { 
       DefaultAzureCredential, 
       getBearerTokenProvider 
   } = require('@azure/identity');
   
   // You will need to set these environment variables or edit the following values
   const endpoint = process.env.AZURE_OPENAI_ENDPOINT || "Your endpoint";
   const imageUrl = process.env.IMAGE_URL || "<image url>";
   
   // Required Azure OpenAI deployment name and API version
   const apiVersion = process.env.OPENAI_API_VERSION || "2024-07-01-preview";
   const deploymentName = process.env.AZURE_OPENAI_DEPLOYMENT_NAME || "gpt-4-with-turbo";
   
   // keyless authentication    
   const credential = new DefaultAzureCredential();
   const scope = "https://cognitiveservices.azure.com/.default";
   const azureADTokenProvider = getBearerTokenProvider(credential, scope);
   
   function getClient(): AzureOpenAI {
     return new AzureOpenAI({
       endpoint,
       azureADTokenProvider,
       apiVersion,
       deployment: deploymentName,
     });
   }
   function createMessages() {
     return {
       messages: [
         { role: "system", content: "You are a helpful assistant." },
         {
           role: "user",
           content: [
             {
               type: "text",
               text: "Describe this picture:",
             },
             {
               type: "image_url",
               image_url: {
                 url: imageUrl,
               },
             },
           ],
         },
       ],
       model: "",
       max_tokens: 2000,
     };
   }
   async function printChoices(completion) {
     for (const choice of completion.choices) {
       console.log(choice.message);
     }
   }
   export async function main() {
     console.log("== Get Vision chats Sample ==");
   
     const client = getClient();
     const messages = createMessages();
     const completion = await client.chat.completions.create(messages);
     await printChoices(completion);
   }
   
   main().catch((err) => {
     console.error("Error occurred:", err);
   });
   

2. Sign in to Azure with the following command:

   az login
   

3. Run the JavaScript file.

   node index.js
   

API key

1. Create the index.js file with the following code:

   const { AzureOpenAI } = require("openai");
   
   // You will need to set these environment variables or edit the following values
   const endpoint = process.env.AZURE_OPENAI_ENDPOINT || "Your endpoint";
   const apiKey = process.env.AZURE_OPENAI_API_KEY || "Your API key";
   const imageUrl = process.env.IMAGE_URL || "<image url>";
   
   // Required Azure OpenAI deployment name and API version
   const apiVersion = process.env.OPENAI_API_VERSION || "2024-07-01-preview";
   const deploymentName = process.env.AZURE_OPENAI_DEPLOYMENT_NAME || "gpt-4-with-turbo";
   
   function getClient() {
     return new AzureOpenAI({
       endpoint,
       apiKey,
       apiVersion,
       deployment: deploymentName,
     });
   }
   function createMessages() {
     return {
       messages: [
         { role: "system", content: "You are a helpful assistant." },
         {
           role: "user",
           content: [
             {
               type: "text",
               text: "Describe this picture:",
             },
             {
               type: "image_url",
               image_url: {
                 url: imageUrl,
               },
             },
           ],
         },
       ],
       model: "",
       max_tokens: 2000,
     };
   }
   async function printChoices(completion) {
     for (const choice of completion.choices) {
       console.log(choice.message);
     }
   }
   export async function main() {
     console.log("== Get Vision chats Sample ==");
   
     const client = getClient();
     const messages = createMessages();
     const completion = await client.chat.completions.create(messages);
     await printChoices(completion);
   }
   
   main().catch((err) => {
     console.error("Error occurred:", err);
   });
   

2. Run the JavaScript file.

   node index.js
   

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

• Azure portal
• Azure CLI

Use this article to get started using the OpenAI JavaScript SDK to deploy and use a vision-enabled chat model.

This SDK is provided by OpenAI with Azure specific types provided by Azure.

Reference documentation | Library source code | Package (npm) | Samples

Prerequisites

• An Azure subscription - Create one for free
• LTS versions of Node.js
• TypeScript
• Azure CLI used for passwordless authentication in a local development environment, create the necessary context by signing in with the Azure CLI.
• An Azure OpenAI resource created in a supported region (see Region availability). For more information, see Create a resource and deploy a model with Azure OpenAI.

This library is maintained by OpenAI. Refer to the release history to track the latest updates to the library.

Microsoft Entra ID prerequisites

For the recommended keyless authentication with Microsoft Entra ID, you need to:

• Install the Azure CLI used for keyless authentication with Microsoft Entra ID.
• Assign the Cognitive Services User role to your user account. You can assign roles in the Azure portal under Access control (IAM) > Add role assignment.

Set up

1. Create a new folder vision-quickstart and go to the quickstart folder with the following command:

   mkdir vision-quickstart && cd vision-quickstart
   

2. Create the package.json with the following command:

   npm init -y
   

3. Update the package.json to ECMAScript with the following command:

   npm pkg set type=module
   

4. Install the OpenAI client library for JavaScript with:

   npm install openai
   

5. For the recommended passwordless authentication:

   npm install @azure/identity
   

Retrieve resource information

You need to retrieve the following information to authenticate your application with your Azure OpenAI resource:

Microsoft Entra ID

Variable name	Value
AZURE_OPENAI_ENDPOINT	This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal.
AZURE_OPENAI_DEPLOYMENT_NAME	This value will correspond to the custom name you chose for your deployment when you deployed a model. This value can be found under Resource Management > Model Deployments in the Azure portal.

Learn more about keyless authentication and setting environment variables.

API key

Variable name	Value
AZURE_OPENAI_ENDPOINT	This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal.
AZURE_OPENAI_API_KEY	This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.
AZURE_OPENAI_DEPLOYMENT_NAME	This value will correspond to the custom name you chose for your deployment when you deployed a model. This value can be found under Resource Management > Model Deployments in the Azure portal.

Learn more about finding API keys and setting environment variables.

To use the recommended keyless authentication with the SDK, make sure that the AZURE_OPENAI_API_KEY environment variable isn’t set.

Create a new JavaScript application for image prompts

Select an image from the azure-samples/cognitive-services-sample-data-files. Use the image URL in the code below or set the IMAGE_URL environment variable to the image URL.

You can also use a base 64 encoded image data instead of a URL. For more information, see the Vision chats how-to guide.

Microsoft Entra ID

1. Create the index.ts file with the following code:

   import { AzureOpenAI } from "openai";
   import { 
       DefaultAzureCredential, 
       getBearerTokenProvider 
   } from "@azure/identity";
   import type {
     ChatCompletion,
     ChatCompletionCreateParamsNonStreaming,
   } from "openai/resources/index";
   
   // You will need to set these environment variables or edit the following values
   const endpoint = process.env.AZURE_OPENAI_ENDPOINT || "Your endpoint";
   const imageUrl = process.env["IMAGE_URL"] || "<image url>";
   
   // Required Azure OpenAI deployment name and API version
   const apiVersion = process.env.OPENAI_API_VERSION || "2024-07-01-preview";
   const deploymentName = process.env.AZURE_OPENAI_DEPLOYMENT_NAME || "gpt-4-with-turbo";
   
   // keyless authentication    
   const credential = new DefaultAzureCredential();
   const scope = "https://cognitiveservices.azure.com/.default";
   const azureADTokenProvider = getBearerTokenProvider(credential, scope);
   
   function getClient(): AzureOpenAI {
     return new AzureOpenAI({
       endpoint,
       azureADTokenProvider,
       apiVersion,
       deployment: deploymentName,
     });
   }
   function createMessages(): ChatCompletionCreateParamsNonStreaming {
     return {
       messages: [
         { role: "system", content: "You are a helpful assistant." },
         {
           role: "user",
           content: [
             {
               type: "text",
               text: "Describe this picture:",
             },
             {
               type: "image_url",
               image_url: {
                 url: imageUrl,
               },
             },
           ],
         },
       ],
       model: "",
       max_tokens: 2000,
     };
   }
   async function printChoices(completion: ChatCompletion): Promise<void> {
     for (const choice of completion.choices) {
       console.log(choice.message);
     }
   }
   export async function main() {
     console.log("== Get Vision chat Sample ==");
   
     const client = getClient();
     const messages = createMessages();
     const completion = await client.chat.completions.create(messages);
     await printChoices(completion);
   }
   
   main().catch((err) => {
     console.error("Error occurred:", err);
   });
   

2. Create the tsconfig.json file to transpile the TypeScript code and copy the following code for ECMAScript.

   {
       "compilerOptions": {
         "module": "NodeNext",
         "target": "ES2022", // Supports top-level await
         "moduleResolution": "NodeNext",
         "skipLibCheck": true, // Avoid type errors from node_modules
         "strict": true // Enable strict type-checking options
       },
       "include": ["*.ts"]
   }
   

3. Transpile from TypeScript to JavaScript.

   tsc
   

4. Sign in to Azure with the following command:

   az login
   

5. Run the code with the following command:

   node index.js
   

API key

1. Create the index.ts file with the following code:

   import { AzureOpenAI } from "openai";
   import type {
     ChatCompletion,
     ChatCompletionCreateParamsNonStreaming,
   } from "openai/resources/index";
   
   // You will need to set these environment variables or edit the following values
   const endpoint = process.env.AZURE_OPENAI_ENDPOINT || "Your endpoint";
   const apiKey = process.env.AZURE_OPENAI_API_KEY || "Your API key";
   const imageUrl = process.env["IMAGE_URL"] || "<image url>";
   
   // Required Azure OpenAI deployment name and API version
   const apiVersion = process.env.OPENAI_API_VERSION || "2024-07-01-preview";
   const deploymentName = process.env.AZURE_OPENAI_DEPLOYMENT_NAME || "gpt-4-with-turbo";
   
   function getClient(): AzureOpenAI {
     return new AzureOpenAI({
       endpoint,
       apiKey,
       apiVersion,
       deployment: deploymentName,
     });
   }
   function createMessages(): ChatCompletionCreateParamsNonStreaming {
     return {
       messages: [
         { role: "system", content: "You are a helpful assistant." },
         {
           role: "user",
           content: [
             {
               type: "text",
               text: "Describe this picture:",
             },
             {
               type: "image_url",
               image_url: {
                 url: imageUrl,
               },
             },
           ],
         },
       ],
       model: "",
       max_tokens: 2000,
     };
   }
   async function printChoices(completion: ChatCompletion): Promise<void> {
     for (const choice of completion.choices) {
       console.log(choice.message);
     }
   }
   export async function main() {
     console.log("== Get Vision chat Sample ==");
   
     const client = getClient();
     const messages = createMessages();
     const completion = await client.chat.completions.create(messages);
     await printChoices(completion);
   }
   
   main().catch((err) => {
     console.error("Error occurred:", err);
   });
   

2. Create the tsconfig.json file to transpile the TypeScript code and copy the following code for ECMAScript.

   {
       "compilerOptions": {
         "module": "NodeNext",
         "target": "ES2022", // Supports top-level await
         "moduleResolution": "NodeNext",
         "skipLibCheck": true, // Avoid type errors from node_modules
         "strict": true // Enable strict type-checking options
       },
       "include": ["*.ts"]
   }
   

3. Transpile from TypeScript to JavaScript.

   tsc
   

4. Run the code with the following command:

   node index.js
   

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

• Azure portal
• Azure CLI

Use this article to get started using the Azure OpenAI .NET SDK to deploy and use a vision-enabled chat model.

Prerequisites

• An Azure subscription. You can create one for free.
• The .NET 8.0 SDK
• An Azure OpenAI in Microsoft Foundry Models resource with a vision-enabled chat model deployed. See Model availability for available regions. For more information about resource creation, see the resource deployment guide.

Microsoft Entra ID prerequisites

For the recommended keyless authentication with Microsoft Entra ID, you need to:

• Install the Azure CLI used for keyless authentication with Microsoft Entra ID.
• Assign the Cognitive Services User role to your user account. You can assign roles in the Azure portal under Access control (IAM) > Add role assignment.

Set up

1. Create a new folder vision-quickstart and go to the quickstart folder with the following command:

   mkdir vision-quickstart && cd vision-quickstart
   

2. Create a new console application with the following command:

   dotnet new console
   

3. Install the OpenAI .NET client library with the dotnet add package command:

   dotnet add package Azure.AI.OpenAI
   

4. For the recommended keyless authentication with Microsoft Entra ID, install the Azure.Identity package with:

   dotnet add package Azure.Identity
   

5. For the recommended keyless authentication with Microsoft Entra ID, sign in to Azure with the following command:

   az login
   

Retrieve resource information

You need to retrieve the following information to authenticate your application with your Azure OpenAI resource:

Microsoft Entra ID

Variable name	Value
AZURE_OPENAI_ENDPOINT	This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal.
AZURE_OPENAI_DEPLOYMENT_NAME	This value will correspond to the custom name you chose for your deployment when you deployed a model. This value can be found under Resource Management > Model Deployments in the Azure portal.

Learn more about keyless authentication and setting environment variables.

API key

Variable name	Value
AZURE_OPENAI_ENDPOINT	This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal.
AZURE_OPENAI_API_KEY	This value can be found in the Keys and Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.
AZURE_OPENAI_DEPLOYMENT_NAME	This value will correspond to the custom name you chose for your deployment when you deployed a model. This value can be found under Resource Management > Model Deployments in the Azure portal.

Learn more about finding API keys and setting environment variables.

Run the quickstart

The sample code in this quickstart uses Microsoft Entra ID for the recommended keyless authentication. If you prefer to use an API key, you can replace the DefaultAzureCredential object with an AzureKeyCredential object.

Microsoft Entra ID

    AzureOpenAIClient openAIClient = new AzureOpenAIClient(new Uri(endpoint), new DefaultAzureCredential()); 

To run the quickstart, follow these steps:

1. Replace the contents of Program.cs with the following code and update the placeholder values with your own.

   using Azure;
   using Azure.AI.OpenAI;
   using Azure.Identity;
   using OpenAI.Chat; // Required for Passwordless auth
   
   string deploymentName = "gpt-4";
   
   string endpoint = Environment.GetEnvironmentVariable("AZURE_OPENAI_ENDPOINT") ?? "https://<your-resource-name>.openai.azure.com/";
   string key = Environment.GetEnvironmentVariable("AZURE_OPENAI_API_KEY") ?? "<your-key>";
   
   // Use the recommended keyless credential instead of the AzureKeyCredential credential.
   AzureOpenAIClient openAIClient = new AzureOpenAIClient(new Uri(endpoint), new DefaultAzureCredential()); 
   //AzureOpenAIClient openAIClient = new AzureOpenAIClient(new Uri(endpoint), new AzureKeyCredential(key));
   
   var chatClient = openAIClient.GetChatClient(deploymentName);
   
   var imageUrl = "YOUR_IMAGE_URL";
   
   var textPart = ChatMessageContentPart.CreateTextPart("Describe this picture:");
   var imgPart = ChatMessageContentPart.CreateImagePart(imageUrl); 
   
   var chatMessages = new List<ChatMessage>
   {
       new SystemChatMessage("You are a helpful assistant."),
       new UserChatMessage(textPart, imgPart)
   
   };
       
   ChatCompletion chatCompletion = await chatClient.CompleteChatAsync(chatMessages);
   
   Console.WriteLine($"[ASSISTANT]:");
   Console.WriteLine($"{chatCompletion.Content[0].Text}");
   

2. Replace YOUR_IMAGE_URL with the publicly accessible of the image you want to upload.

3. Run the application using the dotnet run command or the run button at the top of Visual Studio:

   dotnet run
   

Output

The output of the application will be a description of the image you provided in the imageUri variable. The assistant will analyze the image and provide a detailed description based on its content.

Clean up resources

If you want to clean up and remove an Azure OpenAI resource, you can delete the resource or resource group. Deleting the resource group also deletes any other resources associated with it.

• Azure portal
• Azure CLI

Related content

• Get started with multimodal vision chat apps using Azure OpenAI AI App template
• Learn more about these APIs in the Vision-enabled models how-to guide
• GPT-4 Turbo with Vision frequently asked questions
• GPT-4 Turbo with Vision API reference