Introduction to cloud evaluation with Microsoft Foundry SDK

Set up the Microsoft Foundry SDK and choose a cloud evaluation workflow for datasets, targets, interactions, conversations, or synthetic data.

Use cloud evaluations to test generative AI applications at scale without managing local compute. This article sets up the shared SDK client and helps you choose a workflow for predeployment or production evaluation.

Prerequisites

• A Foundry project.
• An Azure OpenAI deployment with a GPT model that supports chat completion, such as gpt-5-mini.
• The Foundry User role on the Foundry project.

The Foundry RBAC roles were recently renamed. Foundry User, Foundry Owner, Foundry Account Owner, and Foundry Project Manager were previously named Azure AI User, Azure AI Owner, Azure AI Account Owner, and Azure AI Project Manager. You might still see the previous names in some places while the rename rolls out. The role IDs and core permissions are unchanged by the rename.

• For all evaluation role requirements, see Set up permissions for evaluation workflows.
• Optionally, your own storage account for evaluation data.

Some evaluation features have regional restrictions. Review the supported regions before you begin.

Set up the SDK client

Install the SDK for your language. Set these shared environment variables:

• AZURE_AI_PROJECT_ENDPOINT: Your Foundry project endpoint, for example, https://<account_name>.services.ai.azure.com/api/projects/<project_name>.
• AZURE_AI_MODEL_DEPLOYMENT_NAME: The model deployment used by AI-assisted evaluators.
• DATASET_NAME and DATASET_VERSION: Optional values for reusable datasets.

Authenticate with DefaultAzureCredential, create the project client, and get the OpenAI client used by the evaluation API.

Python

pip install "azure-ai-projects>=2.2.0"

import os
from azure.identity import DefaultAzureCredential
from azure.ai.projects import AIProjectClient
from azure.ai.projects.models import TestingCriterionAzureAIEvaluator
from openai.types.eval_create_params import DataSourceConfigCustom
from openai.types.evals.create_eval_jsonl_run_data_source_param import (
    CreateEvalJSONLRunDataSourceParam,
    SourceFileContent,
    SourceFileContentContent,
    SourceFileID,
)

endpoint = os.environ["AZURE_AI_PROJECT_ENDPOINT"]
model_deployment_name = os.environ.get(
    "AZURE_AI_MODEL_DEPLOYMENT_NAME", ""
)
dataset_name = os.environ.get("DATASET_NAME", "")
dataset_version = os.environ.get("DATASET_VERSION", "1")

project_client = AIProjectClient(
    endpoint=endpoint,
    credential=DefaultAzureCredential(),
)
openai_client = project_client.get_openai_client()

Reference: AIProjectClient, DefaultAzureCredential

C#

dotnet add package Azure.AI.Projects --prerelease
dotnet add package Azure.Identity

using System.ClientModel;
using System.Collections.Generic;
using System.Text.Json;
using Azure.AI.Extensions.OpenAI;
using Azure.AI.Projects;
using Azure.Identity;
using OpenAI.Evals;
using OpenAI.Responses;

#pragma warning disable OPENAI001

static string GetString(ClientResult result, string propertyName)
{
  using JsonDocument document = JsonDocument.Parse(
    result.GetRawResponse().Content.ToMemory());
  return document.RootElement.GetProperty(propertyName).GetString()
    ?? throw new InvalidOperationException(
      $"The response doesn't contain {propertyName}.");
}

var projectEndpoint = Environment.GetEnvironmentVariable(
  "AZURE_AI_PROJECT_ENDPOINT")
  ?? throw new InvalidOperationException(
    "AZURE_AI_PROJECT_ENDPOINT isn't set.");
var modelDeploymentName = Environment.GetEnvironmentVariable(
  "AZURE_AI_MODEL_DEPLOYMENT_NAME")
  ?? throw new InvalidOperationException(
    "AZURE_AI_MODEL_DEPLOYMENT_NAME isn't set.");
var datasetName = Environment.GetEnvironmentVariable("DATASET_NAME")
  ?? "evaluation-data";
var datasetVersion = Environment.GetEnvironmentVariable("DATASET_VERSION")
  ?? "1";

AIProjectClient projectClient = new(
  endpoint: new Uri(projectEndpoint),
  tokenProvider: new DefaultAzureCredential());
EvaluationClient evaluationClient = projectClient.ProjectOpenAIClient
  .GetEvaluationClient();

Reference: AIProjectClient, DefaultAzureCredential, and EvaluationClient

JavaScript/TypeScript

npm install @azure/ai-projects @azure/identity dotenv

import { AIProjectClient } from "@azure/ai-projects";
import { DefaultAzureCredential } from "@azure/identity";
import "dotenv/config";

const projectEndpoint = process.env["AZURE_AI_PROJECT_ENDPOINT"] || "";
const modelDeploymentName =
  process.env["AZURE_AI_MODEL_DEPLOYMENT_NAME"] || "";
const datasetName = process.env["DATASET_NAME"] || "";
const datasetVersion = process.env["DATASET_VERSION"] || "1";

const projectClient = new AIProjectClient(
  projectEndpoint,
  new DefaultAzureCredential(),
);
const openaiClient = projectClient.getOpenAIClient();

Reference: AIProjectClient class

To use a model connected through admin connections as a target, judge model, or conversation simulator, see Use admin-connected models in cloud evaluations.

Understand the evaluation workflow

A cloud evaluation has three steps:

1. Define the data shape and the evaluators that score it.
2. Create the evaluation with the evaluation client.
3. Start a run, poll until it completes, and retrieve the scored results.

Cloud evaluation results are stored in your Foundry project. You can retrieve them through the SDK, review them in the portal, or route them to Application Insights when it’s connected.

Choose evaluators

Evaluators bind to fields in your data through column mappings. Dataset workflows expose item fields, while target-generated workflows also expose the model or agent output through the sample schema.

Review the built-in evaluators and custom evaluators before you configure testing criteria.

Choose your starting point

Evaluation unit	Starting point	Workflow
Individual turn	You have JSONL or CSV test data with a query and response.	Evaluate query-response datasets
Individual turn	You have queries and want a model or agent to generate responses.	Evaluate model and agent responses
Individual turn	You have stored responses or traces from individual interactions with a deployed model or agent.	Evaluate deployed model and agent interactions
Individual turn	You need to generate synthetic queries.	Generate synthetic queries
Complete conversation	You have a dataset of complete conversations.	Evaluate conversation datasets
Complete conversation	You have traces that capture complete conversations with a deployed model or agent.	Evaluate deployed model and agent conversations
Complete conversation	You need to generate and evaluate simulated agent conversations.	Simulate agent conversations
Any evaluation unit	You need to poll, interpret, cancel, or troubleshoot a run.	Get evaluation results

For adversarial safety testing, use AI red teaming. To create a standalone dataset, see Generate a synthetic evaluation dataset.

Related content

• Get cloud evaluation results
• View evaluation results in the Foundry portal
• Set up continuous evaluation
• Complete Python SDK evaluation samples
• .NET evaluation samples
• REST API reference