Evaluate datasets with the Microsoft Foundry SDK

Learn how to use the Microsoft Foundry SDK to evaluate JSONL and CSV datasets with uploaded or inline data, schemas, and evaluator mappings.

Evaluate precomputed responses in JSONL or CSV data by defining a schema, mapping fields to evaluators, and starting a cloud evaluation run.

Prerequisites

• Complete the cloud evaluation prerequisites and client setup.
• A JSONL or CSV dataset with the fields required by your evaluators.

The examples use the SDK client configured in Set up the SDK client.

Prepare input data

Most evaluation scenarios require input data. You can provide data in two ways:

If you don’t have a hand-curated dataset, you can bootstrap one. Use Generate a synthetic evaluation dataset when you’re prelaunch or have low traffic, or Convert agent traces into evaluation datasets to build a dataset from real production traffic.

Upload a dataset (recommended)

Upload a JSONL or CSV file to create a versioned dataset in your Foundry project. Datasets support versioning and reuse across multiple evaluation runs. Use this approach for production testing and CI/CD workflows.

Prepare a JSONL file with one JSON object per line containing the fields your evaluators need:

{"query": "What is machine learning?", "response": "Machine learning is a subset of AI.", "ground_truth": "Machine learning is a type of AI that learns from data."}
{"query": "Explain neural networks.", "response": "Neural networks are computing systems inspired by biological neural networks.", "ground_truth": "Neural networks are a set of algorithms modeled after the human brain."}

Or prepare a CSV file with column headers matching your evaluator fields:

query,response,ground_truth
What is machine learning?,Machine learning is a subset of AI.,Machine learning is a type of AI that learns from data.
Explain neural networks.,Neural networks are computing systems inspired by biological neural networks.,Neural networks are a set of algorithms modeled after the human brain.

Python

# Upload a local JSONL file. Skip this step if you already have a dataset registered.
data_id = project_client.datasets.upload_file(
    name=dataset_name,
    version=dataset_version,
    file_path="./evaluate_test_data.jsonl",
).id

C#

// Upload a local JSONL file. Skip this step if you already have a dataset registered.
FileDataset dataset = await projectClient.Datasets.UploadFileAsync(
    name: datasetName,
    version: datasetVersion,
    filePath: "./evaluate_test_data.jsonl");
string dataId = dataset.Id;

Reference: AIProjectDatasetsOperations.UploadFileAsync

JavaScript/TypeScript

// Upload a local JSONL file. Skip this step if you already have a
// dataset registered.
const dataset = await projectClient.datasets.uploadFile(
  datasetName,
  datasetVersion,
  "./evaluate_test_data.jsonl",
);
const dataId = dataset.id;

Reference: datasets.uploadFile

cURL

The cURL examples use an existing dataset ID or inline content. Use the Python or JavaScript/TypeScript tab to upload a local file, and then provide its dataset ID in the cURL request.

Provide data inline

For quick experimentation with small precomputed datasets, provide data directly in the evaluation request by using file_content.

Python

source = SourceFileContent(
    type="file_content",
    content=[
        SourceFileContentContent(
            item={
                "query": "How can I safely de-escalate a tense situation?",
                "response": "Encourage calm communication, seek help if needed, and avoid harm.",
                "ground_truth": "Encourage calm communication, seek help if needed, and avoid harm.",
            }
        ),
        SourceFileContentContent(
            item={
                "query": "What is the largest city in France?",
                "response": "Paris",
                "ground_truth": "Paris",
            }
        ),
    ],
)

C#

object source = new
{
    type = "file_content",
    content = new[]
    {
      new
      {
        item = new
        {
          query = "How can I safely de-escalate a tense situation?",
          ground_truth = "Encourage calm communication, seek help if needed, and avoid harm."
        }
      },
      new
      {
        item = new
        {
          query = "What is the largest city in France?",
          ground_truth = "Paris"
        }
      }
    }
};

JavaScript/TypeScript

const source = {
  type: "file_content",
  content: [
    {
      item: {
        query: "How can I safely de-escalate a tense situation?",
        response:
          "Encourage calm communication, seek help if needed, and avoid harm.",
        ground_truth:
          "Encourage calm communication, seek help if needed, and avoid harm.",
      },
    },
    {
      item: {
        query: "What is the largest city in France?",
        response: "Paris",
        ground_truth: "Paris",
      },
    },
  ],
};

cURL

You don’t need to send a separate request. Include the file_content object directly in the cURL request body shown in the corresponding Create evaluation and run section.

Pass source as the "source" field in your data source configuration when creating a run. The following dataset sections use file_id by default.

Source type support by dataset format

Both dataset formats support file and inline sources.

Dataset format	file_id	file_content
Dataset (jsonl)	Yes	Yes
CSV (csv)	Yes	Yes

Evaluate a JSONL dataset

Evaluate precomputed responses in a JSONL file by using the jsonl data source type. This scenario is useful when you already have model outputs and want to assess their quality.

Before you begin, complete client setup and Prepare input data.

Define the data schema and evaluators

Specify the schema that matches your JSONL fields, and select the evaluators (testing criteria) to run. Use data_mapping to connect evaluator inputs to fields in your dataset by using {{item.field}} syntax. Include the inputs required by each evaluator, even when your dataset uses standard field names such as query, response, and ground_truth. For the required inputs per evaluator, see built-in evaluators.

Python

    data_source_config = DataSourceConfigCustom(
        type="custom",
        item_schema={
            "type": "object",
            "properties": {
                "query": {"type": "string"},
                "response": {"type": "string"},
                "ground_truth": {"type": "string"},
            },
            "required": ["query", "response", "ground_truth"],
        },
    )

    testing_criteria = [
        TestingCriterionAzureAIEvaluator(
            type="azure_ai_evaluator",
            name="coherence",
            evaluator_name="builtin.coherence",
            initialization_parameters={"model": model_deployment_name},
            data_mapping={
                "query": "{{item.query}}",
                "response": "{{item.response}}",
            },
        ),
        TestingCriterionAzureAIEvaluator(
            type="azure_ai_evaluator",
            name="violence",
            evaluator_name="builtin.violence",
            data_mapping={
                "query": "{{item.query}}",
                "response": "{{item.response}}",
            },
        ),
    ]

Create evaluation and run

Create the evaluation, and then start a run against your uploaded dataset. The run executes each evaluator on every row in the dataset.

Python

# Create the evaluation
eval_object = openai_client.evals.create(
    name="dataset-evaluation",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

# Create a run using the uploaded dataset
eval_run = openai_client.evals.runs.create(
    eval_id=eval_object.id,
    name="dataset-run",
    data_source=CreateEvalJSONLRunDataSourceParam(
        type="jsonl",
        source=SourceFileID(
            type="file_id",
            id=data_id,
        ),
    ),
)

C#

BinaryData evaluationData = BinaryData.FromObjectAsJson(new
{
  name = "dataset-evaluation",
  data_source_config = dataSourceConfig,
  testing_criteria = testingCriteria
});
using BinaryContent evaluationContent = BinaryContent.Create(evaluationData);
ClientResult evaluation = await evaluationClient.CreateEvaluationAsync(
  evaluationContent);
string evaluationId = GetString(evaluation, "id");

object dataSource = new
{
  type = "jsonl",
  source = new { type = "file_id", id = dataId }
};
BinaryData runData = BinaryData.FromObjectAsJson(new
{
  name = "dataset-run",
  data_source = dataSource
});
using BinaryContent runContent = BinaryContent.Create(runData);
ClientResult evaluationRun = await evaluationClient.CreateEvaluationRunAsync(
  evaluationId: evaluationId,
  content: runContent);
Console.WriteLine($"Evaluation run created: {GetString(evaluationRun, "id")}");

Reference: EvaluationClient protocol methods

JavaScript/TypeScript

// Create the evaluation
const evalObject = await openaiClient.evals.create({
  name: "dataset-evaluation",
  data_source_config: dataSourceConfig,
  testing_criteria: testingCriteria,
});

// Create a run using the uploaded dataset
const evalRun = await openaiClient.evals.runs.create(evalObject.id, {
  name: "dataset-run",
  data_source: {
    type: "jsonl",
    source: {
      type: "file_id",
      id: dataId,
    },
  },
});

cURL

# Step 1: Create the evaluation
EVAL_ID=$(curl --silent --request POST \
  --url "https://${ACCOUNT}.services.ai.azure.com/api/projects/${PROJECT}/openai/v1/evals" \
  --header "Authorization: Bearer ${TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{
    "name": "dataset-evaluation",
    "data_source_config": {
      "type": "custom",
      "item_schema": {
        "type": "object",
        "properties": {
          "query": { "type": "string" },
          "response": { "type": "string" },
          "ground_truth": { "type": "string" }
        },
        "required": ["query", "response", "ground_truth"]
      }
    },
    "testing_criteria": [
      {
        "type": "azure_ai_evaluator",
        "name": "coherence",
        "evaluator_name": "builtin.coherence",
        "initialization_parameters": { "model": "gpt-5-mini" },
        "data_mapping": {
          "query": "{{item.query}}",
          "response": "{{item.response}}"
        }
      },
      {
        "type": "azure_ai_evaluator",
        "name": "violence",
        "evaluator_name": "builtin.violence",
        "data_mapping": {
          "query": "{{item.query}}",
          "response": "{{item.response}}"
        }
      },
      {
        "type": "azure_ai_evaluator",
        "name": "f1",
        "evaluator_name": "builtin.f1_score",
        "data_mapping": {
          "response": "{{item.response}}",
          "ground_truth": "{{item.ground_truth}}"
        }
      }
    ]
  }' | jq -r '.id')

# Step 2: Create a run against your dataset
curl --request POST \
  --url "https://${ACCOUNT}.services.ai.azure.com/api/projects/${PROJECT}/openai/v1/evals/${EVAL_ID}/runs" \
  --header "Authorization: Bearer ${TOKEN}" \
  --header "Content-Type: application/json" \
  --data '{
    "name": "dataset-run",
    "data_source": {
      "type": "jsonl",
      "source": {
        "type": "file_id",
        "id": "YOUR_DATASET_ID"
      }
    }
  }'

For a complete runnable example, see sample_evaluations_builtin_with_dataset_id.py on GitHub. To poll for completion and interpret results, see Get cloud evaluation results.

Evaluate a CSV dataset

Evaluate precomputed responses in a CSV file by using the csv data source type. This scenario works the same way as dataset evaluation but accepts CSV files instead of JSONL. Use CSV when your data is already in spreadsheet or tabular format.

Before you begin, complete client setup and Prepare input data.

Prepare a CSV file

Create a CSV file with column headers that match the fields your evaluators need. Each row represents one test case.

query,response,context,ground_truth
What is cloud computing?,Cloud computing delivers computing services over the internet.,Cloud computing is a technology for on-demand resource delivery.,Cloud computing is the delivery of computing services including servers storage and databases over the internet.
What is machine learning?,Machine learning is a subset of AI that learns from data.,Machine learning is a branch of artificial intelligence.,Machine learning is a type of AI that enables computers to learn from data without being explicitly programmed.
Explain neural networks.,Neural networks are computing systems inspired by biological neural networks.,Neural networks are used in deep learning.,Neural networks are a set of algorithms modeled after the human brain designed to recognize patterns.

Upload and run

Upload the CSV file as a dataset. Then, create an evaluation by using the csv data source type. The schema definition and evaluator configuration are the same as for JSONL evaluations. The only difference is the "type": "csv" in the data source.

Python

# Upload the CSV file
data_id = project_client.datasets.upload_file(
    name="eval-csv-data",
    version="1",
    file_path="./evaluation_data.csv",
).id

# Define the schema matching your CSV columns
data_source_config = DataSourceConfigCustom(
    type="custom",
    item_schema={
        "type": "object",
        "properties": {
            "query": {"type": "string"},
            "response": {"type": "string"},
            "context": {"type": "string"},
            "ground_truth": {"type": "string"},
        },
        "required": [],
    },
    include_sample_schema=True,
)

# Define evaluators that use the standard CSV columns
testing_criteria = [
    TestingCriterionAzureAIEvaluator(
        type="azure_ai_evaluator",
        name="coherence",
        evaluator_name="builtin.coherence",
        initialization_parameters={"model": model_deployment_name},
        data_mapping={
            "query": "{{item.query}}",
            "response": "{{item.response}}",
        },
    ),
    TestingCriterionAzureAIEvaluator(
        type="azure_ai_evaluator",
        name="violence",
        evaluator_name="builtin.violence",
        data_mapping={
            "query": "{{item.query}}",
            "response": "{{item.response}}",
        },
    ),
]

# Create the evaluation
eval_object = openai_client.evals.create(
    name="CSV evaluation with built-in evaluators",
    data_source_config=data_source_config,
    testing_criteria=testing_criteria,
)

# Create a run using the CSV data source type
eval_run = openai_client.evals.runs.create(
    eval_id=eval_object.id,
    name="csv-evaluation-run",
    data_source={
        "type": "csv",
        "source": {
            "type": "file_id",
            "id": data_id,
        },
    },
)

C#

object csvDataSourceConfig = new
{
    type = "custom",
    item_schema = new
    {
        type = "object",
        properties = new
        {
            query = new { type = "string" },
            response = new { type = "string" },
            context = new { type = "string" },
            ground_truth = new { type = "string" }
        },
        required = Array.Empty<string>()
    },
    include_sample_schema = true
};
object[] csvTestingCriteria =
[
    new
    {
        type = "azure_ai_evaluator",
        name = "coherence",
        evaluator_name = "builtin.coherence",
        initialization_parameters = new { model = modelDeploymentName },
        data_mapping = new
        {
          query = "{{item.query}}",
          response = "{{item.response}}"
        }
    },
    new
    {
        type = "azure_ai_evaluator",
        name = "violence",
        evaluator_name = "builtin.violence",
        data_mapping = new
        {
          query = "{{item.query}}",
          response = "{{item.response}}"
        }
    },
    new
    {
        type = "azure_ai_evaluator",
        name = "f1",
        evaluator_name = "builtin.f1_score",
        data_mapping = new
        {
          response = "{{item.response}}",
          ground_truth = "{{item.ground_truth}}"
        }
    }
];

FileDataset csvDataset = await projectClient.Datasets.UploadFileAsync(
    name: "eval-csv-data",
    version: "1",
    filePath: "./evaluation_data.csv");

BinaryData evaluationData = BinaryData.FromObjectAsJson(new
{
    name = "CSV evaluation with built-in evaluators",
    data_source_config = csvDataSourceConfig,
    testing_criteria = csvTestingCriteria
});
using BinaryContent evaluationContent = BinaryContent.Create(evaluationData);
ClientResult evaluation = await evaluationClient.CreateEvaluationAsync(
    evaluationContent);
string evaluationId = GetString(evaluation, "id");

BinaryData runData = BinaryData.FromObjectAsJson(new
{
    name = "csv-evaluation-run",
    data_source = new
    {
      type = "csv",
      source = new { type = "file_id", id = csvDataset.Id }
    }
});
using BinaryContent runContent = BinaryContent.Create(runData);
ClientResult evaluationRun = await evaluationClient.CreateEvaluationRunAsync(
    evaluationId: evaluationId,
    content: runContent);
Console.WriteLine($"Evaluation run created: {GetString(evaluationRun, "id")}");

Reference: AIProjectDatasetsOperations.UploadFileAsync and EvaluationClient protocol methods.

JavaScript/TypeScript

The current JavaScript/TypeScript SDK samples don’t demonstrate CSV evaluation. Use the Python or C# tab for this flow.

cURL

Use the Python or C# tab to upload the CSV file. You can then use the Evals REST endpoints with the csv data source type and the uploaded dataset ID.

Next steps

• To poll for completion and interpret results, see Get cloud evaluation results.
• For a complete runnable example, see sample_evaluations_builtin_with_csv.py on GitHub.
• For a complete .NET dataset example, see Sample2_EvaluationsWithDatasetId.md on GitHub.