Troubleshoot evaluation and observability issues - Microsoft Foundry

Learn how to troubleshoot common issues with evaluations and observability in Microsoft Foundry, including storage account access, RBAC, and network configuration.

This article provides information to help you solve common issues you might encounter when you use evaluation and observability features in Microsoft Foundry. Some issues relate to storage account configuration, role-based access control (RBAC), or network settings for the Foundry project. Other issues happen while an evaluation runs, such as authentication failures, model capacity or quota limits, data format problems, or missing scores.

Storage account not linked to the Foundry project

Evaluation features require a storage account linked to your Foundry project through a connection. If the storage account isn’t connected, evaluations fail because the service can’t read or write evaluation data.

Symptoms:

• Evaluations fail with errors related to storage access or missing storage configuration.
• The evaluation service can’t upload evaluation results or download datasets.

Connect a storage account to the Foundry project

Connect your storage account to the Foundry project by creating an Azure Blob Storage connection. For step-by-step instructions, see Add a new connection to your project.

You can authenticate the connection by using either an account key or Microsoft Entra ID (recommended). If you use Entra ID, see Missing RBAC role assignment for Entra ID authentication to configure the required permissions.

For more details on bringing your own storage for evaluations, see Rate limits, region support, and enterprise features for evaluation.

Missing RBAC role assignment for Entra ID authentication

If you connect your storage account by using Microsoft Entra ID authentication, the Foundry project’s managed identity must have the Storage Blob Data Contributor role on the storage account. Without this role, the service can’t read or write blob data and evaluations fail.

Symptoms:

• Evaluations fail with 403 Forbidden or AuthorizationPermissionMismatch errors.
• You see errors indicating insufficient permissions to access the storage account.
• Storage operations time out or are denied.

Verify the managed identity role assignment

Use the following Azure CLI commands to check whether the correct RBAC role is assigned to the Foundry project’s managed identity on the storage account.

First, retrieve the managed identity principal ID for your Foundry project:

az resource show \
  --resource-group <your-resource-group> \
  --name <your-foundry-account-name> \
  --resource-type "Microsoft.CognitiveServices/accounts" \
  --query "identity.principalId" \
  --output tsv

Then, list the role assignments on the storage account and filter for the managed identity:

az role assignment list \
  --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account-name>" \
  --assignee <principal-id> \
  --output table

Verify that the output includes a role assignment with RoleDefinitionName set to Storage Blob Data Contributor (or Storage Blob Data Owner).

Assign the Storage Blob Data Contributor role

If the role assignment is missing, assign the Storage Blob Data Contributor role to the Foundry project’s managed identity:

az role assignment create \
  --assignee <principal-id> \
  --role "Storage Blob Data Contributor" \
  --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account-name>"

Role assignments can take up to 10 minutes to propagate. Wait a few minutes after assigning the role before retrying the evaluation.

Storage account network access restrictions

When you use Microsoft Entra ID authentication, the storage account must have public network access enabled. If network access is restricted, the Foundry evaluation service might not be able to reach the storage account.

Symptoms:

• Evaluations fail with network-related errors or timeouts.
• You see 403 Forbidden errors even though RBAC roles are correctly assigned.
• Connections to the storage account are refused.

Verify the storage account network configuration

Use the following Azure CLI command to check the network access settings of your storage account:

az storage account show \
  --resource-group <resource-group> \
  --name <storage-account-name> \
  --query "{publicNetworkAccess: publicNetworkAccess, defaultAction: networkRuleSet.defaultAction, virtualNetworkRules: networkRuleSet.virtualNetworkRules, ipRules: networkRuleSet.ipRules}" \
  --output json

Check the output for the following values:

Property	Expected value	Description
publicNetworkAccess	Enabled	Public network access must be enabled.
defaultAction	Allow	The default network rule should allow access.

If publicNetworkAccess is set to Disabled or defaultAction is set to Deny, the evaluation service can’t reach the storage account.

For virtual network based (network-isolated) agent setups where resources are expected to operate with public network access disabled and rely on private endpoints connectivity virtual network instead, see Set up private networking.

Enable public network access

Enable public network access on the storage account:

az storage account update \
  --resource-group <resource-group> \
  --name <storage-account-name> \
  --public-network-access Enabled

If you need to keep the firewall enabled but allow access, set the default action to Allow:

az storage account update \
  --resource-group <resource-group> \
  --name <storage-account-name> \
  --default-action Allow

Enabling public network access or setting the default action to Allow makes the storage account accessible from all networks. Evaluate this change against your organization’s security requirements.

Troubleshooting checklist

Use this checklist to quickly verify your evaluation setup:

1. Storage connection exists: Confirm that an Azure Blob Storage connection is configured in your Foundry project. Navigate to Build > Tools in the Foundry portal to check.

2. Authentication type: Identify whether the connection uses an account key or Microsoft Entra ID. If Entra ID, complete the remaining checks.

3. RBAC role assigned: Verify that the Foundry project’s managed identity has the Storage Blob Data Contributor role on the storage account.

   az role assignment list \
     --scope "/subscriptions/<subscription-id>/resourceGroups/<resource-group>/providers/Microsoft.Storage/storageAccounts/<storage-account-name>" \
     --assignee <principal-id> \
     --query "[].{Role:roleDefinitionName, Principal:principalId}" \
     --output table
   

4. Network access: Verify that the storage account has public network access enabled.

   az storage account show \
     --resource-group <resource-group> \
     --name <storage-account-name> \
     --query "publicNetworkAccess" \
     --output tsv
   

5. Propagation delay: If you recently made RBAC or network changes, wait at least 10 minutes before retrying.

Evaluation run is slow, stuck, or fails with capacity or quota errors

An evaluation run might stay in the Running or pending state for a long time, run slowly, or fail with quota errors. This condition usually happens when the judge model deployment doesn’t have enough capacity, so the service throttles or retries requests.

Symptoms:

• The run stays in the Running or pending state much longer than expected.
• The run fails with a 429 Too Many Requests error.
• You see errors that mention quota or rate limits.

Resolution:

• Confirm the judge model deployment has enough quota. The judge model used for AI-assisted evaluators counts against your Azure OpenAI quota.
• Increase the tokens-per-minute (TPM) quota for the model deployment in the Azure portal, then run the evaluation again.
• Reduce the size of your dataset, or split it into smaller batches. For simulations, decrease the maximum turns per conversation.
• Use a smaller or lower-cost judge model deployment for faster, cheaper runs.
• For a stuck SDK run, cancel it with client.evals.runs.cancel(run_id, eval_id=eval_id), increase capacity, then resubmit.
• For a 429 error, check the retry-after header for the recommended wait time, and use exponential backoff when you retry.

Authentication or authorization errors (401 or 403)

If an evaluation fails with a 401 Unauthorized or 403 Forbidden error that isn’t related to storage, the cause is usually project authentication or a missing role assignment.

If the 403 error mentions blob or storage access, see Missing RBAC role assignment for Entra ID authentication instead.

Resolution:

• Verify that DefaultAzureCredential is configured correctly. If you use the Azure CLI, run az login. If you use the Azure Developer CLI, run azd auth login.
• Confirm your account has the Foundry User role on the Foundry project.
• Verify the project endpoint URL is correct and includes both the account and project names.

The Foundry RBAC roles were recently renamed. Foundry User, Foundry Owner, Foundry Account Owner, and Foundry Project Manager were previously named Azure AI User, Azure AI Owner, Azure AI Account Owner, and Azure AI Project Manager. You might still see the previous names in some places while the rename rolls out. The role IDs and core permissions are unchanged by the rename.

Data format or field mapping errors

If an evaluation fails with a schema, data mapping, or field mapping error, the test data doesn’t match what the evaluators expect.

Resolution:

• Verify your JSONL file has exactly one valid JSON object per line.
• Confirm that the field names in your data mapping match the field names in your dataset exactly. Field names are case-sensitive.
• Check that the schema you define, such as item_schema in the SDK, matches the fields in your dataset.
• For portal evaluations, verify your dataset contains the required columns for the evaluation scope. For conversation evaluations, make sure the messages column contains properly formatted chat messages.
• If you evaluate at the conversation level, remove turn-only evaluators or switch to turn-level evaluation. A turn-only evaluator used with conversation-level evaluation causes an incompatible evaluation level error.

Missing or zero evaluator scores

After a run completes, some evaluator scores might be missing or unexpectedly zero.

Symptom	Possible cause	Action
An evaluator’s metric is missing	The evaluator wasn’t selected when the evaluation was created	Rerun the evaluation and select the required evaluators.
All safety metrics are zero	The safety category is disabled, or the model doesn’t support the evaluator	Confirm model and evaluator support in Risk and safety evaluators.
Groundedness is unexpectedly low	The retrieval context is incomplete	Verify how context is constructed, and check retrieval latency.
Many rows show errors or low scores	Agent response or evaluator errors during the run	Open the run report, review the errored rows, fix the underlying errors, then rerun.

Agent evaluator tool errors

If an agent evaluator returns an error for unsupported tools:

• Check the supported tools for agent evaluators.
• As a workaround, wrap unsupported tools as user-defined function tools so the evaluator can assess them.

Azure Developer CLI (azd) evaluation issues

These issues apply when you run agent evaluations with the azd ai agent eval commands.

Issue	Solution
azd ai agent eval command not found or fails	Run azd ext list and verify the azd ai agent extension is 0.1.40-preview or later. Upgrade with azd ext upgrade azure.ai.agents.
Evaluation target not found or agent not invokable	Confirm the agent is deployed and invokable with azd ai agent show. Redeploy with azd deploy if needed.
Eval model deployment not found	Verify the chat-completion deployment name exists in your project under Build > Deployments.

For the full azd evaluation workflow, see Run agent evaluations with the azd CLI.

Trace evaluation issues

Trace evaluation runs evaluators against agent interactions that Application Insights already captured, instead of replaying requests.

Project managed identity is missing trace read permissions

The Foundry project’s managed identity reads traces from Application Insights. Without the right role, the service can’t query traces, and the trace evaluation returns no data or fails.

Symptoms:

• Trace evaluation fails with a permission or authorization error.
• The run finds no traces even though traces exist in Application Insights.

Resolution:

Assign the Log Analytics Reader role to the project’s managed identity on both the Application Insights resource and its linked Log Analytics workspace. To find the managed identity principal ID, see Verify the managed identity role assignment.

az role assignment create \
  --assignee <principal-id> \
  --role "Log Analytics Reader" \
  --scope "<application-insights-or-log-analytics-resource-id>"

Run the command twice: once for the Application Insights resource and once for the Log Analytics workspace it’s linked to. Role assignments can take up to 10 minutes to propagate. For setup details, see Set up tracing in Microsoft Foundry.

If the Log Analytics tables that store your traces are protected (their protection level is set to Protected), the Log Analytics Reader role can’t read them. In that case, also assign the Privileged Monitoring Data Reader role to the managed identity at the same scopes so trace evaluation can read the protected trace tables.

Fetched traces have no input or output messages

Quality evaluators read the query and response from each trace. If the fetched invoke_agent spans have neither the gen_ai.input.messages nor the gen_ai.output.messages attribute, the evaluators have no conversation content to score.

Symptoms:

• Quality evaluators, such as coherence, fluency, relevance, and intent resolution, return score=None.
• Safety evaluators run but don’t produce meaningful results.

Cause: The agent doesn’t emit the GenAI message attributes on its invoke_agent spans, so the captured traces don’t contain the conversation content. The evaluation service only reads spans where gen_ai.operation.name equals invoke_agent.

Resolution:

• Make sure your agent emits OpenTelemetry spans that follow the GenAI semantic conventions, including the gen_ai.input.messages and gen_ai.output.messages attributes on invoke_agent spans.

• For Python agents built with the Azure AI Agent Server SDK, install the tracing extra so spans are emitted automatically:

  pip install "azure-ai-agentserver-core[tracing]"
  

• In Application Insights, confirm the invoke_agent spans include the message attributes before you rerun the evaluation.

Human evaluation

This section covers common issues with the human evaluation feature for Foundry agents.

Feedback button doesn’t appear after the agent responds

Cause: No evaluation template is set as active for the agent.

Resolution: In the Human Evaluation tab, select Set as active for the desired template. Only one template can be active at a time. For more information, see Set up human evaluation for your agents.

No results visible in the Evaluation Results section

Cause: Application Insights isn’t configured for the project, or there’s a data ingestion delay (up to 5 minutes after an evaluation is submitted).

Resolution: Verify that Application Insights is connected to your project. For setup instructions, see Configure Application Insights for agent tracing. If Application Insights is already configured, wait a few minutes and refresh the page.

Reviewer can’t access the preview web app

Cause: The reviewer doesn’t have the required role on the Foundry project.

Resolution: Assign the Foundry User role to the reviewer on the Foundry project. For instructions, see Role-based access control in Microsoft Foundry.

Related content

• Add a new connection to your project
• Connect to your own storage
• Rate limits, region support, and enterprise features for evaluation
• Evaluate your AI agents
• Run agent evaluations with the azd CLI
• Run evaluations from the Foundry portal
• View evaluation results in the Foundry portal
• Set up tracing in Microsoft Foundry
• Set up human evaluation for your agents