Optimize agent instructions, skills, tools, and models in Foundry Agent Service (preview)

Run instruction tuning, skill discovery, tool optimization, or model selection using the agent optimizer to automatically improve your hosted agent’s performance in Foundry Agent Service.

Agent Optimizer is currently in preview. This preview is provided without a service-level agreement, and we don’t recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

The agent optimizer improves four aspects of your hosted agent: instructions, skills, tools, and model selection. It automatically detects which of these targets to optimize from your agent’s baseline configuration.

This article shows how to run an optimization, configure and monitor the run, and deploy the results. For what each target does and when it activates, see Optimization targets. To set up the baseline inputs, see Make your agent optimizer-ready. For a quick reference on what the optimizer changes, see What each target changes.

Prerequisites

• A Foundry project with a deployed hosted agent
• The azure.ai.agents CLI extension installed (see Quickstart: Optimize a hosted agent)
• A model deployed for evaluation (for example, gpt-4.1-mini) and an optimization model from the supported list (for example, gpt-5.1)
• Your agent is optimizer-ready (calls load_config())

Run an optimization

Start an optimization run with a single command:

azd ai agent optimize

The optimizer evaluates your baseline, generates candidates, evaluates them, and ranks the results. For the full evaluate-and-improve cycle, see How the agent optimizer works. Which targets run depends on your baseline configuration—instruction tuning, skill improvement, and tool optimization activate automatically when the matching baseline files are present. See Optimization targets.

To control the run with a config file, pass an eval.yaml that references your dataset, evaluators, and options:

azd ai agent optimize --config eval.yaml

For the full eval.yaml schema, see Configure the optimization run.

Target a specific agent

How the CLI resolves the agent depends on whether you run the command from an azd project:

Context	Agent resolution	Example
In an azd project	The CLI detects the hosted-agent service from azure.yaml and resolves its deployed agent name from the current azd environment. Use --agent to select an azure.yaml service when the project contains multiple agents.	azd ai agent optimize --agent support-service
Outside an azd project	The --agent value or positional argument is the deployed Foundry agent name.	azd ai agent optimize --agent my-support-agent
With --config	The agent.name field in eval.yaml supplies the deployed agent name. An explicit --agent value overrides it.	agent:\n name: my-support-agent

The deployed agent name must match a hosted agent in the target Foundry project.

Run azd ai agent invoke "test" to verify your agent responds before starting optimization.

Optimize an existing agent without AZD project files

You can optimize an existing hosted agent without running azd ai agent init and without creating azure.yaml or a .azure environment directory. In this standalone flow, provide the Foundry project endpoint and deployed agent name explicitly.

1. Ensure the deployed agent is optimizer-ready. In a local working directory, create the instruction file, dataset, evaluators, and eval.yaml described in Configure the optimization run.

   Run the command from this working directory. Without an azd project, relative paths in eval.yaml resolve from the current working directory.

   For this standalone flow, omit agent.config. The CLI asks for the baseline instruction when you run the command:

   # eval.yaml
   agent:
     name: my-support-agent
     kind: hosted
     model: gpt-4.1-mini
   dataset:
     local_uri: ./eval.jsonl
   evaluators:
     - builtin.task_adherence
   options:
     eval_model: gpt-4.1-mini
     optimization_model: gpt-5.1
     max_candidates: 2
   

2. Authenticate:

   az login
   azd auth login
   

3. Copy the project endpoint from the Foundry project’s Overview page. Use the project endpoint URL, not the Azure resource ID.

4. Save the endpoint in your user-level azd config so subsequent commands can resolve the same project from any directory:

   azd ai project set "<project-endpoint>"
   azd ai project show
   

   This step writes the default endpoint to ~/.azd/config.json. For the full resolution order and commands to inspect or clear the saved context, see Set the Foundry project context for azd commands.

5. Run the optimization with the deployed agent name:

   azd ai agent optimize --agent "<deployed-agent-name>" --config eval.yaml
   

   When prompted for the agent instruction, provide it inline or select a file such as .agent_configs/baseline/instructions.md.

In the current preview, a standalone run doesn’t expand agent.config from eval.yaml. Run the command interactively so you can provide the baseline instruction. Don’t use --no-prompt for this flow. Loading file-based skill and tool baselines also requires an azd project.

For a one-off command that shouldn’t change your user-level config, pass --project-endpoint:

azd ai agent optimize \
  --project-endpoint "<project-endpoint>" \
  --agent "<deployed-agent-name>" \
  --config eval.yaml

You can also set the endpoint for the current shell:

export FOUNDRY_PROJECT_ENDPOINT="<project-endpoint>"
azd ai agent optimize --agent "<deployed-agent-name>" --config eval.yaml

1. Save the operation ID from the command output. Because this flow has no azd environment, the CLI doesn’t persist the last operation ID locally. Pass the operation ID to follow-up commands:

   azd ai agent optimize status <operation-id> --watch
   
   azd ai agent optimize list
   
   azd ai agent optimize cancel <operation-id>
   

   These commands use the endpoint saved by azd ai project set. If you used the one-off --project-endpoint form instead, pass the flag again to each follow-up command.

azd ai agent optimize apply requires an azd project because it writes candidate files under .agent_configs/ and updates the agent service in azure.yaml. If you don’t want to create AZD project files, review and deploy the winning candidate from the Foundry portal.

Configure the optimization run

Configure optimization runs through an eval.yaml file that ties together your dataset, evaluators, and run options. The command azd ai agent eval generate writes this file for you, or you can create it by hand. The optimizer auto-detects eval.yaml in your project root, or you can pass it explicitly with --config eval.yaml.

# eval.yaml
name: my-optimization              # Optional label for the run
agent:
  name: my-agent                   # Deployed hosted agent name
  kind: hosted
  version: "1"                     # Agent version (optional)
  model: gpt-4.1-mini              # Baseline model deployment
  config: .agent_configs/baseline/metadata.yaml
dataset:
  local_uri: ./eval.jsonl          # A local JSONL file...
  # name: my-foundry-dataset       # ...OR a registered Foundry dataset
  # version: "1"
# validation_dataset:              # Optional held-out dataset
#   name: my-validation-dataset
#   version: "1"
evaluators:
  - builtin.task_adherence         # A built-in evaluator...
  # - name: my-custom-evaluator    # ...or a custom evaluator
  #   version: "1"
  #   local_uri: ./my_evaluator.json
options:
  eval_model: gpt-4.1-mini         # Scores responses
  optimization_model: gpt-5.1      # Generates candidates
  max_candidates: 4
  optimization_config:
    model_search_space:            # Optional: compare model deployments
      - gpt-4.1

Field	Required	Description
name	No	Label for the optimization run.
agent.name	Yes	Name of the deployed hosted agent to optimize.
agent.kind	Yes	Agent kind. Use hosted.
agent.version	No	Agent version to target.
agent.model	Yes	Baseline model deployment name.
agent.config	Conditional	Path to the baseline metadata.yaml in an azd project. For a standalone project with no AZD files, omit this field and provide the instruction interactively.
dataset	Yes	The dataset to evaluate against, as a local JSONL file (local_uri) or a registered Foundry dataset (name and version). See Create a custom dataset.
validation_dataset	No	A held-out dataset used to validate results.
evaluators	Yes	Evaluators applied to every task. See Customize evaluators.
options.eval_model	Yes	Deployed chat model that scores responses. See Choose the eval and optimization models.
options.optimization_model	Yes	Deployed model that generates candidates. Must be on the supported list.
options.max_candidates	No	Number of candidates to generate (default 5). See Set the number of candidates.
options.optimization_config.model_search_space	No	Model deployments to compare during model selection. See Evaluate multiple models.

Author the dataset and evaluators separately; see Create an evaluation dataset and evaluators. The following sections describe the run options.

Choose the eval and optimization models

The optimizer uses two models: an eval model that scores agent responses against criteria, and an optimization model that generates candidate configurations. Set them in eval.yaml or use CLI flags.

options:
  eval_model: gpt-4.1-mini
  optimization_model: gpt-5.1

azd ai agent optimize --eval-model gpt-4.1-mini --optimize-model gpt-5.1

Any chat-completion model deployed in your project works as the eval model. The optimization model must be from the supported list. For roles and supported models, see Models.

The optimization_model field is required. If you don’t specify it and don’t pass --optimize-model, the optimization API returns an error. Always verify that both models are deployed in your project before you run optimization.

Set the number of candidates

The max_candidates option sets the expected number of candidate configurations for the run. The optimizer typically returns after it reaches that count, unless the run stops early because of an error or another stopping condition.

Max candidates	Candidates	Time	Best for
2	2	5 to 10 min	Quick experiments
5 (default)	5	20 to 30 min	Good balance
10	10	30 to 60 min	Thorough exploration

Higher values explore more variations but take longer. The optimizer learns from earlier candidates, so later candidates tend to score higher.

Times are approximate for a dataset of 3 to 10 tasks. Larger datasets or slower eval models increase run duration.

Evaluate multiple models

To compare model deployments in a single run, list them under optimization_config.model_search_space. The optimizer evaluates your agent with each model against the same dataset and ranks the results by score and token cost.

# eval.yaml
options:
  optimization_config:
    model_search_space:
      - gpt-4.1
      - gpt-4.1-mini
      - gpt-4o

Each model listed under model_search_space must be deployed in your Foundry project.

If the list includes your agent’s current model deployment, the optimizer automatically removes it from the candidates because the baseline already represents that model. If no models remain after this removal, you receive a validation error.

Model selection runs alongside the targets that activate automatically from your baseline. A single run can produce candidates that combine improved instructions, skills, and tool descriptions with different model options - you don’t configure the combination yourself.

Monitor a running job

An optimization run is asynchronous. Use these commands when a job is long-running or you want to check its progress:

# Check status and stream progress
azd ai agent optimize status <operation-id> --watch

# List recent optimization jobs
azd ai agent optimize list

# Cancel a running job
azd ai agent optimize cancel <operation-id>

Capture the operation ID, portal URL, scores, and candidate IDs from the run output. You can also monitor the job in the Foundry portal using the URL shown when the run starts.

If you started the job without AZD project files, always pass the operation ID to status and cancel. The commands use the user-level endpoint saved by azd ai project set; otherwise, include --project-endpoint.

Interpret results

After optimization completes, review the results table. An asterisk (*) marks the best candidate. For the results table columns, scoring details, score-improvement thresholds, and the portal view, see Understand optimization results.

Deploy the winner

The recommended workflow is to apply the optimized config locally, then deploy:

# Apply the winning candidate locally
azd ai agent optimize apply --candidate <candidate-id>

# Deploy with the optimized config
azd deploy

This downloads the optimized configuration into .agent_configs/<candidate_id>/ in your project. On next deploy, your agent uses the improved instructions and tool descriptions.

Alternatively, you can deploy directly via the API (useful for quick A/B testing):

azd ai agent optimize deploy --candidate <candidate-id>

Direct deploy updates the agent service without changing your local files. Use the apply -> deploy workflow for production.

In the current preview, direct deploy resolves the optimization job from an azd environment. For a standalone optimization that has no AZD environment, deploy the candidate from the Foundry portal.

If all candidates score lower than the baseline, don’t deploy any candidate. The baseline configuration remains active.

What each target changes

The optimizer automatically activates the targets that apply to your baseline. This section is a reference for what a run changes. Use the following table to anticipate what optimization does for your agent:

Scenario	Target
Improve overall response quality	Instruction tuning
Reduce incorrect information	Instruction tuning
Improve repeatable behaviors (escalation, debugging patterns)	Skill improvement
Refine structured procedures	Skill improvement
Find the best quality/cost model trade-off	Model selection
First optimization, not sure what to expect	All applicable targets run automatically

Your code stays the same across all targets because load_config() returns the optimized values automatically. Only the configuration the model sees changes.

Instructions

The optimizer rewrites the system prompt. Common improvements include:

• Adding explicit constraints that the original prompt implied but didn’t state
• Restructuring instructions for clarity
• Adding output format specifications
• Strengthening safety and scope boundaries

For example, a minimal baseline prompt like You are a helpful assistant. might become:

You are a helpful coding assistant. Follow these guidelines:
1. Always include working code examples
2. Explain your reasoning step by step
3. If a question is outside your expertise, say so clearly
4. Use markdown formatting for code blocks
5. Handle edge cases in code examples

Skills

The optimizer refines each skill’s description, body, and activation criteria while keeping the skill’s purpose intact. The agent loads improved skills through load_config(), which appends them to the instruction set. Skills use the open Agent Skills format. For how your agent loads skills, see Make your agent optimizer-ready.

Tools

The optimizer refines your tools.json definitions. Common improvements include:

• Clearer function descriptions that help the model know when to call a tool
• More specific parameter descriptions that reduce inaccurate arguments
• Added constraints (enums, required fields) that prevent invalid inputs

Your tool implementation code stays the same. Only the definitions the model sees change.

Models

The optimizer ranks each candidate model by composite score and token cost, so you can choose the best quality-to-cost trade-off. To configure the candidates, see Evaluate multiple models.

Troubleshooting

Problem	Cause	Fix
optimize returns 400	Subscription not on allow list	Contact your Microsoft representative to request access
could not resolve project endpoint	No project endpoint is available from an azd environment or user-level config	Run azd ai project set <project-endpoint>, pass --project-endpoint <project-endpoint>, or set FOUNDRY_PROJECT_ENDPOINT
agent name is required	The command is running outside an azd project and no deployed agent name was provided	Pass --agent <deployed-agent-name> or provide the agent name as a positional argument
operation ID is required	A standalone run has no azd environment in which to persist the last operation ID	Copy the operation ID from the optimization output and pass it to status or cancel
instruction is required for optimization in a standalone folder	A standalone run doesn’t expand agent.config from eval.yaml in the current preview	Run without --no-prompt, then provide the baseline instruction inline or select the instruction file
optimize apply can’t resolve an agent service	apply requires an azure.yaml hosted-agent service in an azd project	Deploy the candidate from the Foundry portal, or initialize an azd project before using apply
Protocol validation error	Invalid azure.yaml agent service	Ensure the azure.ai.agent service includes kind: hosted and a protocols: list
Job stuck at “running”	Service issue	Cancel with azd ai agent optimize cancel <id> and retry
No candidate IDs in output	Job still running	Wait for completion or use --watch

Related content

• Agent optimizer overview
• Make your agent optimizer-ready
• Create an evaluation dataset and evaluators
• Quickstart: Optimize a hosted agent