> ## Documentation Index
> Fetch the complete documentation index at: https://hobbyist-e43fa225.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.
# Optimize agent instructions, skills, tools, and models in Foundry Agent Service (preview)
> Run instruction tuning, skill discovery, tool optimization, or model selection using the agent optimizer to automatically improve your hosted agent's performance in Foundry Agent Service.
Agent Optimizer is currently in preview. This preview is provided without a service-level agreement, and we don't recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see [Supplemental Terms of Use for Microsoft Azure Previews](https://azure.microsoft.com/support/legal/preview-supplemental-terms/).
The agent optimizer improves four aspects of your hosted agent: **instructions**, **skills**, **tools**, and **model selection**. It automatically detects which of these targets to optimize from your agent's baseline configuration.
This article shows how to run an optimization, configure and monitor the run, and deploy the results. For what each target does and when it activates, see [Optimization targets](/agents/agent-optimizer-overview#optimization-targets). To set up the baseline inputs, see [Make your agent optimizer-ready](/agents/make-agent-optimizer-ready). For a quick reference on what the optimizer changes, see [What each target changes](#what-each-target-changes).
## Prerequisites
* A [Foundry project](../../how-to/create-projects) with a deployed hosted agent
* The `azure.ai.agents` CLI extension installed (see [Quickstart: Optimize a hosted agent](/agents/quickstart-optimize-hosted-agent))
* A model deployed for evaluation (for example, `gpt-4.1-mini`) and an optimization model from the [supported list](/agents/agent-optimizer-overview#models) (for example, `gpt-5.1`)
* Your agent is [optimizer-ready](/agents/make-agent-optimizer-ready) (calls `load_config()`)
## Run an optimization
Start an optimization run with a single command:
```bash theme={null}
azd ai agent optimize
```
The optimizer evaluates your baseline, generates candidates, evaluates them, and ranks the results. For the full evaluate-and-improve cycle, see [How the agent optimizer works](/agents/agent-optimizer-overview#how-the-agent-optimizer-works). Which targets run depends on your baseline configuration—instruction tuning, skill improvement, and tool optimization activate automatically when the matching baseline files are present. See [Optimization targets](/agents/agent-optimizer-overview#optimization-targets).
To control the run with a config file, pass an `eval.yaml` that references your dataset, evaluators, and options:
```bash theme={null}
azd ai agent optimize --config eval.yaml
```
For the full `eval.yaml` schema, see [Configure the optimization run](#configure-the-optimization-run).
### Target a specific agent
By default, the CLI optimizes the agent detected from your current `azd` environment and your project's local `agent.yaml`. The optimizer resolves the agent name in this priority order:
| Priority | Source | Example |
| ----------- | -------------------------------------------------------------------- | ------------------------------------------------ |
| 1 (highest) | `--agent` CLI flag | `azd ai agent optimize --agent my-support-agent` |
| 2 (default) | Current `azd` environment and the `name` field in local `agent.yaml` | `name: my-support-agent` |
| 3 | `agent.name` field in `eval.yaml` | `agent:\n name: my-support-agent` |
Use the `--agent` flag when you have multiple agents in your project or want to override the default. The agent name must match a deployed hosted agent in your Foundry project.
Run `azd ai agent invoke "test"` to verify your agent responds before starting optimization.
## Configure the optimization run
Configure optimization runs through an `eval.yaml` file that ties together your dataset, evaluators, and run options. The command `azd ai agent eval generate` writes this file for you, or you can create it by hand. The optimizer auto-detects `eval.yaml` in your project root, or you can pass it explicitly with `--config eval.yaml`.
```yaml theme={null}
# eval.yaml
name: my-optimization # Optional label for the run
agent:
name: my-agent # Deployed hosted agent name
kind: hosted
version: "1" # Agent version (optional)
model: gpt-4.1-mini # Baseline model deployment
config: .agent_configs/baseline/metadata.yaml
dataset:
local_uri: ./eval.jsonl # A local JSONL file...
# name: my-foundry-dataset # ...OR a registered Foundry dataset
# version: "1"
# validation_dataset: # Optional held-out dataset
# name: my-validation-dataset
# version: "1"
evaluators:
- builtin.task_adherence # A built-in evaluator...
# - name: my-custom-evaluator # ...or a custom evaluator
# version: "1"
# local_uri: ./my_evaluator.json
options:
eval_model: gpt-4.1-mini # Scores responses
optimization_model: gpt-5.1 # Generates candidates
max_candidates: 4
optimization_config:
model_search_space: # Optional: compare model deployments
- gpt-4.1
```
| Field | Required | Description |
| ------------------------------------------------ | -------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| `name` | No | Label for the optimization run. |
| `agent.name` | Yes | Name of the deployed hosted agent to optimize. |
| `agent.kind` | Yes | Agent kind. Use `hosted`. |
| `agent.version` | No | Agent version to target. |
| `agent.model` | Yes | Baseline model deployment name. |
| `agent.config` | Yes | Path to the baseline `metadata.yaml`. |
| `dataset` | Yes | The dataset to evaluate against, as a local JSONL file (`local_uri`) or a registered Foundry dataset (`name` and `version`). See [Create a custom dataset](/agents/create-optimizer-dataset#create-a-custom-dataset-advanced). |
| `validation_dataset` | No | A held-out dataset used to validate results. |
| `evaluators` | Yes | Evaluators applied to every task. See [Customize evaluators](/agents/create-optimizer-dataset#customize-evaluators-advanced). |
| `options.eval_model` | Yes | Deployed chat model that scores responses. See [Choose the eval and optimization models](#choose-the-eval-and-optimization-models). |
| `options.optimization_model` | Yes | Deployed model that generates candidates. Must be on the [supported list](/agents/agent-optimizer-overview#models). |
| `options.max_candidates` | No | Number of candidates to generate (default 5). See [Set the number of candidates](#set-the-number-of-candidates). |
| `options.optimization_config.model_search_space` | No | Model deployments to compare during model selection. See [Evaluate multiple models](#evaluate-multiple-models). |
Author the dataset and evaluators separately; see [Create an evaluation dataset and evaluators](/agents/create-optimizer-dataset). The following sections describe the run options.
### Choose the eval and optimization models
The optimizer uses two models: an *eval model* that scores agent responses against criteria, and an *optimization model* that generates candidate configurations. Set them in `eval.yaml` or use CLI flags.
```yaml theme={null}
options:
eval_model: gpt-4.1-mini
optimization_model: gpt-5.1
```
```bash theme={null}
azd ai agent optimize --eval-model gpt-4.1-mini --optimize-model gpt-5.1
```
Any chat-completion model deployed in your project works as the eval model. The optimization model must be from the supported list. For roles and supported models, see [Models](/agents/agent-optimizer-overview#models).
The `optimization_model` field is required. If you don't specify it and don't pass `--optimize-model`, the optimization API returns an error. Always verify that both models are deployed in your project before you run optimization.
### Set the number of candidates
The `max_candidates` option sets the expected number of candidate configurations for the run. The optimizer typically returns after it reaches that count, unless the run stops early because of an error or another stopping condition.
| Max candidates | Candidates | Time | Best for |
| -------------- | ---------- | ------------ | -------------------- |
| 2 | 2 | 5 to 10 min | Quick experiments |
| 5 (default) | 5 | 20 to 30 min | Good balance |
| 10 | 10 | 30 to 60 min | Thorough exploration |
Higher values explore more variations but take longer. The optimizer learns from earlier candidates, so later candidates tend to score higher.
Times are approximate for a dataset of 3 to 10 tasks. Larger datasets or slower eval models increase run duration.
### Evaluate multiple models
To compare model deployments in a single run, list them under `optimization_config.model_search_space`. The optimizer evaluates your agent with each model against the same dataset and ranks the results by score and token cost.
```yaml theme={null}
# eval.yaml
options:
optimization_config:
model_search_space:
- gpt-4.1
- gpt-4.1-mini
- gpt-4o
```
Each model listed under `model_search_space` must be deployed in your Foundry project.
If the list includes your agent's current model deployment, the optimizer automatically removes it from the candidates because the baseline already represents that model. If no models remain after this removal, you receive a validation error.
Model selection runs alongside the targets that activate automatically from your baseline. A single run can produce candidates that combine improved instructions, skills, and tool descriptions with different model options - you don't configure the combination yourself.
## Monitor a running job
An optimization run is asynchronous. Use these commands when a job is long-running or you want to check its progress:
```bash theme={null}
# Check status and stream progress
azd ai agent optimize status --watch
# List recent optimization jobs
azd ai agent optimize list
# Cancel a running job
azd ai agent optimize cancel
```
Capture the operation ID, portal URL, scores, and candidate IDs from the run output. You can also monitor the job in the [Foundry portal](https://ai.azure.com) using the URL shown when the run starts.
## Interpret results
After optimization completes, review the results table. An asterisk (`*`) marks the best candidate. For the results table columns, scoring details, score-improvement thresholds, and the portal view, see [Understand optimization results](/agents/agent-optimizer-overview#understand-optimization-results).
## Deploy the winner
The recommended workflow is to apply the optimized config locally, then deploy:
```bash theme={null}
# Apply the winning candidate locally
azd ai agent optimize apply --candidate
# Deploy with the optimized config
azd deploy
```
This downloads the optimized configuration into `.agent_configs//` in your project. On next deploy, your agent uses the improved instructions and tool descriptions.
Alternatively, you can deploy directly via the API (useful for quick A/B testing):
```bash theme={null}
azd ai agent optimize deploy --candidate
```
Direct deploy updates the agent service without changing your local files. Use the `apply` -> `deploy` workflow for production.
If all candidates score lower than the baseline, don't deploy any candidate. The baseline configuration remains active.
## What each target changes
The optimizer automatically activates the targets that apply to your baseline. This section is a reference for what a run changes. Use the following table to anticipate what optimization does for your agent:
| Scenario | Target |
| ------------------------------------------------------------- | ---------------------------------------- |
| Improve overall response quality | Instruction tuning |
| Reduce incorrect information | Instruction tuning |
| Improve repeatable behaviors (escalation, debugging patterns) | Skill improvement |
| Refine structured procedures | Skill improvement |
| Find the best quality/cost model trade-off | Model selection |
| First optimization, not sure what to expect | All applicable targets run automatically |
Your code stays the same across all targets because `load_config()` returns the optimized values automatically. Only the configuration the model sees changes.
### Instructions
The optimizer rewrites the system prompt. Common improvements include:
* Adding explicit constraints that the original prompt implied but didn't state
* Restructuring instructions for clarity
* Adding output format specifications
* Strengthening safety and scope boundaries
For example, a minimal baseline prompt like `You are a helpful assistant.` might become:
```
You are a helpful coding assistant. Follow these guidelines:
1. Always include working code examples
2. Explain your reasoning step by step
3. If a question is outside your expertise, say so clearly
4. Use markdown formatting for code blocks
5. Handle edge cases in code examples
```
### Skills
The optimizer refines each skill's description, body, and activation criteria while keeping the skill's purpose intact. The agent loads improved skills through `load_config()`, which appends them to the instruction set. Skills use the open [Agent Skills](https://agentskills.io) format. For how your agent loads skills, see [Make your agent optimizer-ready](/agents/make-agent-optimizer-ready#load-and-use-the-config).
### Tools
The optimizer refines your `tools.json` definitions. Common improvements include:
* Clearer function descriptions that help the model know when to call a tool
* More specific parameter descriptions that reduce inaccurate arguments
* Added constraints (enums, required fields) that prevent invalid inputs
Your tool implementation code stays the same. Only the definitions the model sees change.
### Models
The optimizer ranks each candidate model by composite score and token cost, so you can choose the best quality-to-cost trade-off. To configure the candidates, see [Evaluate multiple models](#evaluate-multiple-models).
## Troubleshooting
| Problem | Cause | Fix |
| -------------------------- | ---------------------------------- | ----------------------------------------------------------------------------------- |
| `optimize` returns 400 | Subscription not on allow list | Contact your Microsoft representative to request access |
| Protocol validation error | Invalid `azure.yaml` agent service | Ensure the `azure.ai.agent` service includes `kind: hosted` and a `protocols:` list |
| Job stuck at "running" | Service issue | Cancel with `azd ai agent optimize cancel ` and retry |
| No candidate IDs in output | Job still running | Wait for completion or use `--watch` |
## Related content
* [Agent optimizer overview](/agents/agent-optimizer-overview)
* [Make your agent optimizer-ready](/agents/make-agent-optimizer-ready)
* [Create an evaluation dataset and evaluators](/agents/create-optimizer-dataset)
* [Quickstart: Optimize a hosted agent](/agents/quickstart-optimize-hosted-agent)