Quickstart: Optimize a hosted agent (preview)

Deploy the optimization sample agent, run the agent optimizer to automatically improve its instructions, and deploy the winning candidate.

Agent Optimizer is currently in preview. This preview is provided without a service-level agreement, and we don’t recommend it for production workloads. Certain features might not be supported or might have constrained capabilities. For more information, see Supplemental Terms of Use for Microsoft Azure Previews.

In this quickstart, you deploy the optimization sample agent, run the agent optimizer to improve its instructions, and deploy the winning candidate.

For the concepts behind each step and the full end-to-end path, see the optimization workflow.

Prerequisites

Before you begin, you need:

• An Azure subscription—Create one for free.

• Your Azure subscription must be on the allow list for the agent optimizer. Contact your Microsoft representative to request access.

The agent optimizer is currently in preview.

Clean up resources

If you used the preceding Azure Developer CLI workflow, delete the provisioned resources when you finish experimenting:

azd down --force --purge

Why --purge? Foundry accounts use soft-delete by default. Without --purge, the resource name stays reserved for 48 hours, and reprovisioning with the same name fails.

Troubleshooting

Problem	Cause	Fix
azd ai agent optimize command not found	Extension too old	Run azd ext upgrade microsoft.foundry to get 0.1.40-preview or later.
optimization_model is required	Running in non-interactive mode without a model configured	Add --optimize-model gpt-5 to the command, or set optimization_model: gpt-5 under options: in eval.yaml. In interactive mode, the CLI prompts for model selection.
Python script fails with KeyError: 'DATASET_NAME' or another missing variable	The script didn’t load your .env file, or the variable is missing	Run the script from the same folder as .env, or export the required values in your shell before running python optimize_hosted_agent.py.
Python script fails with ResourceNotFound: The project does not exist	FOUNDRY_PROJECT_ENDPOINT doesn’t point to an existing Foundry project	Copy the project endpoint from the Foundry project’s Overview page and update FOUNDRY_PROJECT_ENDPOINT in .env.
Python script fails with Optimization model deployment '<name>' not found	OPTIMIZATION_MODEL is not the name of a deployed model in your Foundry project	Use the exact deployment name from Build > Deployments, such as an existing gpt-5 family or DeepSeek deployment in your project.
Optimization score is 0 or very low	Evaluation has many errored rows	Open the Eval link in the results. Fix response generation or evaluator errors, then rerun.
azd provision fails with quota error	Subscription lacks capacity	Try a different region or request a quota increase.

What you learned

In this quickstart, you:

• Deployed the optimization sample agent by using the customer-support template.
• Ran the agent optimizer to automatically improve agent instructions.
• Deployed the winning candidate and verified the improvement.

Next steps

• Agent optimizer overview
• Make your agent optimizer-ready
• Create an evaluation dataset and evaluators
• Optimize agent instructions, skills, tools, and models
• Run agent evaluations with the azd CLI