Monitor model router in Microsoft Foundry

Learn how to inspect preview model router metadata for routing attempts, fallback, latency, and Chat Completions session affinity in Microsoft Foundry.

Observability helps you understand how model router handles requests, verify routing behavior, and investigate latency, errors, and fallback. Request-level signals complement aggregate metrics and logs, giving developers and operators context to evaluate application performance.

This article covers the per-request routing metadata preview for the Chat Completions API. The metadata identifies the serving model and describes routing attempts for an individual request. For aggregate metrics and logs, see Monitor model deployments.

Prerequisites

• Python 3.9 or later.
• The openai>=1.75.0 and python-dotenv packages. Install them by running pip install "openai>=1.75.0" python-dotenv.
• A model router deployment that you can access through an Azure OpenAI endpoint.
• The endpoint and API key for your Azure OpenAI resource. The complete sample reads them from the AZURE_OPENAI_ENDPOINT and AZURE_OPENAI_API_KEY environment variables.
• Azure OpenAI API version 2024-10-21.

Enable per-request routing metadata

After your application reads the endpoint and API key into endpoint and api_key, create the client with the preview feature header:

"""
Foundry Model Router - Chat Completions Observability Example

This example demonstrates how to use Azure OpenAI's Chat Completions API
with a Foundry Model Router deployment and inspect the selected model,
routing attempts, latency, and status. Model Router automatically
selects the best underlying LLM for each prompt based on your routing mode
(Balanced, Quality, or Cost).

Prerequisites:
  - An Azure OpenAI resource with a "model-router" deployment
    - A .env file beside this script with AZURE_OPENAI_ENDPOINT,
        AZURE_OPENAI_API_KEY, and MODEL_DEPLOYMENT_NAME

Usage:
    pip install -r requirements.txt
    python model-router-chat-completions-observability.py
"""

import os
from pathlib import Path

from dotenv import load_dotenv
from openai import AzureOpenAI

# Load environment variables from .env beside this script
load_dotenv(Path(__file__).resolve().parent / ".env", override=True)

endpoint = os.environ["AZURE_OPENAI_ENDPOINT"]
api_key = os.environ["AZURE_OPENAI_API_KEY"]
deployment = os.environ["MODEL_DEPLOYMENT_NAME"]

# <response_observability_enable>
client = AzureOpenAI(
    azure_endpoint=endpoint,
    api_key=api_key,
    api_version="2024-10-21",
    default_headers={"Foundry-Features": "ModelRouterControls=V1Preview"},
)
# </response_observability_enable>

response = client.chat.completions.create(
    model=deployment,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": "In one sentence, name the most popular tourist destination in Seattle.",
        },
    ],
)

print("--- Chat Completions Response ---")
print(f"Response:{response.choices[0].message.content}")
print(
    f"Usage: {response.usage.prompt_tokens} prompt + {response.usage.completion_tokens} completion = {response.usage.total_tokens} total tokens"
)

# <response_observability_extract>
print(f"\nRouted to model: {response.model}")
print("--- Model Selection Details ---")
model_selection_details = getattr(response, "model_selection_details", None)
if not model_selection_details:
    print("No model selection details were returned.")
else:
    model_router_details = model_selection_details.get("model_router_details", {})
    print(f"Routing mode: {model_router_details.get('mode', 'unknown')}")

    routing_trace = model_router_details.get("routing_trace", [])
    if not routing_trace:
        print("No routing trace was returned.")

    for decision_number, routing_decision in enumerate(routing_trace, start=1):
        latency_ms = routing_decision.get("latency_ms")
        latency = f"{latency_ms} ms" if latency_ms is not None else "not reported"
        print(f"Routing decision {decision_number} (latency: {latency})")

        for attempt_number, attempt in enumerate(
            routing_decision.get("attempts", []), start=1
        ):
            result = attempt.get("result", {})
            status = result.get("status", "unknown")
            outcome = (
                "selected"
                if isinstance(status, int) and 200 <= status < 300
                else "failed"
            )
            print(
                f"  Attempt {attempt_number}: {attempt.get('model', 'unknown')} - HTTP {status} ({outcome})"
            )

            error = result.get("error")
            if error:
                print(
                    f"    Error: {error.get('code', 'unknown')} - {error.get('message', 'No message')}"
                )
    print("\n")
# </response_observability_extract>

The Foundry-Features: ModelRouterControls=V1Preview header requests per-request routing metadata. Because this feature is in preview, the metadata presence and response schema can vary by request and service version.

Send a Chat Completions request

Use the model router deployment name to send a request. The response includes the completion and, when available, the per-request routing metadata:

response = client.chat.completions.create(
   model=deployment,
   messages=[
      {"role": "system", "content": "You are a helpful assistant."},
      {
         "role": "user",
         "content": "In one sentence, name the most popular tourist destination in Seattle.",
      },
   ],
)

Understand routing metadata

The following model_selection_details fragment illustrates a request with two ordered model attempts:

{
   "model_selection_details": {
      "model_router_details": {
         "mode": "balanced",
         "routing_trace": [
            {
               "latency_ms": 19,
               "attempts": [
                  {
                     "model": "example-model-a",
                     "result": {
                        "status": 404,
                        "error": {
                           "code": "NotFound",
                           "message": "The request failed."
                        }
                     }
                  },
                  {
                     "model": "example-model-b",
                     "result": {
                        "status": 200
                     }
                  }
               ]
            }
         ]
      }
   }
}

• mode is the routing mode returned for the request.
• routing_trace contains the routing entries returned for the request.
• latency_ms is the latency reported for a routing-trace entry.
• attempts lists model attempts in order.
• Each attempt contains a model and an HTTP status in result.status.
• A failed attempt can include an optional error with a code and message.

Extract routing and fallback information

After the Chat Completions request returns response, inspect the serving model and model selection details:

"""
Foundry Model Router - Chat Completions Observability Example

This example demonstrates how to use Azure OpenAI's Chat Completions API
with a Foundry Model Router deployment and inspect the selected model,
routing attempts, latency, and status. Model Router automatically
selects the best underlying LLM for each prompt based on your routing mode
(Balanced, Quality, or Cost).

Prerequisites:
  - An Azure OpenAI resource with a "model-router" deployment
    - A .env file beside this script with AZURE_OPENAI_ENDPOINT,
        AZURE_OPENAI_API_KEY, and MODEL_DEPLOYMENT_NAME

Usage:
    pip install -r requirements.txt
    python model-router-chat-completions-observability.py
"""

import os
from pathlib import Path

from dotenv import load_dotenv
from openai import AzureOpenAI

# Load environment variables from .env beside this script
load_dotenv(Path(__file__).resolve().parent / ".env", override=True)

endpoint = os.environ["AZURE_OPENAI_ENDPOINT"]
api_key = os.environ["AZURE_OPENAI_API_KEY"]
deployment = os.environ["MODEL_DEPLOYMENT_NAME"]

# <response_observability_enable>
client = AzureOpenAI(
    azure_endpoint=endpoint,
    api_key=api_key,
    api_version="2024-10-21",
    default_headers={"Foundry-Features": "ModelRouterControls=V1Preview"},
)
# </response_observability_enable>

response = client.chat.completions.create(
    model=deployment,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": "In one sentence, name the most popular tourist destination in Seattle.",
        },
    ],
)

print("--- Chat Completions Response ---")
print(f"Response:{response.choices[0].message.content}")
print(
    f"Usage: {response.usage.prompt_tokens} prompt + {response.usage.completion_tokens} completion = {response.usage.total_tokens} total tokens"
)

# <response_observability_extract>
print(f"\nRouted to model: {response.model}")
print("--- Model Selection Details ---")
model_selection_details = getattr(response, "model_selection_details", None)
if not model_selection_details:
    print("No model selection details were returned.")
else:
    model_router_details = model_selection_details.get("model_router_details", {})
    print(f"Routing mode: {model_router_details.get('mode', 'unknown')}")

    routing_trace = model_router_details.get("routing_trace", [])
    if not routing_trace:
        print("No routing trace was returned.")

    for decision_number, routing_decision in enumerate(routing_trace, start=1):
        latency_ms = routing_decision.get("latency_ms")
        latency = f"{latency_ms} ms" if latency_ms is not None else "not reported"
        print(f"Routing decision {decision_number} (latency: {latency})")

        for attempt_number, attempt in enumerate(
            routing_decision.get("attempts", []), start=1
        ):
            result = attempt.get("result", {})
            status = result.get("status", "unknown")
            outcome = (
                "selected"
                if isinstance(status, int) and 200 <= status < 300
                else "failed"
            )
            print(
                f"  Attempt {attempt_number}: {attempt.get('model', 'unknown')} - HTTP {status} ({outcome})"
            )

            error = result.get("error")
            if error:
                print(
                    f"    Error: {error.get('code', 'unknown')} - {error.get('message', 'No message')}"
                )
    print("\n")
# </response_observability_extract>

Ordered attempts can reveal automatic fallback for an individual request. In the example response, the failed attempt followed by a successful attempt is evidence of fallback for that request. Requests don’t always include multiple attempts, so don’t expect fallback on every request.

For complete application setup and runnable examples, see the Foundry Model Router samples.

Interpret session affinity metadata

When you enable the Chat Completions session affinity preview, model_router_details can include a session_affinity object. The following response fragment shows a request that retained its associated model:

{
   "model": "example-model-a",
   "model_selection_details": {
      "model_router_details": {
         "mode": "balanced",
         "session_affinity": {
            "mode": "sticky",
            "source": "session_id_payload",
            "decision": "retain"
         }
      }
   }
}

Interpret the fields as follows:

Field	Value	Meaning
mode	sticky	Model router attempts the associated eligible model first.
source	session_id_payload or session_id_header	The request body or header supplied the session ID. The response doesn’t return the identifier.
decision	initialize	No previous association was available, and the initially selected model served the response.
decision	retain	The associated model served the response.
decision	switch	A different model served because of eligibility or fallback.

When decision is switch, inspect routing_trace and the top-level model field together. The following fragment shows an associated model that returned a retryable response before fallback selected another model:

{
   "model": "example-model-b",
   "model_selection_details": {
      "model_router_details": {
         "mode": "balanced",
         "session_affinity": {
            "mode": "sticky",
            "source": "session_id_payload",
            "decision": "switch"
         },
         "routing_trace": [
            {
               "latency_ms": 51,
               "attempts": [
                  {
                     "model": "example-model-a",
                     "result": { "status": 429 }
                  },
                  {
                     "model": "example-model-b",
                     "result": { "status": 200 }
                  }
               ]
            }
         ]
      }
   }
}

If affinity lookup or persistence isn’t available, inference continues through normal routing and the response omits the complete session_affinity object. Don’t infer an affinity decision when the object or decision field is absent.

Interpret the results

The following output shows the response and routing metadata for an example request:

--- Chat Completions Response ---
Response:Pike Place Market is Seattle's most popular tourist destination.
Usage: 29 prompt + 278 completion = 307 total tokens

Routed to model: gpt-5-mini-2025-08-07
--- Model Selection Details ---
Routing mode: balanced
Routing decision 1 (latency: 19 ms)
   Attempt 1: grok-4-1-fast-reasoning - HTTP 404 (failed)
      Error: NotFound - The request failed with HTTP status code 404 (NotFound).
   Attempt 2: gpt-5-mini - HTTP 200 (selected)

• If model_selection_details is absent, the sample reports that no model selection details were returned. Don’t infer routing details that aren’t present.
• If routing_trace is empty, the sample reports that no routing trace was returned.
• An attempt can omit error. The extraction code prints an error only when the response includes one.
• Model names, HTTP statuses, attempt counts, and reported latency can vary by request and service version.
• response.model identifies the serving model for the demonstrated request.

Related content

• Use model router
• How model router works
• Monitor model deployments