> ## Documentation Index
> Fetch the complete documentation index at: https://hobbyist-e43fa225.mintlify.site/llms.txt
> Use this file to discover all available pages before exploring further.

# Monitor model router in Microsoft Foundry

> Learn how to inspect preview model router metadata for routing attempts, fallback, latency, and Chat Completions session affinity in Microsoft Foundry.

Observability helps you understand how model router handles requests, verify routing behavior, and investigate latency, errors, and fallback. Request-level signals complement aggregate metrics and logs, giving developers and operators context to evaluate application performance.

This article covers the per-request routing metadata preview for the Chat Completions API. The metadata identifies the serving model and describes routing attempts for an individual request. For aggregate metrics and logs, see [Monitor model deployments](/observability/monitor-models).

## Prerequisites

* Python 3.9 or later.
* The `openai>=1.75.0` and `python-dotenv` packages. Install them by running `pip install "openai>=1.75.0" python-dotenv`.
* A model router deployment that you can access through an Azure OpenAI endpoint.
* The endpoint and API key for your Azure OpenAI resource. The complete sample reads them from the `AZURE_OPENAI_ENDPOINT` and `AZURE_OPENAI_API_KEY` environment variables.
* Azure OpenAI API version `2024-10-21`.

## Enable per-request routing metadata

After your application reads the endpoint and API key into `endpoint` and `api_key`, create the client with the preview feature header:

```python theme={null}
"""
Foundry Model Router - Chat Completions Observability Example

This example demonstrates how to use Azure OpenAI's Chat Completions API
with a Foundry Model Router deployment and inspect the selected model,
routing attempts, latency, and status. Model Router automatically
selects the best underlying LLM for each prompt based on your routing mode
(Balanced, Quality, or Cost).

Prerequisites:
  - An Azure OpenAI resource with a "model-router" deployment
    - A .env file beside this script with AZURE_OPENAI_ENDPOINT,
        AZURE_OPENAI_API_KEY, and MODEL_DEPLOYMENT_NAME

Usage:
    pip install -r requirements.txt
    python model-router-chat-completions-observability.py
"""

import os
from pathlib import Path

from dotenv import load_dotenv
from openai import AzureOpenAI

# Load environment variables from .env beside this script
load_dotenv(Path(__file__).resolve().parent / ".env", override=True)

endpoint = os.environ["AZURE_OPENAI_ENDPOINT"]
api_key = os.environ["AZURE_OPENAI_API_KEY"]
deployment = os.environ["MODEL_DEPLOYMENT_NAME"]

# <response_observability_enable>
client = AzureOpenAI(
    azure_endpoint=endpoint,
    api_key=api_key,
    api_version="2024-10-21",
    default_headers={"Foundry-Features": "ModelRouterControls=V1Preview"},
)
# </response_observability_enable>

response = client.chat.completions.create(
    model=deployment,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": "In one sentence, name the most popular tourist destination in Seattle.",
        },
    ],
)

print("--- Chat Completions Response ---")
print(f"Response:{response.choices[0].message.content}")
print(
    f"Usage: {response.usage.prompt_tokens} prompt + {response.usage.completion_tokens} completion = {response.usage.total_tokens} total tokens"
)

# <response_observability_extract>
print(f"\nRouted to model: {response.model}")
print("--- Model Selection Details ---")
model_selection_details = getattr(response, "model_selection_details", None)
if not model_selection_details:
    print("No model selection details were returned.")
else:
    model_router_details = model_selection_details.get("model_router_details", {})
    print(f"Routing mode: {model_router_details.get('mode', 'unknown')}")

    routing_trace = model_router_details.get("routing_trace", [])
    if not routing_trace:
        print("No routing trace was returned.")

    for decision_number, routing_decision in enumerate(routing_trace, start=1):
        latency_ms = routing_decision.get("latency_ms")
        latency = f"{latency_ms} ms" if latency_ms is not None else "not reported"
        print(f"Routing decision {decision_number} (latency: {latency})")

        for attempt_number, attempt in enumerate(
            routing_decision.get("attempts", []), start=1
        ):
            result = attempt.get("result", {})
            status = result.get("status", "unknown")
            outcome = (
                "selected"
                if isinstance(status, int) and 200 <= status < 300
                else "failed"
            )
            print(
                f"  Attempt {attempt_number}: {attempt.get('model', 'unknown')} - HTTP {status} ({outcome})"
            )

            error = result.get("error")
            if error:
                print(
                    f"    Error: {error.get('code', 'unknown')} - {error.get('message', 'No message')}"
                )
    print("\n")
# </response_observability_extract>
```

The `Foundry-Features: ModelRouterControls=V1Preview` header requests per-request routing metadata. Because this feature is in preview, the metadata presence and response schema can vary by request and service version.

## Send a Chat Completions request

Use the model router deployment name to send a request. The response includes the completion and, when available, the per-request routing metadata:

```python theme={null}
response = client.chat.completions.create(
   model=deployment,
   messages=[
      {"role": "system", "content": "You are a helpful assistant."},
      {
         "role": "user",
         "content": "In one sentence, name the most popular tourist destination in Seattle.",
      },
   ],
)
```

## Understand routing metadata

The following `model_selection_details` fragment illustrates a request with two ordered model attempts:

```json theme={null}
{
   "model_selection_details": {
      "model_router_details": {
         "mode": "balanced",
         "routing_trace": [
            {
               "latency_ms": 19,
               "attempts": [
                  {
                     "model": "example-model-a",
                     "result": {
                        "status": 404,
                        "error": {
                           "code": "NotFound",
                           "message": "The request failed."
                        }
                     }
                  },
                  {
                     "model": "example-model-b",
                     "result": {
                        "status": 200
                     }
                  }
               ]
            }
         ]
      }
   }
}
```

* `mode` is the routing mode returned for the request.
* `routing_trace` contains the routing entries returned for the request.
* `latency_ms` is the latency reported for a routing-trace entry.
* `attempts` lists model attempts in order.
* Each attempt contains a model and an HTTP status in `result.status`.
* A failed attempt can include an optional `error` with a code and message.

## Extract routing and fallback information

After the Chat Completions request returns `response`, inspect the serving model and model selection details:

```python theme={null}
"""
Foundry Model Router - Chat Completions Observability Example

This example demonstrates how to use Azure OpenAI's Chat Completions API
with a Foundry Model Router deployment and inspect the selected model,
routing attempts, latency, and status. Model Router automatically
selects the best underlying LLM for each prompt based on your routing mode
(Balanced, Quality, or Cost).

Prerequisites:
  - An Azure OpenAI resource with a "model-router" deployment
    - A .env file beside this script with AZURE_OPENAI_ENDPOINT,
        AZURE_OPENAI_API_KEY, and MODEL_DEPLOYMENT_NAME

Usage:
    pip install -r requirements.txt
    python model-router-chat-completions-observability.py
"""

import os
from pathlib import Path

from dotenv import load_dotenv
from openai import AzureOpenAI

# Load environment variables from .env beside this script
load_dotenv(Path(__file__).resolve().parent / ".env", override=True)

endpoint = os.environ["AZURE_OPENAI_ENDPOINT"]
api_key = os.environ["AZURE_OPENAI_API_KEY"]
deployment = os.environ["MODEL_DEPLOYMENT_NAME"]

# <response_observability_enable>
client = AzureOpenAI(
    azure_endpoint=endpoint,
    api_key=api_key,
    api_version="2024-10-21",
    default_headers={"Foundry-Features": "ModelRouterControls=V1Preview"},
)
# </response_observability_enable>

response = client.chat.completions.create(
    model=deployment,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": "In one sentence, name the most popular tourist destination in Seattle.",
        },
    ],
)

print("--- Chat Completions Response ---")
print(f"Response:{response.choices[0].message.content}")
print(
    f"Usage: {response.usage.prompt_tokens} prompt + {response.usage.completion_tokens} completion = {response.usage.total_tokens} total tokens"
)

# <response_observability_extract>
print(f"\nRouted to model: {response.model}")
print("--- Model Selection Details ---")
model_selection_details = getattr(response, "model_selection_details", None)
if not model_selection_details:
    print("No model selection details were returned.")
else:
    model_router_details = model_selection_details.get("model_router_details", {})
    print(f"Routing mode: {model_router_details.get('mode', 'unknown')}")

    routing_trace = model_router_details.get("routing_trace", [])
    if not routing_trace:
        print("No routing trace was returned.")

    for decision_number, routing_decision in enumerate(routing_trace, start=1):
        latency_ms = routing_decision.get("latency_ms")
        latency = f"{latency_ms} ms" if latency_ms is not None else "not reported"
        print(f"Routing decision {decision_number} (latency: {latency})")

        for attempt_number, attempt in enumerate(
            routing_decision.get("attempts", []), start=1
        ):
            result = attempt.get("result", {})
            status = result.get("status", "unknown")
            outcome = (
                "selected"
                if isinstance(status, int) and 200 <= status < 300
                else "failed"
            )
            print(
                f"  Attempt {attempt_number}: {attempt.get('model', 'unknown')} - HTTP {status} ({outcome})"
            )

            error = result.get("error")
            if error:
                print(
                    f"    Error: {error.get('code', 'unknown')} - {error.get('message', 'No message')}"
                )
    print("\n")
# </response_observability_extract>
```

Ordered attempts can reveal automatic fallback for an individual request. In the example response, the failed attempt followed by a successful attempt is evidence of fallback for that request. Requests don't always include multiple attempts, so don't expect fallback on every request.

For complete application setup and runnable examples, see the [Foundry Model Router samples](https://github.com/microsoft-foundry/foundry-samples/tree/main/samples/python/foundry-models/model-router).

## Interpret session affinity metadata

When you enable the Chat Completions session affinity preview, `model_router_details` can include a `session_affinity` object. The following response fragment shows a request that retained its associated model:

```json theme={null}
{
   "model": "example-model-a",
   "model_selection_details": {
      "model_router_details": {
         "mode": "balanced",
         "session_affinity": {
            "mode": "sticky",
            "source": "session_id_payload",
            "decision": "retain"
         }
      }
   }
}
```

Interpret the fields as follows:

| Field      | Value                                       | Meaning                                                                                         |
| ---------- | ------------------------------------------- | ----------------------------------------------------------------------------------------------- |
| `mode`     | `sticky`                                    | Model router attempts the associated eligible model first.                                      |
| `source`   | `session_id_payload` or `session_id_header` | The request body or header supplied the session ID. The response doesn't return the identifier. |
| `decision` | `initialize`                                | No previous association was available, and the initially selected model served the response.    |
| `decision` | `retain`                                    | The associated model served the response.                                                       |
| `decision` | `switch`                                    | A different model served because of eligibility or fallback.                                    |

When `decision` is `switch`, inspect `routing_trace` and the top-level `model` field together. The following fragment shows an associated model that returned a retryable response before fallback selected another model:

```json theme={null}
{
   "model": "example-model-b",
   "model_selection_details": {
      "model_router_details": {
         "mode": "balanced",
         "session_affinity": {
            "mode": "sticky",
            "source": "session_id_payload",
            "decision": "switch"
         },
         "routing_trace": [
            {
               "latency_ms": 51,
               "attempts": [
                  {
                     "model": "example-model-a",
                     "result": { "status": 429 }
                  },
                  {
                     "model": "example-model-b",
                     "result": { "status": 200 }
                  }
               ]
            }
         ]
      }
   }
}
```

If affinity lookup or persistence isn't available, inference continues through normal routing and the response omits the complete `session_affinity` object. Don't infer an affinity decision when the object or `decision` field is absent.

## Interpret the results

The following output shows the response and routing metadata for an example request:

```text theme={null}
--- Chat Completions Response ---
Response:Pike Place Market is Seattle's most popular tourist destination.
Usage: 29 prompt + 278 completion = 307 total tokens

Routed to model: gpt-5-mini-2025-08-07
--- Model Selection Details ---
Routing mode: balanced
Routing decision 1 (latency: 19 ms)
   Attempt 1: grok-4-1-fast-reasoning - HTTP 404 (failed)
      Error: NotFound - The request failed with HTTP status code 404 (NotFound).
   Attempt 2: gpt-5-mini - HTTP 200 (selected)
```

* If `model_selection_details` is absent, the sample reports that no model selection details were returned. Don't infer routing details that aren't present.
* If `routing_trace` is empty, the sample reports that no routing trace was returned.
* An attempt can omit `error`. The extraction code prints an error only when the response includes one.
* Model names, HTTP statuses, attempt counts, and reported latency can vary by request and service version.
* `response.model` identifies the serving model for the demonstrated request.

## Related content

* [Use model router](/models/model-router)
* [How model router works](/models/model-router-how-it-works)
* [Monitor model deployments](/observability/monitor-models)