Agents drift. Models change, prompts get tweaked, edge cases accumulate, and the gap between what your agent does and what you need it to do widens without you noticing. Amy and Nitya walk through Microsoft Foundry's observability stack: tracing built on OpenTelemetry, built-in evaluators for quality, safety, and agentic metrics like intent resolution and task adherence, and red teaming where a second AI attacks your agent with adversarial prompts to find vulnerabilities before your users do.

The piece worth watching for is the observe skill demo. You point it at an agent with no eval dataset, no baselines, nothing. It generates the dataset, runs batch evaluations, optimizes the prompt, compares versions, and rolls back to the best one... all from a single prompt to a coding agent. The skill shows its reasoning at each step, which is where the real value is: it surfaces the failures you didn't know to look for.

Speaker info:
- https://x.com/NityaNarasimhan
- https://www.linkedin.com/in/nityan/
- https://x.com/AmyKateNicho
- https://www.linkedin.com/in/amykatenicho/

AI Engineer

A workshop session from Microsoft covering agent observability using Azure AI Foundry. The talk walks through building a multi-agent travel application (with flight, hotel, and car rental sub-agents), then progressively adding tracing via OpenTelemetry and App Insights, running built-in and custom evaluators for quality and safety, and monitoring agents in production. Key themes include trace-linked evaluations to shorten detection-to-diagnosis time, multi-agent workflow composition in Foundry, red teaming for safety, and a GitHub repo with Jupyter notebooks for hands-on practice.

Mind the Gap (In your Agent Observability) — Amy Boyd & Nitya Narasimhan, Microsoft