Prompt Fidelity: Measuring How Much of Your Intent an AI Agent Actually Executes

Prompt Fidelity measures how much of a user's request an AI agent can verify through actual data versus infer through LLM reasoning. Using Spotify's playlist agent as a case study, the author demonstrates that agents often fulfill only 25% of constraints with verified data while silently inferring the rest. The metric uses information theory (bits = -log₂(p)) to weight constraints by selectivity, revealing a structural tradeoff: the most interesting prompts push past verified data capacity and require inference. Every agent has a computable fidelity ceiling based on its tool schema. The framework applies broadly to RAG systems, coding agents, and customer service bots, with practical recommendations including reporting fidelity scores, distinguishing grounded from inferred claims, and disclosing when exact requests cannot be fulfilled.