A practical guide to implementing end-to-end LLM observability in FastAPI using the OpenTelemetry Python SDK without relying on vendor-specific agents. Covers designing a span hierarchy for a RAG pipeline with spans for retrieval, LLM invocation, and post-processing. Explains how to capture semantic attributes including prompt metadata (hashed for privacy), token usage, estimated cost, and model configuration. Also covers evaluation hooks for attaching quality signals to traces, exporting to backends like Jaeger, Grafana Tempo, or Arize Phoenix, and best practices around sampling, privacy, and anti-patterns.
Table of contents
Table of ContentsIntroductionPrerequisites and Technical ContextWhy LLM Observability Is Fundamentally DifferentReference Architecture: A Traceable RAG RequestReference Architecture ExplainedWhy This Design Is Better Than Simpler AlternativesLLM Models That Work Best for This ArchitectureOpenTelemetry Primer (LLM-Relevant Concepts Only)Designing LLM-Aware SpansFastAPI Example: End-to-End LLM Spans (Complete and Explained)Semantic Attributes: Best Practices for LLM ObservabilityEvaluation Hooks Inside TracesExporting and Visualizing Traces (Where This Fits with Vendor Tooling)Operational Patterns and Anti-PatternsExtending the SystemConclusionSort: