Performance testing in AI projects: what really matters

AI systems fail differently than traditional APIs, and standard performance testing misses the most critical failure modes. This framework covers what actually matters: token-based cost tracking per session (not just per request), time-to-first-token vs. total latency, p95/p99 percentiles over averages, RAG retrieval performance, agent workflow step counts, and provider rate limit behavior. Key bottlenecks include prompt growth across conversation turns, over-reserved token limits, sequential model calls, and missing caching. The recommended strategy maps real user flows first, sets explicit thresholds before testing, tests components in isolation, and combines load testing with quality evaluation using tools like LLM-as-judge or RAGAS. Continuous testing integrated into CI/CD and production observability via platforms like Langfuse and OpenTelemetry are presented as essential for catching degradation after launch.