S3 server access logs at scale Nurdan Almazbekov, Infrastructure Security Sep 26, 2025 Introduction Yelp heavily relies on Amazon S3 (Simple Storage Service) to store a wide variety of data,...

Yelp's platform is a central hub for consumers and businesses, offering insights into local businesses, user reviews, and recommendations. Through articles, guides, and user-generated content, Yelp offers insights into discovering restaurants, shops, and services in various locations. Readers can learn about reading and writing reviews, finding popular businesses, and exploring local communities to make informed decisions about where to eat, shop, or visit.

Yelp Engineering

Yelp's infrastructure team built a scalable system to process terabytes of S3 server access logs daily, converting raw logs to Parquet format to reduce storage by 85% and object count by 99.99%. The solution uses AWS Athena for querying, partition projection for efficient data access, and S3 batch operations for lifecycle management. The system enables debugging access issues, cost attribution, incident response, and data retention decisions by identifying unused objects through joining S3 inventory with access logs. Key challenges included handling malformed logs with user-controlled fields, managing Athena query limits, and ensuring idempotent insertions across distributed systems.

S3 server access logs at scale

Joining S3 inventory with S3 server access logs