We gave our AI agent a SQL interface to billions of CI log lines in ClickHouse. How we ingest, store, and query 1.5 billion log lines a week.

Hacker News is a community-driven platform for sharing and discussing technology news, startups, and programming-related topics. Through user submissions and comments, Hacker News offers insights into emerging technology trends, industry developments, and entrepreneurial ventures. Readers can participate in discussions, share their insights, and stay informed about the latest advancements in technology and innovation.

Hacker News

Mendral built an AI agent that investigates CI failures by writing its own SQL queries against a ClickHouse database storing 1.5 billion log lines per week. The system denormalizes 48 metadata columns onto every log row, achieving 35:1 compression (21 bytes per line on disk). The agent runs multi-step investigations averaging 4.4 queries per session, with median query latency of 20ms for job metadata and 110ms for raw log scans. Key engineering decisions include bloom filter skip indexes for full-text search, materialized views for pre-aggregated failure rates, and a throttled GitHub API ingestion pipeline using durable execution (Inngest) to stay under rate limits while keeping data fresh within 5 minutes P95.

LLMs Are Good at SQL. We Gave Ours Terabytes of CI Logs.