Archiving 104K files into tar.gz reduced I/O time by 43x and total processing time by 2.3x. The bottleneck was not disk speed, it was syscall overhead.

Lobsters is a community-driven platform for sharing and discussing links to articles, tutorials, and projects related to technology and programming. Readers can learn about a wide range of topics, from software development and system administration to cybersecurity and artificial intelligence. With user submissions, comments, and voting, Lobsters provides a platform for collaborative learning and knowledge sharing among technology enthusiasts.

Lobsters

Building a 2x faster ARM64 assembly lexer for Dart revealed that I/O, not lexing speed, was the bottleneck. Processing 104K files took 14.5 seconds for I/O versus 2.8 seconds for lexing due to syscall overhead from 300K+ open/read/close operations. Archiving files into tar.gz reduced I/O time by 43x (14.5s to 339ms) and total processing time by 2.3x, even with decompression overhead. This explains why package managers use archives: fewer syscalls, sequential reads, and compression outweigh decompression costs. Modern NVMe SSDs achieve peak performance only with sequential access to large files; thousands of small files cause syscall overhead to dominate. Alternatives like SQLite, io_uring, and ZIP archives offer different tradeoffs between random access and sequential performance.

I built a 2x faster lexer, then discovered I/O was the real bottleneck

The second surprise: the SSD is not the bottleneck #