The CSV magician. Contribute to medialab/xan development by creating an account on GitHub.

Lobsters is a community-driven platform for sharing and discussing links to articles, tutorials, and projects related to technology and programming. Readers can learn about a wide range of topics, from software development and system administration to cybersecurity and artificial intelligence. With user submissions, comments, and voting, Lobsters provides a platform for collaborative learning and knowledge sharing among technology enthusiasts.

Lobsters

A deep technical exploration of how to safely "jump" to arbitrary byte offsets within CSV files without knowing whether you've landed inside a quoted cell or not. The technique works by sampling the first 128 rows to build a statistical profile (column count, max row size, per-column average byte sizes), then testing two hypotheses at each jump point (quoted vs. unquoted) by parsing a fixed window of bytes under each assumption and using cosine similarity as a tie-breaker. This enables the xan CLI tool to perform single-pass parallel map-reduce over large CSV files, O(k) approximate sampling, and O(log n) binary search on sorted CSV data. Caveats include incompatibility with streams, compressed files, inconsistent column counts, and very small files.

xan/docs/blog/csv_base_jumping.md at master · medialab/xan