Recently, I’ve been working on a Rust project again.It deals with bioinformatics data, which can be quite large,so I got to play with profiling and optimizing the code.I’ve done some of this in past, but this time it was actually useful.In this post, I want to talk about a small optimizationin working with LZ4 compressionthat made a big difference in runtime performance.

Authored by Ryan Levick, Deterministic Space focuses on the convergence of Rust programming and game development. Readers can immerse themselves in practical tutorials covering game engine architecture, performance optimization strategies, and leveraging Rust's safety features in game development projects. Additionally, they can explore topics like graphics programming, physics simulation, and game engine design patterns in the context of Rust.

Pascal’s scribbles

A Rust developer working with bioinformatics data discovered a significant performance optimization when using LZ4 compression. By swapping the order of BufWriter and LZ4 Encoder in the chain (buffering writes to the encoder instead of to the file), they achieved a 1.83x speed improvement. The issue was that small MessagePack chunks (~250 bytes) were causing excessive LZ4 block operations, and buffering at the encoder level reduced these costly operations.

Rust BufWriter and LZ4 Compression