Nobody ever got fired for using a struct
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A performance investigation at Feldera revealed that SQL tables with hundreds of nullable columns caused significant serialization overhead when mapped to Rust structs. The root issue: rkyv's ArchivedString loses Rust's niche optimization, forcing an explicit Option discriminant, and with 700+ optional fields the archived struct ballooned to 2x the in-memory size. The fix introduces a bitmap-based serialization layout that strips Option wrappers from the archived format and records nullability in a compact bitfield. A further sparse layout stores only present values with relative pointers, dramatically reducing disk I/O for wide, mostly-null rows. The customer's throughput was restored after serialized row size dropped by roughly 2x.
Table of contents
How it startedHow we use structsHow structs look like (in memory)How structs look like (on disk)Issue 1: Option<ArchivedString>Issue 2: Too Many Optionsrkyv to the rescue!Serializing and Deserializing StructsSparse rowsWhat changedThe lessonsSort: