So long, and thanks for all the fish: how to escape the Linux networking stack

Cloudflare engineers developed a custom service called SLATFATF ("fish") to handle IP packet forwarding using their soft-unicast addressing system, which shares IP addresses across machines. The team encountered fundamental conflicts between Linux's socket subsystem and Netfilter's conntrack module when attempting to use both packet rewriting and bound sockets simultaneously. After exploring solutions including Netlink interfaces, TCP_REPAIR, and TCP Fast Open with cookieless connections, they discovered that Linux's "early demux" optimization bypassed custom routing rules. Despite successfully implementing workarounds, they ultimately chose to terminate TCP connections rather than forward raw IP packets due to better observability and minimal performance impact.