Many products at Cloudflare aren’t possible without pushing the limits of network hardware and software to deliver improved performance, increased efficiency, or novel capabilities such as soft-unicast, our method for sharing IP subnets across data centers. Happily, most people do not need to know the intricacies of how your operating system handles network and Internet access in general. Yes, even most people within Cloudflare. But sometimes we try to push well beyond the design intentions of Linux’s networking stack. This is a story about one of those attempts.

Cloudflare's platform is a leading provider of internet security and performance solutions, offering insights into web security, content delivery, and DNS management. Through documentation, blog posts, and webinars, Cloudflare provides insights into protecting websites and applications from cyber threats and improving performance. Developers and IT professionals can learn about CDN (Content Delivery Network), DDoS mitigation, and firewall configurations to secure and accelerate web traffic.

Cloudflare

Cloudflare engineers developed a custom service called SLATFATF ("fish") to handle IP packet forwarding using their soft-unicast addressing system, which shares IP addresses across machines. The team encountered fundamental conflicts between Linux's socket subsystem and Netfilter's conntrack module when attempting to use both packet rewriting and bound sockets simultaneously. After exploring solutions including Netlink interfaces, TCP_REPAIR, and TCP Fast Open with cookieless connections, they discovered that Linux's "early demux" optimization bypassed custom routing rules. Despite successfully implementing workarounds, they ultimately chose to terminate TCP connections rather than forward raw IP packets due to better observability and minimal performance impact.

So long, and thanks for all the fish: how to escape the Linux networking stack

The workaround at the end of the universe

<p>instead of the original, you remind me of factorio</p>
