A deep dive into Philox and counter-based RNGs

Code Confessions is a platform or blog where developers share their experiences, challenges, and insights from their journey in software development. Through personal anecdotes, lessons learned, and reflections on coding practices, Code Confessions offers a candid and relatable perspective on the joys and struggles of being a programmer. Readers can find inspiration, empathy, and  lessons from the stories shared by fellow developers.

Confessions of a Code Addict

Philox is a counter-based parallel random number generator that enables PyTorch to generate millions of random numbers simultaneously on GPUs. Unlike traditional sequential PRNGs that depend on previous state, Philox uses a cryptographic-like construction to transform a 128-bit counter into pseudorandom output through 10 rounds of multiplication, XOR operations, and permutation. The counter space is partitioned into subsequences (upper 64 bits) and offsets (lower 64 bits), allowing thousands of GPU threads to independently generate random numbers without coordination. PyTorch's implementation maintains only 44 bytes of state per generator instance and efficiently batches 4 numbers at a time, making it ideal for deep learning operations like weight initialization, dropout, and stochastic gradient descent.

How PyTorch Generates Random Numbers in Parallel on the GPU

Parallelizing Philox: Subsequences and Offsets