How PyTorch Generates Random Numbers in Parallel on the GPU

Philox is a counter-based parallel random number generator that enables PyTorch to generate millions of random numbers simultaneously on GPUs. Unlike traditional sequential PRNGs that depend on previous state, Philox uses a cryptographic-like construction to transform a 128-bit counter into pseudorandom output through 10 rounds of multiplication, XOR operations, and permutation. The counter space is partitioned into subsequences (upper 64 bits) and offsets (lower 64 bits), allowing thousands of GPU threads to independently generate random numbers without coordination. PyTorch's implementation maintains only 44 bytes of state per generator instance and efficiently batches 4 numbers at a time, making it ideal for deep learning operations like weight initialization, dropout, and stochastic gradient descent.