A Subtle Trick to Optimize Neural Network Training

Discover a subtle optimization trick for neural network training that involves normalizing data after transferring it to the GPU. This simple rearrangement can significantly reduce data transfer time, especially in tasks like image classification where pixel values are initially 8-bit integers. While the technique may not apply to all use cases, such as NLP, it can offer noticeable performance gains in applicable scenarios.