ADOPT is a new adaptive gradient method developed by researchers from The University of Tokyo to overcome the limitations of the Adam optimizer. Unlike Adam, which struggles without fine-tuning the hyperparameter β2 or assuming uniformly bounded gradient noise, ADOPT achieves optimal convergence rates without such requirements. It excludes the current gradient from the second moment estimate and adjusts momentum and normalization updates, demonstrating superior performance across diverse tasks like image classification, generative modeling, language processing, and reinforcement learning.
Sort: