[2404.07839] RecurrentGemma: Moving Past Transformers for Efficient Open Language Models

RecurrentGemma is an open language model that uses Google's Griffin architecture to achieve excellent performance on language. It has a fixed-sized state, reducing memory usage, and achieves comparable performance to Gemma-2B despite being trained on fewer tokens.