Tokenformer: The Next Generation of Transformers?

Tokenformer is a new neural architecture that rethinks how Transformer models scale. Instead of using fixed linear projections for token-parameter interactions, it replaces them with attention-based 'Pattention' blocks where model parameters are treated as tokens. This allows incremental model size increases by appending new parameter tokens to key/value matrices without retraining from scratch. Results show Tokenformer models scaled incrementally match or exceed standard Transformer performance while using only 10–20% of the training compute, significantly reducing cost and environmental impact.