Low-Rank Adaptation (LoRA) can be considered a major breakthrough towards the ability to train large language models for specific tasks efficiently. It is widely used today in many applications and…

Towards Data Science is a community-powered publication that showcases work in data science, machine learning and artificial intelligence. Every day newcomers, seasoned researchers and industry practitioners publish tutorials, research notes and real-world case studies that help the field move forward.

Towards Data Science

This post provides an overview of various variants of LoRA, a technique used to train large language models efficiently. It explains the basic concept of LoRA and introduces variants such as LoRA+, VeRA, LoRA-FA, LoRA-drop, AdaLoRA, DoRA, and Delta-LoRA. Each variant is described briefly along with its unique features and improvements. The post concludes by mentioning that the field of research on LoRA and related methods is continuously evolving with new breakthroughs expected in the future.

An Overview of the LoRA Family