How might LLMs store facts | Chapter 7, Deep Learning

Large language models, like those using transformer architectures, can store factual information within their numerous parameters. Recent research has identified that this knowledge is often embedded in specific parts of the network called multi-layer perceptrons (MLPs). The process involves vectors in high-dimensional space, where different directions encode different types of information. Understanding how these models operate, particularly the role of the MLPs and the influence of nearly perpendicular vectors, provides insight into how AI models can store and recall vast amounts of data efficiently.