In this video, we dive into Large Concept Models (LCMs), an innovative architecture from a recent Meta paper titled: "Large Concept Models: Language Modeling in a Sentence Representation Space".

Unlike Large Language Models (LLMs), LCMs operate on higher-level semantics, processing concepts instead of tokens, making them more akin to human reasoning, and possibly positioning them as a future rival to the current token-based LLM architecture.

Throughout the video, we explain what reasoning in the abstract concepts space means, and explore the Large Concept Models architecture, diving into few architecture options.

Specifically, two LCMs are based on Diffusion Models, so we provide a brief reminder about Diffusion Models and then explain the Diffusion-based LCMs.

Paper - https://arxiv.org/abs/2412.08821
Code - https://github.com/facebookresearch/large_concept_model
Written review - https://aipapersacademy.com/large-concept-models/

Large Concept Models resembles to Joint Embedding Predictive Architecture (JEPA) in the sense of information prediction in an abstract representation space. We've covered previous JEPA paper thoroughly here:

I-JEPA - https://aipapersacademy.com/i-jepa-a-human-like-computer-vision-model/

V-JEPA - https://aipapersacademy.com/v-jepa/

-----------------------------------------------------------------------------------------------
✉️ Join the newsletter - https://aipapersacademy.com/newsletter/

👍 Please like & subscribe if you enjoy this content

Become a patron - https://www.patreon.com/aipapersacademy

The video was edited using VideoScribe - https://tidd.ly/44TZEiX
-----------------------------------------------------------------------------------------------

Chapters:
0:00 Introduction
1:25 Concepts vs Tokens
3:08 LCM High-Level Architecture
5:31 Base-LCM
6:49 Diffusion-Based LCM
9:31 Results

AI Papers Academy

Meta's research paper introduces Large Concept Models (LCMs), a new AI architecture that operates on sentence-level concept embeddings rather than subword tokens. Unlike traditional LLMs, LCMs use a component called SONAR to encode sentences into language-agnostic concept embeddings, enabling hierarchical reasoning across 200+ languages and multiple modalities. Three architectural variants are explored: Base-LCM (next-concept prediction via MSE loss), One-Tower diffusion LCM, and Two-Tower diffusion LCM. The diffusion-based variants significantly outperform the base version on summarization quality (ROUGE-L) and coherence metrics, though they still trail slightly behind a small LLaMA baseline. The approach draws conceptual parallels to Meta's earlier JEPA work and Yann LeCun's vision for human-like AI.

Large Concept Models (LCMs) by Meta: The Era of AI After LLMs?