Coconut by Meta AI - LLM Reasoning With Chain of Continuous Thought

Meta AI's COCONUT (Chain of Continuous Thought) method enables LLMs to reason in a continuous latent space rather than being constrained to word-based reasoning. Instead of generating intermediate reasoning steps as text tokens, the model feeds its last hidden state back as input embeddings during a 'latent thought mode', alternating with standard language generation. Training uses a curriculum approach that progressively replaces language reasoning steps with continuous thought tokens. Benchmarks show COCONUT outperforms standard Chain of Thought on planning-intensive tasks (ProsQA) while being more token-efficient, and exhibits a BFS-like reasoning pattern that helps explore multiple solution branches before committing to an answer. It underperforms Chain of Thought on pure math (GSM8K).