unknown

A technical reflection on the hardware constraints shaping LLM performance, focusing on the fundamental tension between on-chip SRAM (fast, low capacity) and off-chip DRAM (high capacity, high latency). The author argues that optimally orchestrating memory and compute for LLM workloads—especially inference decode over long token contexts in agentic loops—is today's most intellectually challenging and financially rewarding problem. The post notes that neither HBM-first (NVIDIA) nor SRAM-first (Cerebras) architectures fully solve this simultaneously, and concludes with an endorsement of the MatX team following their funding raise.

With the coming tsunami of demand for tokens, there are significant opportunities to orchestrate the underlying memory+compute *just right* for LLMs.

The fundamental and non-obvious constraint is that due to the chip fabrication process, you get two completely distinct pools of memory (of different physical implementations too): 1) on-chip SRAM that is immediately next to the compute units that is incredibly fast but of very of low capacity, and 2) off-chip DRAM which has extremely high capacity, but the contents of which you can only suck through a long straw. On top of this, there are many details of the architecture (e.g. systolic arrays), numerics, etc.

The design of the optimal physical substrate and then the orchestration of memory+compute across the top volume workflows of LLMs (inference prefill/decode, training/finetuning, etc.) with the best throughput/latency/$ is probably today's most interesting intellectual puzzle with the highest rewards (\cite 4.6T of NVDA). All of it to get many tokens, fast and cheap. Arguably, the workflow that may matter the most (inference decode *and* over long token contexts in tight agentic loops) is the one hardest to achieve simultaneously by the ~both camps of what exists today (HBM-first NVIDIA adjacent and SRAM-first Cerebras adjacent). Anyway the MatX team is A++ grade so it's my pleasure to have a small involvement and congratulations on the raise!

<p>With the coming tsunami of demand for tokens, there are significant opportunities to orchestrate the underlying memory+compute *just right* for LLMs.

The fundamental and non-obvious constraint is that due to the chip fabrication process, you get two completely distinct pools of memory (of different physical implementations too): 1) on-chip SRAM that is immediately next to the compute units that is incredibly fast but of very of low capacity, and 2) off-chip DRAM which has extremely high capacity, but the contents of which you can only suck through a long straw. On top of this, there are many details of the architecture (e.g. systolic arrays), numerics, etc.

The design of the optimal physical substrate and then the orchestration of memory+compute across the top volume workflows of LLMs (inference prefill/decode, training/finetuning, etc.) with the best throughput/latency/$ is probably today's most interesting intellectual puzzle with the highest rewards (\cite 4.6T of NVDA). All of it to get many tokens, fast and cheap. Arguably, the workflow that may matter the most (inference decode *and* over long token contexts in tight agentic loops) is the one hardest to achieve simultaneously by the ~both camps of what exists today (HBM-first NVIDIA adjacent and SRAM-first Cerebras adjacent). Anyway the MatX team is A++ grade so it's my pleasure to have a small involvement and congratulations on the raise!</p>

Swyx offers insights into software engineering, developer productivity, and career growth, providing articles, talks, and resources to help developers level up their skills and advance their careers in tech. By exploring Swyx's curated content, developers can learn about coding strategies, learning methodologies, and networking techniques for becoming more effective and successful in their profession. Whether you're navigating your first job search, mastering new technologies, or aiming for leadership roles, Swyx offers resources to support your personal and professional growth in the software industry.

swyx

Andrej Karpathy highlights the growing demand for tokens and the significant opportunities in orchestrating memory and compute infrastructure to meet it.