Perfectly Normal
This title could be clearer and more informative.Try out Clickbait Shieldfor free (5 uses left this month).
A comprehensive exploration of KL-divergence through six and a half distinct intuitions: expected surprise, hypothesis testing, maximum likelihood estimation, suboptimal coding, gambling games (beating the house and gaming the lottery), and Bregman divergence. Each framing illuminates a different facet of why KL-divergence measures how much a model differs from the true distribution in the world where the true distribution holds — which also explains its asymmetry. The post argues that the 'expected surprise' framing is the most intuitive, while the suboptimal coding framing is the most mathematically elegant.
Table of contents
Summary1. Expected Surprise2. Hypothesis Testing3. MLEs4. Suboptimal Coding5A. Gambling Games - Beating the House5B. Gambling Games - Gaming the Lottery6. Bregman DivergenceFinal ThoughtsSort: