CODEFUSION by Microsoft: A Pre-trained Diffusion Model for Code Generation

CODEFUSION is a Microsoft research model that applies diffusion-based generation to code, allowing iterative refinement unlike traditional autoregressive code LLMs. It uses a pre-trained CodeT5 encoder, a denoiser operating in latent space, and a decoder to produce code tokens. Trained in two phases (unsupervised pre-training and supervised fine-tuning), CODEFUSION achieves competitive results against much larger models like GPT-3 and ChatGPT, outperforming them on top-3 and top-5 accuracy metrics across Python, Bash, and Excel formula benchmarks despite being significantly smaller.