Orca 2 by Microsoft: Teaching Small Language Models How to Reason

Microsoft's Orca 2 research paper introduces two key improvements over Orca 1 for training small language models. First, it maps specific solution strategies (step-by-step, recall-then-generate, direct-answer, etc.) to appropriate task types, ensuring training data is more accurate. Second, it introduces 'cautious reasoning' via a technique called Prompt Erasing, where system instructions are replaced with a generic prompt during training, teaching the model to autonomously select the right reasoning strategy. Built on LLaMA-2 (7B and 13B), Orca 2 outperforms or matches much larger models like LLaMA-2-Chat 70B on most benchmarks, with model weights publicly released.