karpathy/minbpe: Minimal, clean, code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization.

This post provides minimal, clean code for the Byte Pair Encoding (BPE) algorithm commonly used in LLM tokenization. It includes two tokenizers and usage examples. The GPT4Tokenizer is a wrapper around the RegexTokenizer and reproduces the tokenization of GPT-4.