Optimizing llama.cpp AI Inference with CUDA Graphs

The open-source framework llama.cpp, known for efficient AI inference on Meta Llama models, has been optimized using NVIDIA CUDA Graphs to reduce GPU-side launch overheads. This enhancement allows scheduling multiple GPU activities as a single computational graph, improving performance significantly. Initial results show speedups up to 1.2x on NVIDIA H100 GPUs for smaller models. Ongoing work aims to further reduce CPU overheads, potentially offering up to 10% additional improvement.