When you’re using multiple LLM providers, tracking costs manually becomes impossible fast. We needed visibility into our AI spending and LLM performance. Here’s the monitoring engine we built for Rails.

The RubyLA blog offers insights, tutorials, and community updates for Ruby developers in Los Angeles and beyond. Developers can explore Ruby language features, web development frameworks, and best practices for building Ruby applications. Additionally, the blog covers Ruby meetups, events, and job opportunities in the Los Angeles area, fostering collaboration and networking within the local Ruby community.

RUBYLAND

RubyLLM::Monitoring is a Rails engine that tracks LLM API usage, costs, and performance metrics. It automatically instruments requests, calculates costs using built-in pricing data, and stores everything in your database. The tool provides a dashboard showing throughput, costs, response times, and error rates grouped by provider and model. It includes configurable alerts for cost overruns, performance degradation, and error spikes via email or Slack. The engine helps manage variable LLM costs, identify optimization opportunities, compare providers, and plan capacity without requiring external infrastructure.

Monitoring LLM usage in Rails with RubyLLM::Monitoring