So the ask is to do 3 Million Predictions per second with as little resources as possible. Thankfully its one of the simpler model of Recommendation systems, Multi Armed Bandit(MAB). Multi Armed bandit usually involves sampling from distribution like Beta Distribution. That’s where the most time is spent. If we can concurrently do as many sampling as we can, we’ll use the resources well. Maximizing Resource utilization is the key to reducing overall resources needed for the model.

Community Picks is a section on daily.dev where our community members share the most interesting and valuable content they've discovered online. From insightful articles to handy tools, every post is a gem curated by our dedicated coomunity. To contribute to Community Picks, you need to have at least 250 reputation points, ensuring that only active and trusted members can share their finds.

Community Picks

The current Python service with single node can do 192 RPS, about 400 pairs each. Only about 20% average CPU utilization. The limiting factor now was the language, the serving framework and the network call to feature store. I chose Fiber framework for REST API, it seemed most welcoming, good documentation, expressjs like API. Took less than an hour.

Go faster with Go: Golang for ML Serving