Hacker News is a community-driven platform for sharing and discussing technology news, startups, and programming-related topics. Through user submissions and comments, Hacker News offers insights into emerging technology trends, industry developments, and entrepreneurial ventures. Readers can participate in discussions, share their insights, and stay informed about the latest advancements in technology and innovation.

Hacker News

The cost of self-hosting the Llama-3 8B-Instruct model is about $17 per 1M tokens when using EKS, compared to $1 per 1M tokens with ChatGPT. Self-hosting the hardware can reduce the cost to less than $0.01 per 1M tokens, but it takes about 5.5 years to break even.

Cost Of Self Hosting Llama-3 8B-Instruct

Sure, but you still get better results from chatGPT or Claude with zero maintenance so the price is fair.
Unless you mod a gaming GPU with larger/more RAM chips and run bigger or multiple models, it ain’t worth self-hosting IMO.

Could’nt you have used a quantized model and it would have been much more faster. You can run a Llama3.1-8B quantized on 12GB ram easily and if possible use it with vLLM and you can pass ~25 prompts to the model in less than a minute. What are your thoughts on this?

This article was roasted on Hackernews for being too conservative on the costs for self-hosting. Even the AWS setup in the article was considered more expensive than it needed to be for running Llama 3 - 8B locally. Comment section quote: “LLAMA 8B on [Amazon] Bedrock is $0.40 per 1M input tokens and $0.60 per 1M output tokens.”
Check out the comments section on HK for significantly more affordable self-hosting solutions with LLMs: <a href="https://news.ycombinator.com/item?id=40681784" target="_blank" rel="noopener nofollow">https://news.ycombinator.com/item?id=40681784</a>

Wonderful comparison just to get an idea of how the entire hosting works :)

This was really nice comparison of price.