Discord scaled its search infrastructure to efficiently index trillions of messages using Elasticsearch and Kubernetes. They faced challenges like message drops, fault-intolerant bulk indexing, and performance issues due to large clusters. Solutions included deploying Elasticsearch on Kubernetes, adopting a multi-cluster architecture with dedicated nodes, and using PubSub for message queuing. Key achievements include improved indexing throughput, reduced query latency, and seamless cluster upgrades.
Table of contents
Deploy Elasticsearch on KubernetesMulti-cluster “cell” architecture to run smaller Elasticsearch clustersPubSub message queueBatch messages by cluster and index before bulk indexingSupporting new search use casesGive “Big Freaking Guilds” dedicated Elasticsearch clusters, with multiple shards1 Comment
Sort: