Discord scaled its search infrastructure to efficiently index trillions of messages using Elasticsearch and Kubernetes. They faced challenges like message drops, fault-intolerant bulk indexing, and performance issues due to large clusters. Solutions included deploying Elasticsearch on Kubernetes, adopting a multi-cluster
Table of contents
Deploy Elasticsearch on KubernetesMulti-cluster “cell” architecture to run smaller Elasticsearch clustersPubSub message queueBatch messages by cluster and index before bulk indexingSupporting new search use casesGive “Big Freaking Guilds” dedicated Elasticsearch clusters, with multiple shards1 Comment
Sort: