Best of ELK — April 2025

1. 1
   Article

   Lobsters·1y

   How Discord Indexes Trillions of Messages

   Discord scaled its search infrastructure to efficiently index trillions of messages using Elasticsearch and Kubernetes. They faced challenges like message drops, fault-intolerant bulk indexing, and performance issues due to large clusters. Solutions included deploying Elasticsearch on Kubernetes, adopting a multi-cluster architecture with dedicated nodes, and using PubSub for message queuing. Key achievements include improved indexing throughput, reduced query latency, and seamless cluster upgrades.

   67

   1
2. 2
   Article

   Noted·1y

   Diskover: Search Files and Analyze Their Storage Usage

   Diskover helps users efficiently analyze file sizes and search data on their servers. Utilizing Elasticsearch, it offers rapid indexing and data retrieval. The Community Edition enables crawling and indexing data, sorting files by various attributes, and searching within one index. Installation is facilitated by Docker Compose. Enhanced features are behind a paywall. Basic functionalities include size and date searches, with more advanced analytics requiring paid versions.

   10
3. 3
   Article

   Noted·1y

   Index and Search Every File on Your Homelab Server using Sist2

   Sist2 is a high-performance file indexing and searching solution that uses C and VueJS technologies, along with Elasticsearch for efficient content retrieval. It supports features such as scanning and scheduling, multi-platform support, OCR integration, and more. This post provides a guide on how to set up Sist2 using Docker, index files, and customize settings to optimize your search experience on a homelab server.

   9