BigData Boutique blog·1yElasticsearch vs OpenSearch - 2025 update
An in-depth 2025 update comparing Elasticsearch and OpenSearch, touching on project status, performance, licensing, vector search capabilities, cost efficiency, and ecosystem solutions. OpenSearch has gained traction with open-source governance and additional vector search engines, while Elasticsearch maintains proprietary features and extensive integration solutions.
Fireship·1yIs Elon’s Grok 3 the new AI king?
Elon's new AI model, Grok 3, has overtaken benchmarks and reached the number one spot on the LM Marina leaderboard, outperforming other leading models in various tasks. Unique for its access to Twitter data, Grok 3 generates uncensored content and plans to release a subscription-based Super Grok soon. The model was trained on the world’s largest AI supercomputer in Memphis, Tennessee.
Lobsters·1yHow Discord Indexes Trillions of Messages
Discord scaled its search infrastructure to efficiently index trillions of messages using Elasticsearch and Kubernetes. They faced challenges like message drops, fault-intolerant bulk indexing, and performance issues due to large clusters. Solutions included deploying Elasticsearch on Kubernetes, adopting a multi-cluster architecture with dedicated nodes, and using PubSub for message queuing. Key achievements include improved indexing throughput, reduced query latency, and seamless cluster upgrades.
Zalando·23wThe Day Our Own Queries DoS’ed Us: Inside Zalando Search
Zalando's Search & Browse team experienced a self-inflicted DoS attack when an internal application sent resource-intensive faceting queries on high-cardinality fields to their Elasticsearch cluster. The incident caused search slowdowns and empty results for customers. The team mitigated by splitting markets across clusters, implementing load shedding, and eventually traced the issue to a maintenance workload bug generating 50x normal query volume. Key lessons included improving per-client monitoring with X-Opaque-Id headers, implementing query-level rate limiting, adding aggregation size controls, and recognizing that performance issues can stem from unexpected sources rather than common causes.
It's Foss·22wWatch Out Elasticsearch! Tiger Data's PostgreSQL BM25 Search Extension Goes Open Source
Tiger Data has open-sourced pg_textsearch, a PostgreSQL extension that enables BM25 relevance-ranked keyword searches directly within PostgreSQL. Previously available only on Tiger Cloud, the extension is now released under The PostgreSQL License on GitHub. It supports 29+ languages, works with partitioned tables, and uses a memtable architecture for efficient indexing. The extension allows developers to run modern search capabilities without setting up external systems like Elasticsearch, and can be combined with pgvector for hybrid keyword and semantic search within a single database.
Community Picks·1yPulse for Elasticsearch and OpenSearch - Search Cluster Monitoring & Maintenance, Redefined
Pulse enhances the management of Elasticsearch and OpenSearch clusters by offering detailed cluster health assessments, actionable maintenance insights, and automated root cause analysis to optimize cost and performance.
BigData Boutique blog·1yElasticsearch ELSER vs External Vector Embeddings
Elasticsearch's ELSER is a native embedding model for hybrid search, offering memory efficiency and simple integration. However, external models like OpenAI and Cohere provide richer semantic understanding, complex relationship handling, and multilingual support. The choice between ELSER and external embeddings depends on specific application needs, including database commitment, language constraints, and query complexity. ELSER is cost-effective and easy to set up, whereas external models might involve extra costs and complexity due to additional computational requirements.
Neon·49wComparing Native Postgres, ElasticSearch, and pg_search for Full-Text Search
PostgreSQL's native full-text search works for small datasets but struggles with performance and search quality at scale. ElasticSearch provides superior search capabilities but introduces operational complexity with dual data stores and sync challenges. pg_search offers a middle ground by embedding a modern search engine (powered by Tantivy) directly into PostgreSQL, delivering BM25 ranking, fuzzy matching, and real-time indexing without the operational overhead of managing separate systems.
BigData Boutique blog·1yOpenSearch & Elasticsearch Scroll and Deep Paging Methods Compared
Deep paging in Elasticsearch or OpenSearch can be resource-intensive. Various methods are available for deep paging, including the `from` and `size` parameters, Scroll API, PIT API, and `search_after`. Each method has its pros and cons, with `search_after` being the best for real-time pagination and the Scroll API suitable for bulk data retrieval. Choosing the right method based on the use case is crucial for maintaining cluster performance.
Laravel News·52wElasticLens: Eloquent-Powered Elasticsearch for Laravel
ElasticLens is a Laravel package that seamlessly integrates Elasticsearch with Eloquent models, providing advanced search capabilities while maintaining familiar Laravel syntax. The package introduces Index-Model architecture that automatically synchronizes with your database, supports complex queries including full-text search, geolocation filtering, and field boosting. It offers zero-configuration setup, custom field mapping, index migrations, and automatic model synchronization through observers.
Noted·1yDiskover: Search Files and Analyze Their Storage Usage
Diskover helps users efficiently analyze file sizes and search data on their servers. Utilizing Elasticsearch, it offers rapid indexing and data retrieval. The Community Edition enables crawling and indexing data, sorting files by various attributes, and searching within one index. Installation is facilitated by Docker Compose. Enhanced features are behind a paywall. Basic functionalities include size and date searches, with more advanced analytics requiring paid versions.
Noted·1yIndex and Search Every File on Your Homelab Server using Sist2
Sist2 is a high-performance file indexing and searching solution that uses C and VueJS technologies, along with Elasticsearch for efficient content retrieval. It supports features such as scanning and scheduling, multi-platform support, OCR integration, and more. This post provides a guide on how to set up Sist2 using Docker, index files, and customize settings to optimize your search experience on a homelab server.
