Best of NoSQL — 2024

1
Article
gitconnected·2y
System Design Interview: Design WhatsApp
This post outlines how to design a high-level system architecture for a messaging app similar to WhatsApp. Key points include handling 10 billion messages daily with plans to double, using RESTful APIs for compatibility, employing WebSockets for real-time communication, and using message queues like Kafka or RabbitMQ for efficient message distribution. The databases considered are NoSQL options like Cassandra and DynamoDB, emphasizing sharding and partitioning for scalability. End-to-end encryption, handling peak traffic, and future enhancements like group chats and media handling are also discussed.
1.3K
37
2
Article
DEV·2y
When to use SQL vs NoSQL
Choosing between SQL and NoSQL databases can be daunting. SQL excels in data consistency and complex querying, while NoSQL offers scalability and flexibility. Common misconceptions include inflexibility of SQL databases and the lack of transactional support in NoSQL.
167
2
3
Article
DEV·2y
Guide To Choose A Database For Your Next Design
Guide to choosing the right database for your next design. Discusses the differences between relational and non-relational databases, their features and drawbacks, and provides guidance on choosing the appropriate database for specific requirements.
132
3
4
Article
Substack·2y
A Primer on Databases
Databases have been fundamental to software development for decades. The post discusses their history, from the invention of SQL databases to the rise of unstructured and cloud-based databases. It highlights the current database landscape, including the significance of transaction processing and analytics. The piece also touches on emerging technologies like vector databases, which are crucial for AI development. The author emphasizes that while the core technology of databases is not extremely complex, distribution and platformization will be key factors for future success in the database market.
125
1
5
Article
swizec.com·2y
Why SQL is Forever
SQL and relational databases remain fundamental for transactional data, despite the advances and popularity of NoSQL technologies over the past decades. Many NoSQL systems have either been removed, adapted to include SQL/natively support transactions, or are mainly used for caching and analytics. This demonstrates the enduring flexibility and utility of SQL, including new features like JSON support and vector databases, which relational databases have successfully integrated while maintaining ACID properties.
121
7
6
Article
Hacker News·2y
Just use Postgres
When developing a new application requiring persistent data storage, Postgres should be the default choice, especially for web applications. SQLite is suitable for desktop or mobile apps but not ideal for websites. NoSQL databases like DynamoDB, MongoDB, Cassandra, and others offer performance benefits if the access patterns are well-defined beforehand but are less flexible when handling evolving access needs. Redis is efficient for caching but can be used as a primary database with some trade-offs. Datomic and XTDB present innovative approaches with specific benefits and challenges. While advanced databases like Kafka and ElasticSearch excel in niche tasks, Postgres often suffices for general needs, potentially complemented by simpler built-in search functions.
62
13
7
Video
codeHeim·1y
#65 Golang - Build REST APIs with Golang and MongoDB
Learn how to build REST APIs using Golang and MongoDB. The guide covers setting up the MongoDB Go driver, structuring the project using MVC, connecting to the database, and implementing CRUD operations for movie records. The tutorial includes examples of inserting, updating, deleting, and querying data in MongoDB.
40
1
8
Article
Metadata·2y
Designing Data Intensive Applications Book
The post discusses the book 'Designing Data Intensive Applications' as part of a book club, focusing on the first two chapters which cover reliability, scalability, and maintainability of applications, as well as different data models and query languages. It explains the historical context and comparisons between relational and document databases, including the benefits and drawbacks of each model, and the impact of these choices on application design.
36
1
9
Article
Software Testing Magazine·2y
Open Source Database Testing Tools
Database testing has fewer open-source tools compared to programming languages. This post lists various tools for unit, load, and security testing on relational and NoSQL databases. With the growing prominence of NoSQL and open-source relational databases, more tools are emerging. Key tools mentioned include Database Benchmark, Database Rider, DBChaos, DBKover, dbstress, DbUnit, DB Test Driven, GODBT, HammerDB, JdbcSlim, JdbcRunner, and many others.
32
10
Article
ByteByteGo·2y
A Deep Dive into Amazon DynamoDB Architecture
Learn about the architecture and operational requirements of Amazon DynamoDB, a highly scalable and available NoSQL cloud database service.
29
11
Article
Firebase Developers·2y
Persisting LLM chat history to Firestore
Learn how to persist chat history for LLM powered applications using Firestore. The post walks through maintaining in-memory chat history with LangChain's RunnableWithMessageHistory and transitioning to persistent storage using Firestore. Users will understand how to enhance their chat applications by saving session history, making conversations more context-aware and meaningful.
25
12
Article
AWS in Plain English·2y
Understanding Database Indexing
Database indexing is used to facilitate faster data retrieval by creating a separate data structure that contains pointers or references to the actual data in a table. Indexing helps optimize disk operations and improve search performance in databases. There are different types of indexing, such as B-tree indexing and LSM tree indexing. The sequence of columns in a composite index is important for efficient searching. Certain considerations should be made when using indexes in queries.
25
1
13
Article
Medium·2y
Embracing Simplicity and Composability in Data Engineering
The post highlights the importance of simplicity and composability in data engineering, drawing lessons from decades of industry experience. It discusses the Unix philosophy of treating data as files, the evolution of databases and NoSQL, and the complexity introduced by new ecosystems like Hadoop and Kubernetes. The post also critiques the over-complication of agile methodologies and stresses the necessity of adhering to fundamental principles to maintain flexibility and long-term value in software systems.
24
14
Article
Community Picks·2y
🗄️🚀 SQL vs NoSQL: Viability and Comparison
SQL databases like PostgreSQL offer rigid, table-based schemas and follow ACID principles, making them ideal for structured data and applications requiring consistent transactions. NoSQL databases like MongoDB and ElasticSearch provide flexible schemas, are horizontally scalable, and are suited for unstructured data, web apps, IoT, and real-time text searching. PostgreSQL is trusted in corporate sectors for reliability, MongoDB is favored by startups for scalability, and ElasticSearch is essential for real-time observability and log analysis.
23
15
Article
Trendyol Tech·1y
Optimizing Elasticsearch with Custom Routing and Handling Routing Value Changes
Optimizing the integration between Couchbase and Elasticsearch, this piece discusses implementing custom routing to improve search performance. It highlights the benefits of routing queries based on itemNumbers, which reduces query scope, speeds up search operations, and efficiently uses resources. The post explains how to handle changes to itemNumbers, ensuring documents are correctly routed. Key insights from load testing reveal significant performance enhancements with custom routing, achieving faster query response times and higher query volumes.
22
16
Article
Hacker News·2y
ATProto for distributed systems engineers
AT Protocol aims to revolutionize social networking by decentralizing backend systems, allowing for shared state and user accounts across applications. It leverages eventual consistency and stream processing architectures to achieve high scalability. Key components include NoSQL data repositories, cryptographically signed records, and public APIs for external service integration. The protocol combines high-scale systems practices with peer-to-peer technology to create a highly scalable, open network.
21
17
Article
ByteByteGo·2y
Counting Billions of Content Usage at Canva
Canva faced significant challenges in accurately counting content usage for their Creators Program due to the rapidly growing data volume which was originally handled by a MySQL-based system. The MySQL design, while simple, created scalability issues. Canva explored transitioning to DynamoDB but found it wouldn't solve their processing problems. Ultimately, they moved to an OLAP-based solution with Snowflake, significantly improving latency and handling billions of records efficiently. The new architecture provided numerous benefits including reduced latency, simplified incident handling, and decreased data storage needs, though it also introduced some new complexities in data transformation and infrastructure management.
20
18
Article
Collections·2y
Appwrite Database Tutorial: Introduction, Setup, and Integration with Next.js
Learn how to set up and integrate Appwrite, an open-source backend-as-a-service platform, with a Next.js application. This guide provides detailed steps for setting up a project, installing necessary dependencies, creating a client connection, and saving data to an Appwrite database.
17
19
Article
P99 Conf·2y
A Tale of Database Performance Pains
The post tells the story of Patrick's challenges with database performance after starting his own online business. He learned critical lessons about selecting the right database, managing provisioned throughput, setting up observability tools, ensuring regular backups, and understanding concurrency limitations. His experiences highlight the importance of thoroughly evaluating database options, preparing for unexpected traffic spikes, and scheduling maintenance operations during low-demand periods.
17
1
20
Article
Dev Squad·2y
MongoDB Essentials: What is MongoDB, Collections, and Documents?
MongoDB is a NoSQL database that uses a flexible, document-oriented approach to store data instead of the traditional table-based structure. A collection in MongoDB groups together multiple documents and acts like a folder in a file system. Documents, the basic unit of data in MongoDB, are similar to records in a spreadsheet or JSON objects, containing various fields and values representing the data.
16
21
Article
The New Stack·1y
Why Now Is the Time to Migrate From Redis to Valkey
Redis has shifted away from open-source licensing, prompting many to consider migrating to Valkey, a highly compatible fork. The switch is recommended now due to low migration costs and future uncertainties with Redis's licensing. Valkey is supported by major players like AWS and Google and offers the same open-source benefits, potentially saving costs and avoiding lock-in issues associated with Redis.
14
1
22
Article
Tokio·2y
Announcing Toasty, an async ORM for Rust
Toasty is an asynchronous ORM for Rust, prioritizing ease of use for SQL and NoSQL databases like DynamoDB and Cassandra. It generates Rust code from a schema file, allowing efficient data model management. Toasty aims to balance productivity with performance by minimizing the complexity of traits and lifetimes in its generated code. While still in the early stages and not yet suitable for real-world use, Toasty hopes to bridge the gap in Rust's ecosystem for higher-level web applications.
13
23
Article
DigitalOcean Community·2y
How to Set Up a Fast API Application with a NoSQL Database
Learn how to set up a FastAPI application with a NoSQL database, specifically MongoDB, on an Ubuntu server. FastAPI is known for its speed and support for asynchronous programming, making it ideal for scalable modern applications. The guide covers prerequisites, Python and pip installation, creating a virtual environment, installing necessary libraries like FastAPI, Uvicorn, and Motor, and setting up MongoDB. It includes examples of creating API routes for AI prediction and MongoDB interactions, and how to run the application using Uvicorn. The post also touches on containerizing the application using Docker Compose.
13
24
Article
Tinybird·2y
Top Use Cases for DynamoDB in 2024
Amazon DynamoDB is a high-performance, scalable NoSQL key-value database integrated within the AWS ecosystem. It offers benefits such as serverless architecture, schemaless flexibility, low-latency writes and reads, load balancing, and high availability. Common use cases include gaming, content streaming, banking, mobile and web apps, and IoT. Though efficient, it has limitations in analytical processing, complex data modeling, and cloud vendor lock-in. Alternatives like MongoDB, Apache Cassandra, and Tinybird can help in scenarios where DynamoDB isn't suitable.
12
25
Video
YouTube·2y
Fundamentals Of Data Engineering Masterclass
This Data Engineering masterclass covers the fundamentals of Data Engineering, including the life cycle, data generation, storage, database management, data modeling, and the distinction between SQL and NoSQL. It delves into data processing systems like OLTP and OLAP, ETL processes, and building data architecture from scratch. The session also explores data warehousing, dimensional modeling, data marts, data lakes, big data, cloud services (AWS, GCP, Azure), and key tools for data engineering such as Python, SQL, Apache Spark, Databricks, Apache Airflow, and Apache Kafka. Real-world architecture case studies on AWS and GCP are discussed as well.
12

See all NoSQL archives