Best of Storage — 2024
- 1
- 2
Community Picks·2y
The State of Data Engineering 2024
The 2024 State of Data Engineering report discusses the influence of GenAI on software infrastructure, the expansion of product offerings due to the economic downturn, and the impact of open table formats and their catalogs in the data lake industry. It also highlights the importance of data version control and observability in AI/ML systems.
- 3
- 4
- 5
Substack·2yS3 Is Showing Its Age
S3 is falling behind its competitors in terms of features such as compare-and-swap operation, multi-region buckets, and object appends. These missing features are becoming a bigger problem as new infrastructure relies on object storage as the primary persistence layer. The decision to abandon or embrace S3 depends on the system's use cases and design goals. Preconditions on S3 and a unified S3Express One Zone API would make this decision easier.
- 6
gitconnected·2y
Object Storage (BLOBs) Explained for System Design
Object storage, also known as Binary Large Object (BLOB) storage, is designed for handling massive amounts of unstructured data like images, videos, and virtual machines. Platforms such as Amazon S3, Google Cloud Storage, and Azure Blob Storage offer scalable and fast retrieval of large files compared to traditional file systems and databases. These storage solutions use unique identifiers and metadata for each object and often provide a RESTful API for data manipulation. Object storage is ideal for cloud environments due to its scalability, speed, and additional security features like signed URLs and signed cookies.
- 7
Hacker News·2y
The Raspberry Pi 5 is no match for a tini-mini-micro PC
Mini PCs are a compelling alternative to Raspberry Pi for home servers, offering better performance, built-in SSD/NVME support, and the ability to go beyond 8 GB of RAM. They also have lower idle power consumption compared to Raspberry Pi 5.
- 8
Why Running Databases on Kubernetes is Like Storing Critical Data on a Fragile Flash Drive
Running databases on Kubernetes can be risky due to the potential for storage issues such as dangling Persistent Volumes (PV) and Persistent Volume Claims (PVC). The concept is likened to using a flash drive for critical data storage, with similar risks of data corruption and resource waste. The reliance on the CSI controller in Kubernetes can lead to data inaccessibility and manual recovery, making it less suitable for highly stateful, storage-intensive applications like databases.
- 9
omg! ubuntu!·2y
Raspberry Pi Launch Own-Brand SSDs Priced From $30
Raspberry Pi has launched its own range of branded SSDs, making it simpler to add high-performance storage to the Raspberry Pi 5. The drives, available in 256 GB and 512 GB sizes, are competitively priced and guaranteed to work well with official and most third-party M.2 HAT expansions. Additionally, Raspberry Pi offers an SSD kit that includes the required M.2 HAT, providing an all-in-one solution for users who don’t have an expansion board.
- 10
- 11
All Things Distributed·2y
Continuous reinvention: A brief history of block storage at AWS
Marc Olson shares his decade-long experience shaping Elastic Block Store (EBS) at AWS. He reflects on its evolution from a basic block storage service to a system delivering over 140 trillion daily operations. Key takeaways include lessons from queueing theory, the effectiveness of incremental improvements, solving noisy neighbor problems, and the importance of performance measurement. The post also highlights significant shifts like transitioning from HDDs to SSDs and moving to Nitro hardware, which enhanced performance metrics substantially. EBS's journey showcases how continuous innovation meets growing customer demands.
- 12
- 13
Laravel News·2y
Easy management of your application settings with Setting Pro
Setting Pro is a package for easy management of application settings in Laravel. It offers flexible storage options, caching, queue support, event triggers, a global helper function and facade, and more. Visit the GitHub repo for installation instructions and more information.
- 14
ITNEXT·1y
Kubernetes Storage Performance Comparison Rook Ceph and Piraeus Datastore (LINSTOR)
Understanding Kubernetes storage options is vital for persistent volumes within K8s. This post compares Rook Ceph and Piraeus Datastore (LINSTOR) for their performance in providing replicated block storage. Details on various software options, commercial solutions, and open-source tools like LongHorn, OpenEBS, Vitastor are provided. Comprehensive performance testing, implementation challenges, and specific setup details for Rook Ceph and Piraeus Datastore (LINSTOR) are covered. Rook Ceph is noted for large-scale setups, whereas Piraeus Datastore is appreciated for lower latency and simpler deployment.
- 15
ExplainingComputers·1y
Switching to Linux: Drives & Partitions
The post provides a guide for users transitioning from Windows to Linux, focusing on how to work with drives and partitions. It covers the basics of storage management in Linux, including mounting and unmounting drives, navigating file systems without drive letters, and understanding block devices. The post also explains the differences in file systems and partition management between the two operating systems.
- 16
Lobsters·1yZero Disk Architecture
Zero Disk Architecture involves offloading data storage to Amazon S3, allowing for scalable and elastic systems without managing stateful storage servers. This approach leverages the durability, availability, and cost-effectiveness of S3, making it suitable for various database systems, especially those prioritizing minimal latency and cost efficiency. Multiple systems, such as Snowflake and Clickhouse, already use S3 or similar services as their primary storage solutions.
- 17
AWS in Plain English·2y
AWS S3 and Endpoints Benefits
AWS S3 is a highly available object storage service with security and encryption features. There are different ways to access S3 bucket, including using VPC Gateway Endpoint or VPC Interface Endpoint. Consider using interface endpoint for secure access across regions or from on-premises.
- 18
InfoWorld·2y
Docker tutorial: Get started with Docker volumes
Docker containers are designed to be immutable, making persistent data challenging. Docker volumes offer a solution by allowing containers to write and retrieve data through a local file system interface. They can be created manually or using the Docker volume API for named volumes. Sharing volumes between containers requires careful management, as Docker does not automatically mediate conflicts. Volumes must also be manually removed, using commands like `docker volume prune`. However, volumes have limitations such as difficulties with file locking and challenges in application portability due to different storage topologies.
- 19
Disaggregated Storage
Disaggregated storage refers to the separation of compute and storage in database systems, enabling independent scaling of CPU and I/O resources and eliminating disk space limitations. While offering benefits like instant failover and easier scaling, it introduces latency challenges due to network I/O and requires a robust storage server. This approach is used by large tech companies and database vendors for better flexibility and scalability.
- 20
finout·2y
7 Ways to Reduce AWS S3 Spend
AWS S3 is a widely used scalable object storage service, but it can be costly if not managed efficiently. This guide provides seven strategies to reduce AWS S3 costs, including choosing the right storage class, setting up billing alarms, deleting unused objects, automating object management with lifecycle rules, minimizing cross-regional data transfers, compressing data, and partitioning data before querying. It highlights the importance of understanding S3 pricing components and leveraging tools like Finout for detailed cost management and optimization.
- 21
- 22
Flystorage is a Filesystem Equivalent Package for Node.js
Flystorage is a package for Node.js that provides a single API to integrate with various storage solutions. It supports adapters such as Amazon S3, Azure Storage Blob, Google Cloud Storage, In-Memory, and Local FS. The complete documentation can be found on flystorage.dev.
