Best of Data Management — September 2024

1. 1
   Article

   The Polymathic Engineer·2y

   How to design a system for scale

   Scalability is essential for software engineers as applications grow. Three key techniques for scaling systems are adding server clones, functional partitioning, and data partitioning. Adding server clones involves creating interchangeable copies of existing servers to distribute loads. Functional partitioning breaks down the system into smaller, independent components each handling specific tasks. Data partitioning divides datasets across multiple machines to speed up processing and storage. Each technique has its pros and cons and requires careful consideration for effective implementation.

   161

   1
2. 2
   Article

   Decube·2y

   Introducing the Decube Dashboard

   Decube has launched a revamped Dashboard with powerful new features aimed at enhancing data management. The updated dashboard provides a comprehensive snapshot of data assets and activities, including incident management and data quality scores. Customization options allow organizations to personalize their dashboard with their own logo, improving user experience.

   14
3. 3
   Video

   The Serious CTO·2y

   Data Mesh: The Future of Data Engineering Explained

   Data Mesh redefines data architecture by decentralizing data management. Instead of centralizing all data in one big system, each department manages its own data, ensuring it's clean and accessible. This approach aims to eliminate bottlenecks, improve data quality, and foster better collaboration with shared standards across the company.

   13
4. 4
   Article

   Hugging Face·2y

   Introducing the SQL Console on Datasets

   Hugging Face has introduced an SQL Console for querying datasets directly in the Hub, powered by DuckDB WASM. This browser-based tool allows for local queries without dependencies, supports full DuckDB syntax, and enables result exportation to Parquet files. Designed to manage even large datasets, it simplifies tasks like converting data formats and performing complex queries efficiently.

   12
5. 5
   Article

   ITNEXT·2y

   Why Data in Enterprise Keeps Breaking

   Maintaining data consistency in enterprise settings is challenging due to a lack of a single record system, involving multiple services. This complexity leads to data anomalies, vulnerabilities, and financial risks. The article discusses the need for a central, trusted authority to reconcile data inconsistencies and highlights the evolution from Enterprise Service Bus to modern data-platforms, addressing integration and decentralization issues in a more reliable way.

   11