Best of Big Data — July 2024

1. 1
   Video

   Community Picks·2y

   7 Must-know Strategies to Scale Your Database

   Understanding when and why to scale your database is essential to maintain optimal performance as your application grows. Key strategies include indexing for quick data retrieval, using materialized views for pre-computed snapshots of data, and implementing denormalization to simplify complex queries. Vertical scaling, adding resources to a single server, and caching frequently accessed data in a fast storage layer can enhance responsiveness. Replication bolsters availability and fault tolerance by creating database copies on multiple servers. Sharding, which involves splitting a database into smaller sections, enables horizontal scaling and manages large data loads efficiently.

   89
2. 2
   Article

   KDnuggets·2y

   5 Free Online Courses to Learn Data Engineering Fundamentals

   Explore five free online courses designed to teach the fundamentals of data engineering. These courses range from beginner-friendly introductions to comprehensive professional certificates. Key areas covered include data pipelines, databases, Python and Pandas, cloud computing, and big data tools like Hadoop and Apache Spark.

   46

   1
3. 3
   Article

   Community Picks·2y

   How SQL Enhances Your Data Science Skills

   SQL is vital for data scientists due to its ability to efficiently retrieve, manipulate, and analyze large datasets. Key SQL concepts such as SELECT statements, WHERE clauses, JOIN operations, and aggregate functions enhance data exploration, preparation, and integration. Mastering these SQL skills complements other data science tools and improves overall data handling capabilities.

   45

   1
4. 4
   Article

   KDnuggets·2y

   How to Perform Memory-Efficient Operations on Large Datasets with Pandas

   Learn effective techniques to handle and perform memory-efficient operations on large datasets using Pandas. Tips include using the `low_memory` parameter when loading data, converting data types, processing data in chunks, and employing vectorized operations instead of `apply` with lambda functions. Additional suggestions include using `inplace=True` for DataFrame modifications and filtering data before performing operations.

   29
5. 5
   Article

   Community Picks·2y

   Uber’s Secret to Handle Millions of Logs per second with ClickHouse

   Uber overhauled its logging infrastructure by switching to ClickHouse, an open-source OLAP database, to handle millions of logs per second. The change addressed key issues they faced with ElasticSearch, such as developer productivity, performance, and scalability. ClickHouse offers high throughput ingestion, fast query performance, efficient storage, dynamic indexing, and clustering capabilities, making it a robust and scalable solution for Uber's massive logging needs.

   29
6. 6
   Article

   Hacker News·2y

   Building and scaling Notion’s data lake

   Notion's data has grown 10x in three years, necessitating the creation and scaling of a dedicated data lake. Their initial architecture involved a complex sharded Postgres infrastructure but faced challenges with operability, speed, and cost. To manage these issues, they developed an in-house data lake using AWS S3 for storage and Apache Spark for processing, coupled with a Kafka-based ingestion system using Debezium CDC connectors. This scalable setup improved data ingestion times, reduced costs, and supported their AI and analytical needs. The data lake supports update-heavy block data and allows complex data transformations, making it efficient for both small and large-scale data operations.

   23
7. 7
   Article

   Lobsters·2y

   Bufstream: Kafka at 10x lower cost

   Bufstream is a Kafka-compatible queue that's 10x less expensive to operate and excels when paired with Protobuf. It integrates with the Buf Schema Registry for data quality and governance, and offers plans for adding granular RBAC and Apache Iceberg tables support. Bufstream uses S3-compatible storage to cut costs and deploys into AWS or GCP Kubernetes clusters. It charges a usage-based fee of $0.002 per GiB of write traffic.

   17
8. 8
   Article

   Substack·2y

   Lessons from the Frontlines of AI Training

   Top AI labs are facing a potential high-quality data shortage by 2026, emphasizing the critical importance of data quality over quantity. Successful AI models depend on meticulously curated datasets, balancing synthetic and real-world data. Advanced techniques like Joint Example Selection and GraphRAG improve efficiency and performance, while strategic data partnerships and scalable management solutions are pivotal. The future of AI hinges not just on model-building but on strategic data sourcing and refinement.

   13
9. 9
   Article

   Airbnb·2y

   Apache Flink® on Kubernetes

   Airbnb transitioned its stream processing architecture from Apache Hadoop Yarn to Kubernetes. This migration improved developer velocity, job availability, and infrastructure costs. The current setup integrates Flink directly with Kubernetes, offering a better developer experience, secure secrets management, isolated environments, enhanced monitoring, and simpler service discovery. The future focus will involve improving job availability, enabling autoscaling, and utilizing the Flink Kubernetes Operator for streamlined operations.

   11
10. 10
    Article

    Machine Learning News·2y

    Stumpy: A Powerful and Scalable Python Library for Modern Time Series Analysis

    Stumpy is a scalable Python library designed for modern time series analysis, specializing in identifying patterns, anomalies, and classificatory subsequences. Leveraging optimized algorithms, parallel processing, and early termination techniques, it efficiently computes matrix profiles, significantly reducing computational overhead and enhancing scalability. This enables data scientists and analysts to extract valuable insights from large time series datasets more effectively.

    11