Best of Google BigQuery — 2024

1. 1
   Article

   Hacker News·1y

   anvaka/map-of-github: Inspirational Mapping

   This project maps over 400,000 GitHub projects, clustering them based on common stargazers. Using data from GitHub's public activity events, Jaccard Similarity, and AWS for processing, the map visualizes connections between repositories. The layout was computed with ngraph.forcelayout and rendered with maplibre. Clusters and labels were generated with the help of various tools, including ChatGPT. The project is open-source under the MIT license, with contributions welcome.

   52

   3
2. 2
   Article

   Data Engineer Things·2y

   I spent 5 hours learning how Google manages terabytes of metadata for BigQuery.

   Google BigQuery uses innovative techniques to manage massive amounts of metadata efficiently, treating it as crucial as the data itself. BigQuery's architecture includes Colossus for storage, Dremel for querying, and a dedicated shuffle service, all coordinated by Borg. Metadata is handled in a distributed manner using a unique columnar storage format called CMETA, improving efficiency and performance. Real-time data ensures physical query plans adapt dynamically for optimized results, while integrated metadata scans enhance query processing.

   51
3. 3
   Article

   Pointer·2y

   bruin-data/ingestr: ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

   Ingestr is a CLI tool that allows you to copy data between databases with a single command, providing features such as incremental loading and easy installation.

   30

   1
4. 4
   Article

   freeCodeCamp·2y

   Unlock the Power of Data Analytics with Free Google Services

   Learn data analytics using free Google services like Google Sheets, Google BigQuery, Google Colab, and Google Looker Studio for data visualization.

   26
5. 5
   Article

   freeCodeCamp·2y

   Learn SQL for Analytics

   Learn SQL for Analytics is an 11-hour course on the freeCodeCamp.org YouTube channel. It covers the fundamentals of SQL, includes practical exercises, and is suitable for beginners.

   16
6. 6
   Article

   DEV·2y

   How to Build a RAG Chat App With Agent Cloud and BigQuery

   Learn how to build a RAG chat app using Agent Cloud and BigQuery. This comprehensive guide takes you through the process step by step, covering topics such as setting up BigQuery, creating a GCP service account key, running Agent Cloud locally, adding models, connecting to BigQuery as a data source, creating tools, agents, tasks, and conversation chat apps.

   15
7. 7
   Article

   KDnuggets·2y

   Machine Learning Made Simple for Data Analysts with BigQuery ML

   BigQuery ML democratizes machine learning for data analysts by enabling the creation and execution of ML models using SQL queries. It supports tasks such as predictive analytics, classification, recommendation engines, and anomaly detection without requiring knowledge of Python or R. BigQuery ML is scalable, integrated with data storage, fast, and cost-effective, making it ideal for analysts looking to add ML capabilities to their workflows. Key steps include data preparation, model selection, training, evaluation, and prediction.

   13
8. 8
   Article

   Medium·2y

   Rust in production, for fun & profit

   The author shares their experience of rewriting a legacy application in Rust for improved correctness and efficiency. They discuss the building blocks, feature parity, performance, documentation, middlewares, security, and deployment of the Rust application.

   12