Daily Dose of Data Science | Avi Chawla | Substack·1yPandas Mind Map
A detailed mind map of various Pandas methods categorized by their operation types, including I/O methods, DataFrame creation, statistical information, renaming, plotting, time-series, grouping, pivot, and categorical data methods. Additional ML resources and techniques are also provided for developing industry-relevant skills.
Oxylabs·1yBuilding a Real Estate Monitoring System
Alex discusses building a real estate monitoring system, focusing on the types of data that can be extracted from real estate websites, the use cases for the extracted data including price comparisons and market trends, and the challenges faced such as getting fresh data, overcoming anti-bot measures, and scaling the system. He then advises using Oxylabs' Real Estate Scraper API to handle these challenges efficiently.
Javarevisited·1yHow to Learn Data Analytics in 2025? (with Resources)
Data analytics is a highly sought-after skill in 2025, offering competitive salaries and diverse opportunities across industries. To master this field, it's recommended to use a combination of reading books, watching online tutorials and courses, doing projects, joining bootcamps, and gaining real-world experience. Google offers two key certificate programs on Coursera: the Google Data Analytics Professional Certificate for beginners and the Google Advanced Data Analytics Certificate for those looking to dive deeper. These programs cover essential skills such as data cleaning, statistical analysis, data visualization, and SQL. Additionally, platforms like DataCamp and Kaggle can further enhance your learning experience.
JetBrains·1yData Cleaning in Data Science
Data cleaning is essential for transforming real-world, messy datasets into reliable sources for analysis or machine learning. This involves removing duplicates, dealing with implausible values, addressing formatting issues, outliers, and missing values. Proper data cleaning ensures that conclusions drawn from the data can be generalized to a defined population. Best practices include defining your population boundaries, ensuring reproducibility, and keeping methods well-documented.
Aaron Jack·1yHow to Build Powerful Web Scrapers with AI - 3 Steps
Combining AI with web scraping has enormous potential, providing a way to create applications and services by extracting and transforming data from the web efficiently. The post details the challenges of traditional web scraping, such as brittle scripts and diverse HTML structures, and explains how AI can standardize this process. It includes step-by-step methods using tools like Puppeteer, Selenium, and proxies to avoid detection and manage large-scale scraping. Example applications and a brief overview of reducing costs by running models locally are also discussed.
Data Engineer Things·1yNetflix Movie Analytics (Homemade)
A data engineer combines a passion for film with data analytics by analyzing their Netflix viewing habits. Using data exported from Netflix and enriched through The Movie Database (TMDB) API, they store and process the data on Google Cloud Platform (GCP). The data is modeled into a Star Schema on Google BigQuery, orchestrated with Airflow, and visualized using Tableau. Key insights include favorite genres, preferred viewing days, and overall streaming patterns.
Community Picks·1yIs sending Factorio to your competitors' engineers a cost-effective means of sabotage?
The post evaluates the potential productivity loss in competitive engineering firms if Factorio, a highly addictive game, is gifted to their engineers. Using probabilistic estimation tools and compensation data, the author estimates the monetary loss and concludes that gifting Factorio may indeed be a costly form of sabotage.
Towards Data Science·1yScaling Statistics: Incremental Standard Deviation in SQL with dbt
Incremental aggregation in SQL helps maintain efficiency when recalculating metrics for large datasets. This is particularly useful for complex calculations like standard deviation, which involves updating both the mean and the sum of squared differences. By using algebraic manipulation, a formula for incremental computation can be derived, avoiding the need to recalculate from scratch with each new data point. The example provided demonstrates how to implement this using dbt, enabling efficient and scalable real-time data aggregation.
Community Picks·1yAI Text2SQL Tool for Easy Database Management
Chat2DB is an open-source AI tool that facilitates easy database management by generating SQL queries and coding efficiently with an AI SQL editor. It supports various databases, ensuring data security with features like local query processing, AES and RSA encryption, and SSH tunneling. Chat2DB also aids in data analysis, offering fast insights and dashboard creation. It ensures high stability for big data queries and supports easy collaboration through shared links and API integration.
