Best of Crawling — March 2025

  1. 1
    Video
    Avatar of TechWithTimTech With Tim·1y

    I Built a Web Scraping AI Agent From Scratch - It's Insane...

    Building powerful AI applications requires the integration of large language models (LLMs) with real-time data and useful tools. In this post, the author demonstrates the development of an AI travel agent using Python. This agent uses Bright Data APIs for real-time travel data, Google Flights, and hotel information to provide relevant and current recommendations. The post covers the project's architecture, details the steps of web scraping with automated browsers, and explains how the AI processes and combines data to generate personalized travel plans.

  2. 2
    Video
    Avatar of bytegradByteGrad·1y

    AI-Scraping Is Getting Crazy Easy Now

    Traditional web scraping required manually pinpointing data within HTML structures and managing infrastructure for requests. AI-based solutions like Scraperless simplify this process by allowing users to describe the desired data without specifying HTML details. Scraperless utilizes free-form prompts to determine scraping targets and offers integration via API keys, allowing developers to incorporate it into their applications seamlessly. Results are available in formats like CSV, making data handling straightforward with minimal user effort.

  3. 3
    Article
    Avatar of neontechNeon·1y

    Building RagRabbit, An Open Source RAG Search with Postgres as the Vector Store

    RagRabbit is an open-source tool designed to simplify Retrieval-Augmented Generation (RAG) workflows by using Postgres with pgVector for handling vector embeddings. It can crawl websites, convert pages to Markdown, generate embeddings, and use MCP servers to integrate with development tools. RagRabbit supports secure user authentication and can be deployed effortlessly using Vercel.

  4. 4
    Article
    Avatar of gcgitconnected·1y

    I Tried 20+ No-Code Web Scraping Tools to Make Money — These 3 Are the Absolute Best

    The post explores three effective no-code web scraping tools - Octoparse, Magical Chrome Extension, and Browse AI. It provides detailed instructions on how to use these tools and highlights their unique features, such as ease of use, data extraction capabilities, and cloud execution options. Furthermore, it offers insights on how to monetize web scraping by providing services like lead generation, competitor analysis, and product data scraping for clients on freelance platforms or through direct outreach.

  5. 5
    Article
    Avatar of microsaasexamplesMicro SaaS Examples·1y

    CaptureKit: A Powerful Web Scraping API for Developers

    CaptureKit is a powerful web scraping API that simplifies content extraction and visualization for developers and businesses. It offers high-quality full-page or viewport screenshots, effortless data extraction including HTML and metadata, smart link scraping for SEO analysis, AI-powered summaries, and distraction-free captures by blocking ads and pop-ups. CaptureKit allows customizable rendering and provides easy integration with its straightforward API requests.

See all Crawling archives