Best of CrawlingMay 2025

  1. 1
    Article
    Avatar of freecodecampfreeCodeCamp·51w

    How To Build A Simple Portfolio Blog With Next.js

    A comprehensive guide to building a portfolio blog with Next.js that automatically aggregates articles from multiple platforms. The tutorial covers creating server and client components, implementing web scraping with Cheerio to extract metadata from article URLs, building search and filtering functionality, and structuring a JSON-based content management system without requiring a database.

  2. 2
    Article
    Avatar of aiAI·1y

    Maven AI: Building an AI-Powered Product Research Assistant

    Maven AI is an open-source project created to streamline electronic product research with AI. It offers features like personalized recommendations, fast product searches, deep insights, and side-by-side comparisons to enhance user decision-making. The project utilizes a modular architecture with an orchestrator and agent tools, leveraging TypeScript and React for type-safety and a responsive UI. To handle data, it uses web scraping, APIs, and various technologies like Google Gemini, Firecrawl, and Upstash Redis. The project is open-source for community collaboration and future scalability improvements.

  3. 3
    Article
    Avatar of dailydoseofdsDaily Dose of Data Science | Avi Chawla | Substack·1y

    Building a Multi-agent Book Writer Using Qwen 3

    The post guides readers through creating a multi-agent workflow to write a 20k-word book using a title as brief as 3-5 words. Key technologies include Firecrawl for web scraping, CrewAI for workflow orchestration, and Ollama for serving Qwen 3 locally. The system combines research, outline creation, and parallel chapter writing, leading to the assembly of a complete book in Markdown format.

  4. 4
    Article
    Avatar of dailydoseofdsDaily Dose of Data Science | Avi Chawla | Substack·1y

    One MCP Server to Access the Web

    Bright Data's open-source MCP server offers over 30 tools helping AI agents access and navigate the web without encountering blocks. These tools include a browser tool, APIs for web unlocking, and platform-specific scrapers for various websites like Instagram and YouTube. The MCP server selects the best tool for each site, facilitating seamless data interactions for agents.

  5. 5
    Article
    Avatar of phProduct Hunt·1y

    Capture - Fast browser API to capture, extract, & create from web

    Capture offers a fast and reliable browser API to efficiently capture screenshots, create PDFs, scrape content, and extract metadata from websites. It's designed for developers and marketers seeking high performance automation tools. Recent upgrades have improved speed, ad blocking, bot detection bypass, and render quality, enhancing user experience.