Self-hosted Web Crawlers
A curated collection of the best self-hosted tools for automatically crawling and extracting data from websites.
Turn any website into clean, AI-ready data via API
Stars
180,814Last commit
8 hours agoLicense
AGPL-3.0
API for AI agents to search, scrape, crawl, and interact with the live web, returning clean Markdown, structured JSON, or screenshots from any page.
Open Source Alternative to:
LLM-ready web crawler built for AI data pipelines
Stars
83,562Last commit
2 days agoLicense
Apache-2.0
Open-source web crawler and scraper that produces clean, structured output optimized for LLMs, RAG pipelines, and AI agents. Supports async crawling, CSS/XPath/LLM extraction, and stealth browser control.
Open Source Alternative to:
c15t – Open-source cookie banner, built for control and lightening fast modern web apps.
CodeRabbit