Open Source Projects tagged "Web Crawling"
A curated collection of the best open source projects tagged "Web Crawling". Each listing includes a website screenshot along with a detailed review of its features.
LLM-ready web crawler built for AI data pipelines
Stars
84,447Last commit
4 days agoLicense
Apache-2.0
Open-source web crawler and scraper that produces clean, structured output optimized for LLMs, RAG pipelines, and AI agents. Supports async crawling, CSS/XPath/LLM extraction, and stealth browser control.
Open Source Alternative to:
Windshift – The self-hosted platform that replaces Jira.
Windshift