Crawl4AI: Open-source LLM Friendly Web Crawler & Scraper

Overview

Crawl4AI is an open-source web crawler designed for large-scale web extraction, optimized for generating clean, LLM-ready Markdown. It is currently in closed beta for its cloud API, which aims to provide a cost-effective solution compared to existing options.

Latest Updates

• Version 0.8.8: Introduced a backward-compatible security patch for the self-hosted Docker API server.
• Version 0.8.6: Security hotfix replacing litellm with unclecode-litellm due to a supply chain compromise.
• Version 0.8.0: Added crash recovery and prefetch mode for faster URL discovery.

Features

• Markdown Generation: Generates clean, structured Markdown with accurate formatting and citation hints.
• LLM-Driven Extraction: Supports structured data extraction using various LLMs.
• Browser Integration: Managed browser support with session management and proxy capabilities.
• Dynamic Crawling: Executes JavaScript and waits for dynamic content to load.
• Deployment Options: Can be deployed using Docker or as a Python package.

Installation

To install Crawl4AI, use the following command:

pip install -U crawl4ai

For Docker deployment:

docker pull unclecode/crawl4ai:latest
docker run -d -p 11235:11235 --name crawl4ai unclecode/crawl4ai:latest

Quick Start Example

Run a simple web crawl with Python:

import asyncio
from crawl4ai import *

async def main():
    async with AsyncWebCrawler() as crawler:
        result = await crawler.arun(url="https://www.nbcnews.com/business")
        print(result.markdown)

if __name__ == "__main__":
    asyncio.run(main())

Sponsorship Program

Crawl4AI offers a sponsorship program to support development and maintain independence. Tiers include:

• Believer: $5/mo
• Builder: $50/mo
• Growing Team: $500/mo
• Data Infrastructure Partner: $2000/mo

Mission

Crawl4AI aims to democratize data extraction and empower individuals and organizations to transform digital footprints into structured assets.

Contact

For questions or feedback, reach out via GitHub or Twitter.

License

Crawl4AI is licensed under the Apache License 2.0.