Crawlee is an open-source web scraping and browser automation framework developed by Apify for JavaScript and Python. It provides a unified toolkit for building scalable crawlers, browser automation workflows, and structured web data extraction systems across both HTTP-based and browser-based scraping environments.

The framework is designed to simplify production-grade scraping by handling common infrastructure concerns such as request queues, retries, session management, parallelization, browser lifecycle management, proxy rotation, and anti-blocking protections. Developers can focus on extraction logic while Crawlee manages scaling and reliability.

Crawlee supports multiple crawler types through a shared interface. Developers can use lightweight HTTP crawlers with tools like Cheerio, BeautifulSoup, or JSDOM for static pages, or full browser automation with Playwright and Puppeteer for JavaScript-heavy websites. This flexibility allows the same framework to be used for both high-speed scraping and interactive browser workflows.

A key feature of Crawlee is its automatic scaling system. Crawlers dynamically adjust concurrency based on available CPU and memory resources, enabling efficient large-scale crawling without requiring manual tuning. The framework also includes persistent URL queues, configurable retries, hooks, storage systems, and export pipelines for datasets and files.

Crawlee integrates closely with the Apify platform, where projects can be deployed as cloud “Actors” with managed infrastructure, scheduling, storage, monitoring, and proxy support. However, the framework itself remains fully open source and can run locally or on any cloud infrastructure.

Recent versions of Crawlee have expanded support for AI and LLM workflows. Developers use Crawlee to gather structured web data for retrieval-augmented generation (RAG), AI agents, search systems, and dataset pipelines. The Python edition, officially launched after an extended beta period, introduced features such as adaptive crawling and unified storage APIs for AI-native scraping systems.

The framework has become widely adopted in the web scraping ecosystem due to its combination of browser automation, anti-blocking capabilities, and scalable crawling infrastructure in a single developer-oriented toolkit.

Key features include:

• Unified framework for HTTP and browser-based scraping
• Support for Playwright, Puppeteer, Cheerio, JSDOM, and BeautifulSoup
• Automatic scaling based on system resources
• Persistent request queues and retry handling
• Proxy rotation and anti-blocking tooling
• Browser fingerprinting and session management
• Parallel crawling and concurrency controls
• Dataset export and structured storage systems
• JavaScript, TypeScript, and Python support
• Open-source Apache 2.0 license

Common use cases include:

• Large-scale web scraping and crawling
• Browser automation workflows
• AI data collection and RAG pipelines
• Structured dataset generation
• Monitoring websites and extracting dynamic content
• Building cloud-deployed scraping systems
• Automating repetitive browser interactions

Crawlee is developed by Apify and positioned as a scalable, developer-focused framework for web scraping, browser automation, and AI-oriented data extraction workflows.