https://spatie.be/docs/crawler
2.8k
Stars
367
Forks
0
Open issues
30
Contributors
AI Analysis
Spatie Crawler is a PHP library for crawling websites with concurrent HTTP requests via Guzzle, supporting both static HTML and JavaScript-rendered content through Puppeteer integration. It serves web developers and DevOps engineers who need programmatic site scraping, link discovery, or content auditing; it is not a general-purpose web scraper framework for non-technical users.
Inferred from signals mentioned in the README (tests, CI, type safety) — not a review of the actual code.
AI's overall editorial judgment — not an average of the bars above, can weigh other factors too.
Mature PHP web crawler with concurrent requests and JavaScript rendering support
spatie/crawler is a PHP library for crawling websites at scale, built on Guzzle for concurrent HTTP requests and Puppeteer/Chrome for JavaScript-rendered content. It targets PHP developers who need programmatic site scraping, link discovery, and content indexing. The project has been stable since 2015 and is actively maintained by Spatie, a established open-source organization. Real-world adoption appears solid but is difficult to quantify precisely.
Created in 2015 by Spatie, a Belgian web development agency known for quality PHP packages. The project evolved from basic link-following to include concurrent request handling via Guzzle and JavaScript rendering via Puppeteer integration. It reflects the maturation of PHP tooling for web automation tasks.
Growth appears modest and plateaued. Star count (2,829) is slightly higher than the older php-spider (1,345) but trails node-crawler (6,797). Recent activity shows 4 stars gained in 7 days, indicating steady but not accelerating adoption. Last push was 2026-06-12, suggesting active maintenance rather than explosive growth.
Adoption not verified from README alone. Packagist download metrics referenced but not quantified in excerpt. No case studies, testimonials, or enterprise deployments mentioned. Spatie's reputation as a package producer suggests real-world usage, but concrete production evidence is not presented. The presence of 367 forks and GitHub workflow badges suggests integration into some production pipelines, but scale is unknown.
Appears to wrap Guzzle HTTP client for concurrent requests and Spatie's Browsershot library for JavaScript execution. Based on README, the API is fluent (method chaining), with callbacks for handling crawled responses and control flow via shouldStopCallback. Likely uses URL filtering, depth limiting, and internal/external URL discrimination.
README mentions `composer test` command exists, but coverage percentage not documented. Test workflow badge present in README, suggesting CI/CD is configured, but specifics not disclosed.
Last push 2026-06-12 (23 days before analysis date) indicates active maintenance. Repository is not dormant. Issue response time and PR merge velocity cannot be assessed from metadata alone. Changelog referenced but not detailed in excerpt. Framework requires PHP and Composer ecosystem, suggesting compatibility work needed for new PHP versions and dependency updates.
ADOPT IF: you are a PHP developer needing concurrent web crawling with JavaScript support, have existing Spatie package investment, and require fluent API design. AVOID IF: you need language-agnostic tooling, prefer pure async/await patterns over Guzzle promises, or require extensive third-party integrations beyond HTTP and rendering. MONITOR IF: you need to compare with php-spider or QueryList for specific feature gaps (concurrent depth, custom middleware, performance under high concurrency), or if you depend on Puppeteer/Chrome availability in your deployment environment.
Independent dimensions
Mainstream potential
3/10
Technical importance
6/10
Adoption evidence
4/10
- Dependency on Chrome/Puppeteer for JavaScript rendering adds significant overhead and requires browser binary availability in production — may be impractical in containerized or serverless environments.
- Guzzle-based concurrency may not scale to thousands of concurrent requests; real limits not documented in README.
- Adoption appears concentrated in PHP ecosystem; Node.js alternative (node-crawler) has 2.4× higher star count, suggesting possible feature or ecosystem advantage in competitor.
- No mention of proxy support, rate limiting, or robots.txt compliance in README — critical for production web crawling; may exist but undocumented.
- Dependency chain includes Guzzle, Browsershot, and Chrome; version conflicts or breaking changes in dependencies could affect stability. Active maintenance mitigates but does not eliminate risk.
Project will remain stable and actively maintained as part of Spatie's portfolio. Adoption likely stays within PHP developer community at modest scale. Unlikely to see explosive growth given Node.js ecosystem dominance and Python's Scrapy; positioned as niche solution for PHP shops. May benefit from newer PHP versions and async/await features but not expected to catalyze major market shift.
Newsletter
Get analyses like this every Monday
Free weekly digest of the most interesting open-source discoveries.
Languages
Information
- Language
- PHP
- License
- MIT
- Last updated
- 4w ago
- Created
- 130mo ago
- Analyzed with
- anthropic/claude-haiku-4-5
Stars over time
Contributors over time
Top 100 contributors only — repos with more will plateau at 100.
Open issues
No open issues — clean slate.
Open pull requests
No open pull requests.
Top contributors
Recent releases
Similar repos
bda-research/node-crawler
node-crawler is a web scraping library for Node.js that combines HTTP requests...
| Repository | Stars | Week Δ | Language | Score | Updated |
|---|---|---|---|---|---|
|
spatie/crawler
this
|
2.8k | +1 | PHP | 8/10 | 4w ago |
|
spatie/async
|
2.8k | — | PHP | 8/10 | 1mo ago |
 mvdbos/php-spider
|
1.3k | — | PHP | 7/10 | 2w ago |
 jae-jae/QueryList
|
2.7k | — | PHP | 7/10 | 3w ago |
|
bda-research/node-crawler
|
6.8k | — | TypeScript | 7/10 | 3w ago |
|
spatie/fork
|
1k | — | PHP | 8/10 | 1mo ago |
Older PHP crawler (lower stars: 1,345 vs 2,829). Comparison suggests spatie/crawler may have superseded or captured more recent mindshare, though both remain viable. Direct feature parity not assessable from README.
Similar star count (2,696 vs 2,829), also PHP-based. Likely different design philosophy; unclear if it supports concurrent requests or JavaScript rendering without inspecting code. Both occupy PHP scraping niche.
Related Spatie project (2,811 stars), handles async/parallel execution. Likely complementary to crawler rather than competitive; crawler may use similar concurrency primitives under the hood.
TypeScript equivalent with significantly higher adoption (6,797 stars). Suggests Node.js ecosystem has stronger web-crawling demand or better tooling maturity. Language barrier means not direct replacement for PHP teams.
