/ 
WWW-Crawl4AI-0.003

Changes for version 0.003 - 2026-06-11

  • Detect: captcha signal no longer false-positives on cookie-banner / privacy-policy mentions of reCAPTCHA. A content-rich page now only walls when a markdown captcha marker co-occurs with captcha-PROMPT language ("complete the captcha to continue", "I'm not a robot", "verify you are human", …); thin pages still wall on any marker (JS-rendered gate) and html-only markers on rich pages still never wall
  • Detect: signals now also flags WAF / bot-management gates that REDIRECT to a challenge URL instead of embedding a widget. When the page's final_url matches a known challenge endpoint, blocked is raised for Cloudflare / DataDome / PerimeterX (/cdn-cgi/challenge, __cf_chl, /challenge-platform/, datadome, geo.captcha-delivery.com, /px/captcha, perimeterx) and captcha for provider verification endpoints (google.com/recaptcha, /recaptcha/api, hcaptcha.com). Additive and OR-ed in; cosmetic redirects (http->https, www, trailing slash) and an absent final_url never trigger

Documentation

www-crawl4ai-doctor
probe Crawl4AI / CloakBrowser / proxy reachability and print the chain
www-crawl4ai-test-url
run the full WWW::Crawl4AI strategy chain against one URL

Modules

WWW::Crawl4AI
Perl client and fallback orchestrator for Crawl4AI
WWW::Crawl4AI::Attempt
one strategy attempt in a WWW::Crawl4AI fallback chain
WWW::Crawl4AI::Client
UA-agnostic REST client for the Crawl4AI Docker API
WWW::Crawl4AI::DeepCrawlIterator
breadth-first iterator for deep_crawl, separating frontier management from crawl logic
WWW::Crawl4AI::Detect
service detection and content-quality classification for Crawl4AI
WWW::Crawl4AI::Error
structured error class for WWW::Crawl4AI
WWW::Crawl4AI::Markdown
markdown field resolution across Crawl4AI response shapes
WWW::Crawl4AI::Request
builds Crawl4AI /crawl and /md request payloads
WWW::Crawl4AI::Result
normalized result of a WWW::Crawl4AI strategy chain
WWW::Crawl4AI::Strategy
role for a single crawl strategy in the WWW::Crawl4AI fallback chain
WWW::Crawl4AI::Strategy::Browser
Crawl4AI strategy with full JS rendering (wait for networkidle)
WWW::Crawl4AI::Strategy::Callback
last-resort Crawl4AI strategy delegating to a user coderef
WWW::Crawl4AI::Strategy::CloakBrowser
Crawl4AI strategy attaching to CloakBrowser over CDP
WWW::Crawl4AI::Strategy::Plain
cheapest Crawl4AI strategy — headless text mode, no escalation
WWW::Crawl4AI::Strategy::Proxy
Crawl4AI strategy routing through a configured proxy
WWW::Crawl4AI::Strategy::Stealth
Crawl4AI strategy with enable_stealth and randomized fingerprint
WWW::Crawl4AI::StrategyChain
ordered list of strategy objects, pluggable at construction time

Examples

Other files