Changes for version 1.08_01
- Add clear() call to the DNS cache on startup. On systems with prolonged process life time, it's likely that the DNS cache is holding bad data, so we might as well reinitialize.
- add tools/checkreqs.pl and custom Makefile.PL
- Fix Push.pm so that shutdown_broker state kills alarms.
- Try to work with servers that send 400/408 upon too many connections (the sysadmins must hate me) by gracefully reducing the number of HTTP fetchers upon encountering those return status
Learn How To Write Crawlers With Xango
Event Based High Performance Web Crawler Framework
Broker HTTP Requests (Deprecated)
Base Class for Xango Broker
Global Xango Config