combineCtrl - controls a Combine crawling job
combineCtrl <action> --jobname <name>
where action can be one of start, kill, load, recyclelinks, reharvest, stat, howmany, records, hosts, initMemoryTables, open, stop, pause, continue
jobname is used to find the appropriate configuration (mandatory)
takes an optional switch --harvesters n where n is the number of crawler processes to start
--harvesters n
n
kills all active crawlers (and their associated combineRun monitors) for jobname
Read a list of URLs from STDIN (one per line) and schedules them for crawling
Schedule all newly found (since last invocation of recyclelinks) links in crawled pages for crawling
Schedules all pages in the database for crawling again (in order to check if they have changed)
opens database for URL scheduling (maybe after a stop)
stops URL scheduling
pauses URL scheduling
continues URL scheduling after a pause
prints out rudimentary status of the ready queue (ie eligible now) of URLs to be crawled
prints out rudimentary status of all URLs to be crawled
prints out the number of ercords in the SQL database
prints out rudimentary status of all hosts that have URLs to be crawled
initializes the administrative MySQL tables that are kept in memory
Implements various control functionality to administer a crawling job, like starting and stoping crawlers, injecting URLs into the crawl queue, scheduling newly found links for crawling, controlling scheduling, etc.
This is the preferred way of controling a crawl job.
echo 'http://www.yourdomain.com/' | combineCtrl load --jobname aatest
Seed the crawling job aatest with a URL
aatest
combineCtrl start --jobname aatest --harvesters 3
Start 3 crawling processes for job aatest
combineCtrl recyclelinks --jobname aatest
Schedule all new links crawling
combineCtrl stat --jobname aatest
See how many URLs that are eligible for crawling right now.
combine
Combine configuration documentation in /usr/share/doc/combine/.
Anders Ardö, <anders.ardo@it.lth.se>
Copyright (C) 2005 Anders Ardö
This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.4 or, at your option, any later version of Perl 5 you may have available.
See the file LICENCE included in the distribution at http://combine.it.lth.se/
1 POD Error
The following errors were encountered while parsing the POD:
Non-ASCII character seen before =encoding in 'Ardö,'. Assuming CP1252
To install Combine::UA, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Combine::UA
CPAN shell
perl -MCPAN -e shell install Combine::UA
For more information on module installation, please visit the detailed CPAN module installation guide.