CPAN::Search::Lite::Index - set up or update database tables.
my $index = CPAN::Search::Lite::Index->new(config => 'cpan.conf', setup => 1); $index->index();
This is the main module used to set up or update the database tables used to store information from the CPAN and ppm indices. The creation of the object
my $index = CPAN::Search::Lite::Index->new(%args);
accepts three arguments:
config => /path/to/config.conf
This argument specifies where to find the configuration file used to determine the remaining options. In lieu of this option, the environment variable
CSL_CONFIG_FILEpointing to the configuration file may be specified.
setup => 1
This (optional) argument specifies that the database is being set up. Any existing tables will be dropped.
reindex => value
This (optional) argument specifies distribution names that one would like to reindex in an existing database. These may be specified as either a scalar, for a single distribution, or as an array reference for a list of distributions.
Most of the options used to control the behaviour of the indexing are contained in a configuration file. An example of the format of such a file is
[CPAN] CPAN = /var/ftp/pub/CPAN pod_root = /usr/local/POD html_root = /usr/local/httpd/htdocs/CPAN [DB] db = pause user = sarah passwd = lianne [WWW] css = cpan.css up_img = up.gif tt2 = /usr/local/tt2 geoip = /usr/local/share/geoip/cpan.txt
This consists of 3 sections.
This is associated with various things related to CPAN.
CPAN = /var/ftp/pub/CPAN
This specifies the root directory of the local CPAN mirror, if this exists, or the location where the CPAN index files will be downloaded and kept, if the
no_mirroroption is specified.
pod_root = /usr/local/POD
This specifies where the extracted pod files from a distribution will be kept. A subdirectory
dist_nameunder this directory will be created corresponding to the name of the distribution.
pod_only = 1
This specifies that, if the module files are to be extracted, fetch only those that contain pod.
split_pod = 1
This specifies that, if the module files are to be extracted, when generating the html pages create two pages for each module: one containing just the documentation, and the other containing the code run through
Perl::Tidy. For a module such as
Foo::Bar, the documentation will be saved as a file Foo/Bar.html, while the sources will be saved as Foo/Bar.pm.html.
html_root = /usr/local/httpd/htdocs/CPAN
This specifies where the html files created from the pod files will be kept. A subdirectory
dist_nameunder this directory will be created corresponding to the name of the distribution.
ignore = some_dist_name_to_ignore
This specifies a name of a distribution (without a version number) to ignore in indexing. This option may be given a number of times to specify an array of values, or may be specified as
ignore = <<EOL Module-CPANTS-asHash CORBA-IDL EOL
This array of values (which may include regular expressions) is joined together as
$pat = join '|', @ignore_dists
and if the distribution name matches
$dist_name =~ /^($pat)$/
the distribution is ignored.
no_mirror = 1
This specifies that a local CPAN mirror isn't available, and as such no pod or html files will be extracted nor created.
no_ppm = 1
This can be used to signal to not gather information on Win32 ppm packages from the repositories specified in
remote_mirror = http://cpan.wherever.edu
no_mirroris specified, the value of
remote_mirrorwill be used to fetch the CPAN indices. If not given, http://www.cpan.org will be used.
multiplexer = http://cpan.redirect.edu/cpan
This can be used to specify a multiplexer to redirect downloads to nearby CPAN mirrors. See, for example, Apache::GeoIP for one implementation.
cat_threshold = 0.99
Many modules do not have a category (chapter) associated with them. In such cases, when populating the database, the AI::Catgorizer module is used to guess which category should be assigned to such a module, based on available information for those modules that do have a category. The value of cat_threshold is used to determine if the guessed category should be accepted (a perfect match has a score of 1, and no match has 0). If no such value is given, a default of 0.995 is used.
no_cat = 1
Set no_cat equal to a true value if you don't want AI::Categorizer to try categorizing modules which don't have a category assigned.
This is used to store connection information to the database used to populate the tables.
db = pause
This is the name of the database used. It is assumed here that the database has already been created, and that appropriate read, write, update, create, and delete permissions for the user specified below have already been granted.
user = sarah
This is the user under which to connect to the database.
passwd = lianne
This is the password to use for the user.
This is used for various information related to a web interface.
css = cpan.css
If specified, this will be used as the css file when generating the html files from the pod files of the distribution. It is assumed this file appears directly beneath the
up_img = up.gif
If specified, this will be used as an image in the generated html files linking each section to the top-most index. If not specifed, the text __top will be used. It is assumed this image appears directly beneath the
dist_info = http://cpan.uwinnipeg.ca/dist/
If specified, this will be used to provide a link on the generated html pages to information on the distribution. The name of the distribution will be added at the end of the link (for example, using http://cpan.uwinnipeg.ca/dist/ will result, for the libnet distribution, in a link to http://cpan.uwinnipeg.ca/dist/libnet.
tt2 = /usr/local/tt2
This gives the location of the Template-Toolkit pages used to provide a web interface. This is used to place a file mirror_list (extracted from $CPAN/MIRRORED.BY) containing a list of CPAN mirrors.
geoip = /usr/local/share/geoip/cpan.txt
If the module
Apache::GeoIPis used to provide a redirection service to a nearby CPAN mirror based on the location of origin, this file will be created to provide the necessary country of origins of the CPAN mirrors.
will start the indexing procedure. Various messages detailing the progress will written to STDOUT, which by default will be captured into a file cpan_search_log.dddddddddd, where the extension is the
time that the method was invoked. Passing
index an argument of
log => log_file will save these messages into log_file. Error messages are not captured, and will appear in STDERR.
The steps of the indexing procedure are as follows.
fetch index data
no_mirroroption is specified, the necessary CPAN index files $CPAN/MIRRORED.BY, $CPAN/indices/ls-lR.gz, $CPAN/authors/01mailrc.txt.gz, $CPAN/modules/02packages.details.txt.gz, and $CPAN/modules/03modlist.data.gz will be fetched from the CPAN mirror specified by the
$cpanvariable at the beginning of CPAN::Search::Lite::Index. If you are using this option, it is recommended to use the same CPAN mirror with subsequent updates, to ensure consistency of the database. As well, the information on the locations of the CPAN mirrors used for Template-Toolkit and GeoIP is written.
get index information
Information from the CPAN indices and, if desired, the ppm repositories is extracted. This is done through CPAN::Search::Lite::Info (for the CPAN indices) and CPAN::Search::Lite::PPM (for the ppm repositories).
get state information
setupargument within the
newmethod of CPAN::Search::Lite::Index is specified, this will get information on the state of the database through CPAN::Search::Lite::State. A comparision is then made between this information and that gathered from the CPAN indices, and if there's a discrepency in some items, those items are marked for either insertion, updating, or deletion, as appropriate.
no_mirroroption is specified, this will extract, through CPAN::Search::Lite::Extract, the available pod sections of files of a distribution, placing them under a subdirectory
dist_name(corrsponding to the name of the distribution) under the specified
pod_rootin the configuration file.
pod2htmlis then run on them, with the results placed under
html_root. Also, a README, Changes, INSTALL, and META.yml file, if present, will be copied over into
pod_root. Finally, information on the prerequisites of the distribution, and distribution and module descriptions, if available and needed, is extracted.
populate the database
At this stage the gathered information is used to populate the database, through CPAN::Search::Lite::Populate, either inserting new items, updating existing ones, or deleting obsolete items.
As well, unless the
no_mirroroption is specified, the html files created under
html_rootwill be edited to adjust the links to module files. This is necessary because when the html files are created no cache is used (in order to maintain consistency between updates), and consequently links to packages outside of a given package may be incorrect. This is fixed by querying the database to see what module documentation is actually present, and then adjusting the links in the html files accordingly (or removing a link if the indicated documentation is missing).
CPAN::Search::Lite::Info, CPAN::Search::Lite::PPM, CPAN::Search::Lite::State, CPAN::Search::Lite::Extract, CPAN::Search::Lite::Populate, and CPAN::Search::Lite::Util. Development takes place on the CPAN-Search-Lite project at http://sourceforge.net/projects/cpan-search/.
This software is copyright 2004 by Randy Kobes <email@example.com>. Use and redistribution are under the same terms as Perl itself.