Nicolas Franck
and 1 contributors


Catmandu::Importer::MediaWiki - Catmandu importer that imports pages from mediawiki


This importer uses the query api from mediawiki to get a list of pages that match certain requirements.

It retrieves a list of pages and their content by using the generators from mediawiki:

The default generator is 'allpages'.

The list could also be retrieved with the module 'list':

But this module 'list' is very limited. It retrieves a list of pages with a limited set of attributes (pageid, ns and title).

The module 'properties' on the other hand lets you add properties:

But the selecting parameters (titles, pageids and revids) are too specific to execute a query in one call. One should execute a list query, and then use the pageids to feed them to the submodule 'properties'.

To execute a query, and add properties to the pages in one call can be accomplished by use of generators.

These parameters are set automatically, and cannot be overwritten:

action = "query" indexpageids = 1 generator = <generate> format = "json"

Additional parameters can be set in the constructor argument 'args'. Arguments for a generator origin from the list module with the same name, but must be prepended with 'g'.



type: string

explanation: type of generator to use. For a complete list, see because Catmandu::Iterable already defines 'generator', this parameter has been renamed to 'generate'.

default: 'allpages'.


type: hash

explanation: extra arguments. These arguments are merged with the defaults.


        prop => "revisions",
        rvprop => "ids|flags|timestamp|user|comment|size|content",
        gaplimit => 100,
        gapfilterredir => "nonredirects"

which means:

    prop             add revisions to the list of page attributes
    rvprop           specific properties for the list of revisions
    gaplimit         limit for generator 'allpages' (every 'generator' has its own limit).
    gapfilterredir   filter out redirect pages

type: string

explanation: login name. Only used when both lgname and lgpassword are set.


type: string

explanation: login password. Only used when both lgname and lgpassword are set.


    use Catmandu::Sane;
    use Catmandu::Importer::MediaWiki;

    binmode STDOUT,":utf8";

    my $importer = Catmandu::Importer::MediaWiki->new(
        url => "",
        generate => "allpages",
        args => {
            prop => "revisions",
            rvprop => "ids|flags|timestamp|user|comment|size|content",
            gaplimit => 100,
            gapprefix => "plato",
            gapfilterredir => "nonredirects"
        my $r = shift;
        my $content = $r->{revisions}->[0]->{"*"};
        say $r->{title};


Nicolas Franck <nicolas.franck at>


Catmandu, MediaWiki::API