Ted Pedersen
and 1 contributors

NAME

huge-merge.pl - Merge the results of multiple huge-sort generated files into a single sorted file.

SYNOPSIS

count.pl --tokenlist input1.out input1 count.pl --tokenlist input2.out input2

huge-sort.pl input1.out huge-sort.pl input2.out

mv input1.out-sorted output-directory mv input2.out-sorted output-directory

huge-merge.pl output-directory

DESCRIPTION

Combine the sorted bigram files generated by huge-sort.pl efficiently.

This program is used internally by huge-count.pl.

USGAE

huge-merge.pl [OPTIONS] SOURCEDIR

INPUT

Required Arguments:

SOURCEDIR

Input to huge-merge.pl should be a single flat directory containing multiple plain text files generated by huge-sort.pl. The result file, merge.* (* is a number, the final result file has the maximum number), is in the source directory.

Optional Arguments:

--keep

Switches ON the --keep option will keep all the intermediate merging files.

--remove L

Bigrams with counts less than L in the entire SOURCE data are removed from the sample. The counts of the removed bigrams are not counted in any marginal totals. This has same effect as count.pl's --remove option.

--uremove L

Bigrams with counts more than L in the entire SOURCE data are removed from the sample. The counts of the removed bigrams are not counted in any marginal totals. This has same effect as count.pl's --uremove option.

--frequency F

Bigrams with counts less than F in the entire SOURCE are not displayed. The counts of the skipped bigrams ARE counted in the marginal totals. In other words, --frequency in huge-count.pl has same effect as the count.pl's --frequency option.

--ufrequency F

Bigrams with counts more than F in the entire SOURCE are not displayed. The counts of the skipped bigrams ARE counted in the marginal totals. In other words, --frequency in huge-count.pl has same effect as the count.pl's --ufrequency option.

Other Options:

--help

Displays the help information.

--version

Displays the version information.

AUTHOR

Ying Liu, University of Minnesota, Twin Cities. liux0395 at umn.edu

Ted Pedersen, University of Minnesota, Duluth. tpederse at umn.edu

COPYRIGHT

Copyright (C) 2009-2010, Ying Liu and Ted Pedersen

This program is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 2 of the License, or (at your option) any later version. This program is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.

You should have received a copy of the GNU General Public License along with this program; if not, write to the Free Software Foundation, Inc., 59 Temple Place - Suite 330, Boston, MA 02111-1307, USA.