ppll
ppll [OPTIONS] COMMAND [ARGS] ppll [OPTIONS] -c COMMAND [PARAMETERS] ppll --help ppll --version
In the first form, all arguments after COMMAND are considered arguments to COMMAND, not options to ppll, even if they begin with a hyphen.
ppll is a tool to control the execution of commands. It can run commands in parallel, construct commands from lists of parameters, and more.
It handles the output of commands and can prefix lines with which command produced it, print timestamps, etc.
ppll has functionality similar to xargs and parallel.
xargs
parallel
‘Parameters’ is what makes ppll run multiple commands. In one way or another ppll can be given a number of parameters, and it will create and run one ‘job’ for each parameter.
If no parameter-related option is used ppll will read the parameters from STDIN (if it’s not a TTY). Ordinarily, each line read becomes one parameter, but there are options to control this.
The other important input to ppll is the ‘command’. The command-line arguments after all options make up the command and its arguments.
Each job will execute the command. All occurrences of {} in any part of the command (and its arguments) will be replaced with the job’s parameter. If there are no such occurrences the parameter will be appended as an additional argument.
{}
For example:
ppll echo {} <<EOF cow pig sheep EOF
Will create jobs that execute echo cow, echo pig and so on. An other way to do the same thing would be:
echo cow
echo pig
ppll echo <<EOF cow pig sheep EOF
{} becomes useful when it’s not supposed to be the last argument, e.g:
ppll cp {} farm/ <<EOF cow pig sheep EOF
Or when it’s supposed to be just a part of an argument, or show up more than once, e.g:
ppll curl -o photo-of-a-{}.jpg http://example.com/img/{}.jpg <<EOF cow pig sheep EOF
The above would run curl -o photo-of-a-cow.jpg http://example.com/img/cow.jpg, etc.
curl -o photo-of-a-cow.jpg http://example.com/img/cow.jpg
The default is to run several such jobs in parallel. Once a job is finished another one is started, until there are no more jobs to run.
The number of jobs run at the same time is controlled by the --jobs option. The default is one per CPU (but not necessarily one on each CPU).
If a job fails (exits with a code other than 0), no more jobs will be started. ppll attempts to stop other running jobs.
The --force option tells ppll to not care about failed jobs and always keep starting new ones. Any failed jobs are still noted and will still affect ppll’s own exit code.
This feature deserves its own section.
It can be used to parallelise heavy tasks, while also making sure certain tasks are run in a particular order.
Each line of input will create a ‘group’ of parameters, that will be run before the next group starts. Each line is split up into individual parameters (using the IFS environment variable), that will be run at the same time.
IFS
An example:
ppll --slpf wake-up.sh <<EOF rooster hens chickens cows pigs sheep cat EOF
The above would run wake-up.sh first for rooster. Then it would run it for hens and chickens, at the same time, but after the rooster job has completed. Then cows, pigs and sheep in parallel, and so on.
wake-up.sh
rooster
hens
chickens
cows
pigs
sheep
Print (this) help and exit.
Print ppll’s version and exit.
Run COMMAND as a shell command with each parameter passed as an argument. This allows for shell constructs, such as piping, redirecting, etc.
If used then the rest of the command-line arguments are used as parameters, instead of a command.
More or less the same as:
ppll $SHELL -c <COMMAND> -
($SHELL defaults to /bin/sh.)
$SHELL
/bin/sh
Use DELIMITER, as a regex, to split lines into fields. Implies --fields.
Include empty parameters. These are otherwise discarded.
Alias for --mode fields.
--mode fields
Keep starting new jobs even if a job has failed.
Run (up to) this many jobs in parallel.
Alias for --mode lines.
--mode lines
How input should be turned into parameters. MODE is one of:
auto
If there’s only one line of input, behave as if fields, otherwise as if lines.
fields
lines
Split lines into fields and use each field as a parameter. Splitting is done using a delimiter set with --delimiter, or the IFS environment variable.
Use each line as a parameter.
slpf
Split each line into parameters. All parameters from a single line will be run in parallel, but lines will be run serially.
See also: --fields, --lines, --serial-lines-parallel-fields
Shuffle the parameters.
Use STRING instead of {} as a placeholder for the parameter.
Use the parameters in reverse order.
Use SPEC as parameters.
SPEC is in one of the forms:
N..M
Parameters are from N to M, inclusive.
M
Parameters are from 1 to M, inclusive.
Alias for --mode slpf.
--mode slpf
Turn on or off ANSI colours in the output.
The default is to enable colours if STDOUT is a TTY.
Prefix (the default), or don’t prefix, jobs’ STDOUT and STDERR output with > or ≫, respectively.
>
≫
Print less. Can be repeated for less and less.
Print (the default if there’s more than one parameter), or don’t print, a summary of how many jobs ran and which of them failed.
Format timestamps using FORMAT. See DateTime::strftime.
The default is %T.%3N.
%T.%3N
Implies --timestamps.
Prefix lines from jobs’ output with a timestamp. The time printed will be the approximate time the line was printed by the job.
Print more. Can be repeated for more and more.
ppll uses some environment variables in some cases:
COLUMNS
If set then used when printing things across the whole width of the terminal. If not set then ppll will try to work out the terminal width in other ways, falling back to 80.
Used for splitting lines into fields, e.g. with --fields or --serial-lines-parallel-fields.
--fields
--serial-lines-parallel-fields
SHELL
Used with --commmand (-c).
--commmand
-c
ppll -t make
Run a long-running command, e.g. make, and prefix the output with timestamps.
make
ppll -j 10 -s 1..1000 gobbledygook --with-rigmarole --number
For each number from 1 to 1,000 (inclusive), run the gobbledygook command with some arguments. Add the a number as the last argument, e.g. gobbledygook --with-rigmarole --number 42.
gobbledygook
gobbledygook --with-rigmarole --number 42
Run no more than 10 such commands at a time.
Download three files in parallel. Note the use of {} to use each parameter in two places in each command.
POSIX, and thus available almost everywhere, and can run multiple commands based on a list of parameters, much like ppll.
Also runs commands in parallel, but with an interface quite different to ppll’s.
ts
moreutils contains a parallel tool similar yet different to GNU Parallel.
It also has a separate tool for adding timestamps to output.
Theo -q Willows, <theo@willows.se>
<theo@willows.se>
Please report any bugs or feature requests through the web interface at https://gitlab.com/munkei-software/ppll/issues.
You can find documentation for this module with the perldoc command.
perldoc ppll
Or:
ppll --help
You can also look for information at:
MetaCPAN
https://metacpan.org/pod/App::ppll
GitLab
https://gitlab.com/munkei-software/ppll
Copyright 2019 Theo Willows.
This program is free software; you can redistribute it and/or modify it under the terms of either: the GNU General Public License as published by the Free Software Foundation; or the Artistic License.
See http://dev.perl.org/licenses/ for more information.
To install App::ppll, copy and paste the appropriate command in to your terminal.
cpanm
cpanm App::ppll
CPAN shell
perl -MCPAN -e shell install App::ppll
For more information on module installation, please visit the detailed CPAN module installation guide.