/ 
App-Actions-0.001.0
/ App::Actions::Runner::Parallel

NAME

App::Actions::Runner::Parallel - run the jobs of one matrix beside each other

SYNOPSIS

my $parallel = App::Actions::Runner::Parallel->new(
	capacity => 4,
	temp_path => $temp_path,
	run_job => sub { my ($entry) = @_; ...; return $result },
	on_result => sub { my ($result) = @_; ... },
);

my $outcomes = $parallel->run(\@plans, $fail_fast);
if (!$outcomes) { ... run them one at a time ... }

Count the failures, and say where the logs are:

my $outcomes = $parallel->run(\@plans, 0) or return;
my @failed = grep { $_->{result}{result} eq 'failure' } grep { $_->{started} } @{$outcomes};
print scalar(@failed), " jobs failed\n";
print "$_->{name}: $_->{path}\n" for @{ $parallel->logs };

DESCRIPTION

In short: this module runs several jobs of one matrix at the same time, each in its own process, and shows their output as it happens with the job name in front of every line. It is a part of App::Actions::Runner; you rarely use it on its own.

Each job runs in a child process of its own, with its own runner directory, its output written to a log, and its result written to a file the parent reads when it exits. The parent shows each log as it grows, a line at a time, with the job's name in front of every line - [test (n=one)] - so the lines of jobs working at the same moment can be told apart. With fail-fast, a failed job stops further jobs from starting; the ones already running finish.

METHODS

new

ARGUMENTS

  • capacity - how many jobs may run at once.

  • temp_path - the run's temporary directory; each job gets job-N under it, with job-N.log and job-N.json beside it.

  • run_job - a code reference that runs one job in the current process and returns its result hash; it is called in the child.

  • on_result - optional; called in the parent with each result read back, so the caller can take what the child learned (the images it pulled).

  • quiet - optional; show nothing of the jobs' logs.

RETURNS

An App::Actions::Runner::Parallel object.

SIDE EFFECTS

None.

EXAMPLE

See the "SYNOPSIS".

API SPECIFICATION

INPUT

{
	capacity  => { type => 'integer', min => 1 },
	temp_path => { type => 'string', min => 1 },
	run_job   => { type => 'coderef' },
	on_result => { type => 'coderef', optional => 1 },
	quiet     => { type => 'boolean', optional => 1 },
}

OUTPUT

{ type => 'object', isa => 'App::Actions::Runner::Parallel' }

MESSAGES

+---------------------------+------------------------------+-----------------------------+
| Message                   | Meaning                      | Resolution                  |
+---------------------------+------------------------------+-----------------------------+
| Invalid arguments to      | An argument is missing or    | Pass the documented         |
| ...Parallel::new: REASON  | has the wrong type           | arguments                   |
+---------------------------+------------------------------+-----------------------------+

run

my $outcomes = $parallel->run(\@plans, $fail_fast);

Runs one matrix group's jobs beside each other.

ARGUMENTS

  • \@plans - the runnable plans, each with display_name and max_parallel, in the order the jobs should start.

  • $fail_fast - whether a failure stops the jobs not yet started.

RETURNS

undef when the group is better run one at a time - a single job, a capacity below two once max-parallel is applied, or a host without a reliable fork() - so the caller can fall back to its simple loop. Otherwise an array reference of { plan, started, result }, one per plan in plan order.

SIDE EFFECTS

Forks; prints the jobs' output, prefixed; writes their logs and results under the temporary directory. Kills the jobs still running if one could not report or the parent is interrupted, and then dies.

EXAMPLE

my $outcomes = $parallel->run(\@plans, 1) or return $self->_run_serially(\@plans);
for my $outcome (@{$outcomes}) {
	print "$outcome->{plan}{display_name}: ", ($outcome->{started} ? $outcome->{result}{result} : 'skipped'), "\n";
}

API SPECIFICATION

INPUT

{
	plans     => { type => 'arrayref' },
	fail_fast => { type => 'boolean', optional => 1 },
}

OUTPUT

{ type => 'arrayref', optional => 1 }

MESSAGES

+-----------------------------+------------------------------+-----------------------------+
| Message                     | Meaning                      | Resolution                  |
+-----------------------------+------------------------------+-----------------------------+
| job 'X' did not run         | The job's process died       | Read its log, shown above   |
|                             | before it reported a result  |                             |
| action-run: E (in a log)    | The job itself died          | Read the message            |
+-----------------------------+------------------------------+-----------------------------+

PSEUDOCODE

work out the capacity: the configured one, capped by max-parallel and
	by the number of plans
give up (return undef) for fewer than two plans, a capacity below two,
	or Windows
start jobs until the capacity is reached
until no job is running:
	show what each running job has written since the last look
	when a job exits, read its result and show the rest of its log
	a job with no result stops new starts and is an error
	a failed job under fail-fast stops new starts
	otherwise start more jobs
on an error, kill the running jobs and die with it
return the outcomes

logs

Returns an array reference of the logs the jobs wrote: { slot, name, file, path }, where file is a name that says which job the log belongs to once the numbered directory is gone.

EXAMPLE

print "$_->{name}: $_->{path}\n" for @{ $parallel->logs };

API SPECIFICATION

INPUT

{}

OUTPUT

{ type => 'arrayref' }

MESSAGES

None.

LIMITATIONS

  • Windows has no reliable fork(), so its jobs run one at a time.

  • A run that keeps its containers, a dry run and an explain run are not run in parallel; the caller decides that, because only it knows why.

  • The logs are polled rather than watched, twenty times a second.

  • A killed job's containers are not removed: only the process that started them knew their names.

COMMON PITFALLS

  • run may return undef. That is not an error: it means "run these one at a time". Always have a fallback.

  • run_job runs in a child process. Changes it makes to variables are not seen by the parent. Return everything the parent needs in the result.

  • max-parallel wins. The workflow's strategy.max-parallel can lower capacity, never raise it.

  • Fail-fast does not stop running jobs. It only stops jobs that have not started yet.

  • Output lines can be delayed. The logs are read 20 times a second, and a line is shown only when it is complete.

  • A killed parent leaves containers. If the parent process is killed with SIGKILL, the children's containers are not removed. Use docker ps --filter name=action-run- to find them.

ENCODING

Job output is copied from the log files to your terminal as bytes, unchanged, so it may contain any characters the job printed. Windows line ends (\r\n) are shown as \n. Job names in the line prefix are Perl character strings and may contain non-ASCII characters and emoji; log file names are made from them with unsafe characters replaced, so they are ASCII.

AUTHOR

Nigel Horne, njh@nigelhorne.com

FORMAL SPECIFICATION

This section describes each routine in the Z notation, a precise mathematical language for describing programs. You do not need to read it to use the module: everything here is also said in plain words in the sections above. It is for readers who want to check exactly what a routine promises.

How to read it: a box names a routine. The lines above the middle rule declare the values it works with. A name ending in ? is an input, and a name ending in ! is an output. ΞState means the routine changes nothing, and ΔState means it can change the state. The lines below the rule are the conditions that are always true when the routine returns.

new

┌─ Parallel ────────────────────────────────────────
│ capacity : ℕ₁
│ slots    : ℕ          -- jobs started so far, numbering their files
│ logs     : seq LOG
├───────────────────────────────────────────────────
│ #logs = slots
└───────────────────────────────────────────────────

run

┌─ RunParallel ─────────────────────────────────────
│ ΔParallel
│ plans?    : seq PLAN
│ failFast? : 𝔹
│ out!      : optional seq OUTCOME
├───────────────────────────────────────────────────
│ #plans? < 2 ∨ cap < 2 ∨ os = MSWin32 ⇒ out! = ⊥
│ otherwise #out! = #plans? ∧
│     (∀ t : TIME • #{o : ran out! | running(o, t)} ≤ cap) ∧
│     (failFast? ∧ (∃ o : ran out! • failed(o)) ⇒ no plan after the
│         first failure is started once it is seen)
│ where cap = min(capacity, maxParallel(head plans?), #plans?)
└───────────────────────────────────────────────────

logs

┌─ Logs ────────────────────────────────────────────
│ ΞParallel
│ l! : seq LOG
├───────────────────────────────────────────────────
│ l! = logs
└───────────────────────────────────────────────────

STATE DIAGRAM

The parent process goes through these states for one call to "run"; each child goes through the states on the right:

  run(\@plans, $fail_fast)
        |
        v
  one plan, capacity < 2, or no reliable fork()? --yes--> return undef
        | no                                      (caller runs them serially)
        v
+------------------+                       Child (one per job):
| STARTING         |  fork while fewer      +--------------------+
|                  |  than capacity run --->| RUNNING            |
+------------------+                        | own runner folder; |
        |                                   | output -> log file |
        v                                   +--------------------+
+------------------+                                 |
| WATCHING         |  every 1/20 s: new complete     | run_job returns
|                  |  log lines printed as           v
|                  |  "[job name] line"      +--------------------+
+------------------+                         | RESULT WRITTEN     |
   |       |      |                          | to a file; exit    |
   |       |      |                          +--------------------+
   |       |      | a child exits
   |       |      v
   |       |  +------------------+  result file missing or unreadable:
   |       |  | COLLECTING       |  kill the others -> die
   |       |  | on_result(...)   |
   |       |  +------------------+
   |       |      |
   |       |      | failure and fail-fast: start no more jobs
   |       |      | otherwise: back to STARTING if plans are left
   |       |
   |       | SIGINT / SIGTERM: kill the children -> die
   |
   | all children done
   v
  return [ { plan, started, result }, ... ]   (logs() lists the log files)

LICENSE AND COPYRIGHT

Copyright 2026 Nigel Horne.

This program is released under the GNU General Public License, version 2. See the LICENSE file that comes with this distribution. If you use it, please let me know.