NAME

App::Actions::Runner::Images - work out, pull and track the images a run needs

SYNOPSIS

my $images = App::Actions::Runner::Images->new(
	docker => $docker,
	expression => $expression,
	workflow => $workflow,
	project_dir => $project_dir,
	context => sub { $runner->preflight_context(shift) },
);

my $prepared = $images->prepare($instances);
die $images->failure_message($prepared->{failures}[0]) if @{ $prepared->{failures} };

List the images a run will need, without pulling anything:

for my $entry (@{ $images->required($instances) }) {
	print defined $entry->{image}
		? "$entry->{image}  (for $entry->{source})\n"
		: "deferred: $entry->{expression}\n";
}

Make sure one image is there, then record what was downloaded:

$images->ensure('perl:5.40');
$images->remember($cache_file);    # for "action-run cache"

DESCRIPTION

In short: this module finds every Docker image a run will need, downloads the missing ones before any job starts, and remembers what it downloaded. It is a part of App::Actions::Runner; you rarely use it on its own.

Every image a run will use - each job's container, each service, and each docker:// action, including those inside local composite actions - is named before the first container starts, so that a mistyped image fails the run at once rather than halfway through a job. An image whose name depends on a value that does not exist yet (a step's output) is deferred: the record says what could not be resolved, and the step that needs it prepares it when it runs.

The object remembers which images it pulled, so that --pull refreshes each image once per run, a run told not to keep its images removes what it fetched, and a run that keeps them records them for action-run cache.

METHODS

new

Creates the image preparer for one run.

ARGUMENTS

docker - the App::Actions::Docker object, or one with the same methods.
expression - the App::Actions::Expression evaluator.
workflow - the App::Actions::Workflow.
project_dir - the project's absolute path, where local actions are read.
context - a code reference that returns the expression context known before the run for a job instance.
image, perl_image - optional; the default image and the template for a setup-perl job.
dry_run, verbose, quiet, pull, no_pull - optional flags.
pull_concurrency - optional; how many pulls run at once (default 4).

RETURNS

An App::Actions::Runner::Images object.

SIDE EFFECTS

None.

EXAMPLE

my $images = App::Actions::Runner::Images->new(
	docker => App::Actions::Docker->new,
	expression => App::Actions::Expression->new,
	workflow => App::Actions::Workflow->load('.github/workflows/ci.yml'),
	project_dir => '/home/me/project',
	context => sub { { matrix => $_[0]{matrix} } },
	pull => 1,
);

API SPECIFICATION

INPUT

{
	docker           => { type => 'object', can => 'image_exists' },
	expression       => { type => 'object', can => 'interpolate' },
	workflow         => { type => 'object', can => 'container_image' },
	project_dir      => { type => 'string', min => 1 },
	context          => { type => 'coderef' },
	image            => { type => 'string', optional => 1 },
	perl_image       => { type => 'string', optional => 1 },
	dry_run          => { type => 'boolean', optional => 1 },
	verbose          => { type => 'boolean', optional => 1 },
	quiet            => { type => 'boolean', optional => 1 },
	pull             => { type => 'boolean', optional => 1 },
	no_pull          => { type => 'boolean', optional => 1 },
	pull_concurrency => { type => 'integer', min => 1, optional => 1 },
}

OUTPUT

{ type => 'object', isa => 'App::Actions::Runner::Images' }

MESSAGES

+---------------------------+------------------------------+-----------------------------+
| Message                   | Meaning                      | Resolution                  |
+---------------------------+------------------------------+-----------------------------+
| Invalid arguments to      | A collaborator is missing or | Pass the documented objects |
| ...Images::new: REASON    | an option has the wrong type |                             |
+---------------------------+------------------------------+-----------------------------+

concurrency

Returns the number of pulls allowed at once: the configured number, but never more than one on Windows, where the pulls run one after another. This is the number handed to the puller, and so the one the preflight reports.

EXAMPLE

print $images->concurrency;    # 4

API SPECIFICATION

INPUT

{}

OUTPUT

{ type => 'integer', min => 1 }

MESSAGES

None.

required

my $entries = $images->required($instances);

Lists every image the jobs will use, in the order the run needs them, each once. An entry is { image, source, kind }; one the values known before the run cannot name is { image => undef, expression, source, error }. Jobs whose runner is not Linux are left out, because they will not run.

EXAMPLE

for my $entry (@{ $images->required($instances) }) {
	print $entry->{image} // $entry->{expression}, " for $entry->{source}\n";
}

API SPECIFICATION

INPUT

{ instances => { type => 'arrayref' } }

OUTPUT

{ type => 'arrayref' }

MESSAGES

None; an expression that fails to resolve is reported in its entry.

PSEUDOCODE

for each instance whose runner is Linux:
	build the context known before the run
	for each image expression the job uses:
		try to resolve it strictly
		if it resolved to a name not seen before, add { image, source, kind }
		if it did not resolve, add { expression, source, error } once per source

job_expressions

my $entries = $images->job_expressions($job, $runs_on, $context);

Lists what one job needs, as expressions rather than resolved names, so that each caller can resolve them with the values it has: the preflight with what is known before the run, the job itself with its full context. An entry is { expression, source, kind }, where kind is job, service:ID, or absent for an action.

EXAMPLE

my $entries = $images->job_expressions($job, 'ubuntu-latest', $context);

API SPECIFICATION

INPUT

{
	job     => { type => 'hashref' },
	runs_on => { type => 'string' },
	context => { type => 'hashref' },
}

OUTPUT

{ type => 'arrayref' }

MESSAGES

None; a local action that cannot be read contributes no images.

PSEUDOCODE

add the job's container image, if the workflow can name one
add each service's image, sorted by service id
for each step, add its docker:// image, descending into local composite actions

prepare

my $prepared = $images->prepare($instances, quiet => 1);

Pulls or verifies every image the run will need. The presence checks run in turn, because an image that is already there needs no pull; the pulls run together, up to "concurrency" at a time.

ARGUMENTS

$instances - the selected job instances.
quiet - optional; print nothing and discard the pull output.

RETURNS

{ records => [...], failures => [...] }. A record is { image, source, status, error } where status is present, pulled, failed or deferred; a deferred record carries the expression instead of an image. The caller decides whether a failure stops the run or is only reported.

SIDE EFFECTS

Pulls images, prints progress unless quiet, and remembers what it pulled.

EXAMPLE

my $prepared = $images->prepare($instances);
for my $record (@{ $prepared->{records} }) {
	print "$record->{image}: $record->{status}\n" if defined $record->{image};
}

API SPECIFICATION

INPUT

{
	instances => { type => 'arrayref' },
	quiet     => { type => 'boolean', optional => 1 },
}

OUTPUT

{ type => 'hashref' }

MESSAGES

+-----------------------------+------------------------------+-----------------------------+
| Message (in a record)       | Meaning                      | Resolution                  |
+-----------------------------+------------------------------+-----------------------------+
| Docker image 'X' is not     | --no-pull was given and the  | Pull it, or drop --no-pull  |
| available locally (--no-pull)| image is absent             |                             |
| Failed to pull Docker image | The pull exited non-zero     | Check the name, the registry|
| 'X' (exit N)                |                              | and the Docker login        |
+-----------------------------+------------------------------+-----------------------------+

PSEUDOCODE

for each required image:
	an unnamed one becomes a deferred record
	a present one (without --pull) becomes a present record
	with --no-pull, an absent one becomes a failed record
	otherwise it becomes pending
if anything failed, return now, before downloading anything
pull the pending images together
mark each pending record pulled or failed, remembering what arrived
return the records and the failed ones

prepare_job

my $job_images = $images->prepare_job($job, $context);

The final check of one job's images, made with the values the job itself has. An image built from another job's outputs was deferred by the preflight; here it is resolved and prepared, while the job still has no network and no containers, so a failure ends the run before the job starts.

ARGUMENTS

$job - the job instance.
$context - its full expression context.

RETURNS

{ image => $job_image, deferred => [ { expression, source, error }, ... ] }. The deferred entries are the images a step decides, which cannot be known until that step has run.

SIDE EFFECTS

Pulls images that are not present.

EXAMPLE

my $job_images = $images->prepare_job($job, $context);
print "image: $job_images->{image}\n";

API SPECIFICATION

INPUT

{
	job     => { type => 'hashref' },
	context => { type => 'hashref' },
}

OUTPUT

{ type => 'hashref' }

MESSAGES

+-----------------------------+------------------------------+-----------------------------+
| Message                     | Meaning                      | Resolution                  |
+-----------------------------+------------------------------+-----------------------------+
| cannot resolve the image    | A job or service image's     | Fix the expression, or the  |
| 'X', needed by S: E         | expression has no value      | output it refers to         |
| cannot prepare the image    | The image could not be       | Check the name and registry |
| 'X', needed by S: E         | pulled                       |                             |
+-----------------------------+------------------------------+-----------------------------+

PSEUDOCODE

for each image expression the job uses:
	resolve it strictly with the job's context
	if it has no value:
		a job or service image is an error the run reports now
		an action image is deferred to its step
	remember the job's own image
	ensure each distinct image is present, failing the run if one cannot be

probe

my $image = $images->probe($instances);

Chooses the image the environment check probes with: the one the first runnable job will use, which costs no extra pull and exercises the image the run needs. Falls back to the configured default image, then to alpine:3.20.

EXAMPLE

my $check = App::Actions::Check->new(image => $images->probe($instances));

API SPECIFICATION

INPUT

{ instances => { type => 'arrayref' } }

OUTPUT

{ type => 'string', min => 1 }

MESSAGES

None.

ensure

$images->ensure('perl:5.40');

Makes sure an image is present, pulling it if it is absent or if --pull asked for fresh images. --pull refreshes an image once per run: the preflight has prepared every image already, so a job, a second matrix job or a second step that names the same image would otherwise download it again.

EXAMPLE

$images->ensure($image);
$docker->run('--rm', $image);

API SPECIFICATION

INPUT

{ image => { type => 'string', min => 1 } }

OUTPUT

{ type => 'boolean' }

MESSAGES

+-----------------------------+------------------------------+-----------------------------+
| Message                     | Meaning                      | Resolution                  |
+-----------------------------+------------------------------+-----------------------------+
| Docker image 'X' is not     | --no-pull was given and the  | Pull it, or drop --no-pull  |
| available locally (--no-pull)| image is absent             |                             |
+-----------------------------+------------------------------+-----------------------------+

note_pulled

$images->note_pulled(@images);

Records images this run downloaded somewhere else: the probe the environment check ran, or a step in a job that ran in another process. A run told not to keep its images removes them too. Returns the object.

EXAMPLE

$images->note_pulled(@{ $result->{pulled} || [] });

API SPECIFICATION

INPUT

{ images => { type => 'arrayref' } }

OUTPUT

{ type => 'object', isa => 'App::Actions::Runner::Images' }

MESSAGES

None.

pulled

Returns a sorted array reference of the images this run downloaded that were not there before.

EXAMPLE

$docker->image_remove($_) for @{ $images->pulled };

API SPECIFICATION

INPUT

{}

OUTPUT

{ type => 'arrayref' }

MESSAGES

None.

remember

$images->remember($cache_file);

Records the images this run pulled in the cache file that action-run cache reads, because the images a run keeps are the next run's cache. A failure to write the record does not fail a run that otherwise worked: the images are still on disk, and the next run records them again.

ARGUMENTS

$cache_file - optional; the record to update, as for App::Actions::Cache.

RETURNS

The object.

SIDE EFFECTS

Writes the cache record; warns (when verbose) if it cannot.

EXAMPLE

$images->remember($ENV{ACTION_RUN_CACHE_FILE});

API SPECIFICATION

INPUT

{ cache_file => { type => 'string', optional => 1 } }

OUTPUT

{ type => 'object', isa => 'App::Actions::Runner::Images' }

MESSAGES

+-----------------------------+------------------------------+-----------------------------+
| Message                     | Meaning                      | Resolution                  |
+-----------------------------+------------------------------+-----------------------------+
| action-run: could not update| The cache record could not   | Check the cache directory's |
| the image cache: E          | be written (verbose only)    | permissions                 |
+-----------------------------+------------------------------+-----------------------------+

failure_message

die $images->failure_message($record), "\n";

Returns the sentence a failed image record ends a run with: the image, what needed it, and what went wrong.

EXAMPLE

my $prepared = $images->prepare($instances);
print $images->failure_message($_), "\n" for @{ $prepared->{failures} };

API SPECIFICATION

INPUT

{ record => { type => 'hashref' } }

OUTPUT

{ type => 'string' }

MESSAGES

The text it returns is runner.image.unprepared: "cannot prepare the image 'X', needed by S: E".

LIMITATIONS

An image named by a step's output cannot be known until the step has run, so it is pulled by that step, inside the job, after its containers have started. A bad name there fails the step rather than the preflight.
Images are compared by name. perl:5.40 and docker.io/library/perl:5.40 are the same image to Docker but two entries here, so the second is checked (and, with --pull, pulled) again.
The walk into local composite actions reads each action's metadata again for every job that uses it.

COMMON PITFALLS

Some images cannot be known in advance. An image whose name uses a value that a step produces is deferred: "required" and "prepare" list its expression instead of a name, and "prepare_job" pulls it when the job starts.
Image names are compared as text. perl:5.40 and docker.io/library/perl:5.40 count as two images.
On Windows, pulls run one at a time, whatever pull_concurrency says. "concurrency" tells you the real number.
With pull, each image is pulled once per run, not once per job. Call "ensure" freely.
Images pulled in another process are not known until you call "note_pulled" with them (the parallel runner does this for its children).
"remember" never fails the run. If the cache file cannot be written, a warning is printed and the run goes on.

ENCODING

Image names are ASCII, as Docker requires; a name built from workflow values containing other characters is refused by Docker when it is pulled. Expressions and source descriptions are Perl character strings and may contain any Unicode text. The cache file is written as UTF-8 JSON (see App::Actions::Cache).

AUTHOR

Nigel Horne, njh@nigelhorne.com

FORMAL SPECIFICATION

This section describes each routine in the Z notation, a precise mathematical language for describing programs. You do not need to read it to use the module: everything here is also said in plain words in the sections above. It is for readers who want to check exactly what a routine promises.

How to read it: a box names a routine. The lines above the middle rule declare the values it works with. A name ending in ? is an input, and a name ending in ! is an output. ΞState means the routine changes nothing, and ΔState means it can change the state. The lines below the rule are the conditions that are always true when the routine returns.

new

┌─ Images ──────────────────────────────────────────
│ pulled     : ℙ IMAGE     -- images this run downloaded that were absent
│ pulledOnce : ℙ IMAGE     -- images this run has pulled at all
├───────────────────────────────────────────────────
│ pulled ⊆ pulledOnce
└───────────────────────────────────────────────────

┌─ NewImages ───────────────────────────────────────
│ Images'
├───────────────────────────────────────────────────
│ pulled' = ∅ ∧ pulledOnce' = ∅
└───────────────────────────────────────────────────

concurrency

┌─ Concurrency ─────────────────────────────────────
│ ΞImages
│ n! : ℕ₁
├───────────────────────────────────────────────────
│ os = MSWin32 ⇒ n! = 1
│ os ≠ MSWin32 ⇒ n! = pullConcurrency
└───────────────────────────────────────────────────

required

┌─ Required ────────────────────────────────────────
│ ΞImages
│ instances? : seq JOB
│ entries!   : seq ENTRY
├───────────────────────────────────────────────────
│ ∀ e : ran entries! • e.image ≠ ⊥ ∨ e.expression ≠ ⊥
│ ∀ i, j : dom entries! | i ≠ j ∧ entries!(i).image ≠ ⊥ •
│     entries!(i).image ≠ entries!(j).image
└───────────────────────────────────────────────────

job_expressions

┌─ JobExpressions ──────────────────────────────────
│ ΞImages
│ job? : JOB
│ e!   : seq ENTRY
├───────────────────────────────────────────────────
│ e! = jobImage(job?) ⁀ serviceImages(job?) ⁀ ⁀/ (map actionImages job?.steps)
└───────────────────────────────────────────────────

prepare

┌─ Prepare ─────────────────────────────────────────
│ ΔImages
│ instances? : seq JOB
│ records!   : seq RECORD
│ failures!  : seq RECORD
├───────────────────────────────────────────────────
│ failures! = records! ↾ {r | r.status = failed}
│ pulled' = pulled ∪ {r.image | r ∈ ran records! ∧ r.status = pulled ∧ ¬ present(r.image)}
│ pulledOnce' = pulledOnce ∪ {r.image | r ∈ ran records! ∧ r.status = pulled}
└───────────────────────────────────────────────────

prepare_job

┌─ PrepareJob ──────────────────────────────────────
│ ΔImages
│ job?      : JOB
│ image!    : optional IMAGE
│ deferred! : seq ENTRY
├───────────────────────────────────────────────────
│ ∀ e : jobExpressions(job?) | e.kind ∈ {job} ∪ services •
│     resolves(e) ∨ raises Unresolved
│ image! = resolve(e) where e.kind = job
│ ran deferred! = {e : jobExpressions(job?) | ¬ resolves(e) ∧ e.kind = ⊥}
└───────────────────────────────────────────────────

probe

┌─ Probe ───────────────────────────────────────────
│ ΞImages
│ instances? : seq JOB
│ image!     : IMAGE
├───────────────────────────────────────────────────
│ (∃ i : ran instances? • linux(i) ∧ known(jobImage(i)))
│     ⇒ image! = jobImage(first such i)
│ otherwise image! = (if defaultImage ≠ ⊥ then defaultImage else alpine:3.20)
└───────────────────────────────────────────────────

ensure

┌─ Ensure ──────────────────────────────────────────
│ ΔImages
│ image? : IMAGE
├───────────────────────────────────────────────────
│ dryRun ∨ image? ∈ pulledOnce ⇒ θImages' = θImages
│ present(image?) ∧ ¬ pull ⇒ θImages' = θImages
│ (¬ present(image?) ∨ pull) ∧ noPull ⇒ raises NotLocal
│ (¬ present(image?) ∨ pull) ∧ ¬ noPull ⇒
│     pulledOnce' = pulledOnce ∪ {image?} ∧
│     pulled' = pulled ∪ (if present(image?) then ∅ else {image?})
└───────────────────────────────────────────────────

note_pulled

┌─ NotePulled ──────────────────────────────────────
│ ΔImages
│ images? : ℙ IMAGE
├───────────────────────────────────────────────────
│ pulled' = pulled ∪ images?
└───────────────────────────────────────────────────

pulled

┌─ Pulled ──────────────────────────────────────────
│ ΞImages
│ images! : seq IMAGE
├───────────────────────────────────────────────────
│ ran images! = pulled ∧ images! is sorted
└───────────────────────────────────────────────────

remember

┌─ Remember ────────────────────────────────────────
│ ΞImages
│ ΔCacheFile
├───────────────────────────────────────────────────
│ pulled ≠ ∅ ∧ ¬ dryRun ⇒ cache' = cache ∪ pulled
│ pulled = ∅ ∨ dryRun ⇒ cache' = cache
└───────────────────────────────────────────────────

failure_message

┌─ FailureMessage ──────────────────────────────────
│ record? : RECORD
│ text!   : seq CHAR
├───────────────────────────────────────────────────
│ text! = render(unprepared, record?.image, record?.source, record?.error)
└───────────────────────────────────────────────────

STATE DIAGRAM

The object remembers which images it has made sure of, and which it downloaded. One image goes through these states during a run:

  required($instances) / job_expressions(...)
        |   the image is found in a job, service or docker:// step
        v   (also inside local composite actions)
+-------------------+  name uses a step's output
| NAMED             |--------------------------------+
+-------------------+                                |
        |                                            v
        | prepare($instances)                 +---------------+
        v                                     |   DEFERRED    |
+-------------------+                         +---------------+
| CHECKING          |  image_exists?                 |
+-------------------+                                | prepare_job($job, $context)
   |            |                                    | when the job starts:
   | present,   | missing, or --pull                 | resolve with the job's
   | no --pull  v                                    | values, then CHECKING
   |    +-------------------+                        |
   |    | PULLING           |  up to concurrency()   |
   |    | (pull_many)       |  at once               |
   |    +-------------------+                        |
   |       |            |                            |
   |       | ok         | failed                     |
   |       v            v                            |
   |   +--------+   +-------------------------+      |
   |   | PULLED |   | FAILED: run stops with  |      |
   |   +--------+   | failure_message(record) |      |
   |       |        +-------------------------+      |
   v       v                                         |
+-------------------+ <------------------------------+
| READY             |  ensure($image) later: no second pull,
+-------------------+  even with --pull
        |
        | end of run
        v
  keep_images? --no--> images in pulled() are removed
        | yes
        v
  remember($cache_file): pulled() recorded for "action-run cache"

LICENSE AND COPYRIGHT

This program is released under the GNU General Public License, version 2. See the LICENSE file that comes with this distribution. If you use it, please let me know.

To install App::Actions, copy and paste the appropriate command in to your terminal.

cpanm

cpanm App::Actions

CPAN shell

perl -MCPAN -e shell
install App::Actions

For more information on module installation, please visit the detailed CPAN module installation guide.

	Global
`s`	Focus search bar
`?`	Bring up this help dialog

	GitHub
`g` `p`	Go to pull requests
`g` `i`	Go to GitHub issues (only if GitHub is preferred repository)

	POD
`g` `a`	Go to author
`g` `c`	Go to changes
`g` `i`	Go to issues
`g` `d`	Go to dist
`g` `r`	Go to repository/SCM
`g` `s`	Go to source
`g` `b`	Go to file browse

Search terms
module: (e.g. module:Plugin)
distribution: (e.g. distribution:Dancer auth)
author: (e.g. author:SONGMU Redis)
version: (e.g. version:1.00)