Doit - a scripting framework
use Doit; # automatically does use strict + warnings my $doit = Doit->init; $doit->...;
Some boilerplate is needed if parts of the Doit script should be used on a remote machine or as another user — basically the script should be written like a "modulino":
use Doit; sub function_which_may_run_on_remote { my($doit, @args) = @_; ... return ...; } return 1 if caller; my $doit = Doit->init; { my $ssh = $doit->do_ssh_connect('user@host'); # will sync all necessary bits to remote automatically my $result = $ssh->call_with_runner('function_which_may_run_on_remote', $arg, ...); } { my $sudo = $doit->do_sudo; my result = $sudo->call_with_runner('function_which_may_run_on_remote', $arg, ...); }
Call a Doit-using script in dry-run mode:
doit-script.pl --dry-run [other parameters]
Call a Doit-using script in real mode:
doit-script.pl [other parameters]
Doit is a scripting framework. Some core principles implemented here are:
Failing commands throw exceptions — similar to autodie or Fatal (but implemented consistently) or Bourne shell's set -e, or make(1)'s default mode without -k
set -e
-k
Commands are checked first whether execution is required — making it possible to write "converging" scripts
Command execution is logged — like with Bourne shell's set -x or make's default mode
set -x
There's a dry-run mode which just shows what would happen — like make's -n switch
-n
Doit scripts are normal Perl scripts which happen to run Doit commands — no limiting DSL involved, but the full expressiveness of Perl is available.
To achieve the principles it's required to wrap existing functions. A number of Perl builtins and module functions which do side-effects on a system (mostly changes on the file system) are available as Doit commands. Additionally there's a component system for supporting typical tasks like managing system packages, adding users, dealing with source repositories.
Additionally it's possible to run Doit functionality on remote servers (through ssh(1)) or as different users (using sudo(8)).
It is possible to create Doit scripts for running in bootstrapping situations. This means that prerequisites for Doit should be minimal. No mandatory CPAN modules are required, just standard Perl modules. Only for remote connections Net::OpenSSH is needed on the local side. For convenient system command execution IPC::Run may be used. Scripts run with Perl 5.8.x (maybe even 5.6.x is possible).
my $doit = Doit->init;
Generates an object (technically it's a Doit::Runner object) which is used for calling Doit commands. The constructor looks for a command-line option --dry-run (or the short -n alias) and configures the runner for dry-run mode (just print what would be executed), otherwise for real mode (actually execute everything). Other command-line options are still available and may be used in the script, e.g. by using Getopt::Long or looking into @ARGV:
Doit::Runner
--dry-run
@ARGV
use Doit; use Getopt::Long; my $doit = Doit->init; # already handles --dry-run and -n GetOptions(...) or die "usage: ..."; my @files = @ARGV or die "usage: ...";
All core commands throw exceptions on errors. If not stated otherwise, then the return value is the number of changes, typically the number of files affected --- in dry-run mode it's the number of changes which would be done, and in real mode it's the number of changes performed.
$doit->chmod($mode, $file ...);
Make sure that the permission of the listed files is set to $mode (which is typically expressed as an octal number). Fails if not all files could be changed. See "chmod" in perlfunc for more details.
$doit->chown($user, $group, $file ...);
Make sure that the owner (and group) of the listed files is set to the given values. The user and group may be specified as uid/gid or as username/groupname. A value of -1 or undef for $user and $group is interpreted to leave that value unchanged. This command is not useful on Windows systems. See "chown" in perlfunc for more details.
undef
$doit->create_file_if_nonexisting($file ...);
Make sure that the listed files exist. Contrary to the Doit touch command and the system command touch(1) this does nothing if the file already exists.
$doit->copy($from, $to); $doit->copy({quiet => $bool}, $from, $to);
Make sure that the file $from is copied to $to unless there's already a file with same contents. Copying is done with File::Copy::copy. File attributes are not copied — this can be done using Doit::Util::copy_stat.
The logging includes a diff between both files, if the diff(1) utility is available. This can be turned off by specifying the quiet=>1 option.
quiet=>1
$doit->ln_nsf($oldfile, $newfile);
Make sure that $newfile is a symlink pointing to $oldfile, possibly replacing an existing symlink. Implemented by running the system's ln -nsf. See ln(1) for more details and "symlink" for an alternative.
ln -nsf
$doit->make_path($directory ...); $doit->make_path($directory ..., { key => val ... });
Make sure that the listed directories exist, together with any missing intermediate directories. Additional options may be specified as key-value pairs in a hashref, and will be passed to File::Path::make_path.
Note that it's possible to set the directory permissions with the mode option, but the make_path command does not check if an already existing directory has these permission bits set.
mode
make_path
See also "mkdir".
$doit->mkdir($directory); $doit->mkdir($directory, $mode);
Make sure that the given $directory exist. The $mode will be used only if creating a new directory and not effective if the directory already exists. See "mkdir" in perlfunc for more details. See "make_path" for a command which will also create mising intermediate directories.
$doit->move($from, $to);
Move file $from to $to. This command probably cannot be used in converging scripts without an accompanying condition. See File::Copy::move for details. For an alternative command see "rename".
Always returns 1 (unless there's an exception).
1
$doit->remove_tree($directory ...); $doit->remove_tree($directory ..., { key => $val ... });
Make sure that the listed directories don't exist anymore, together with containing files and sub-directories. See File::Path::remove_tree for details.
$doit->rename($from, $to);
Rename $from to $to. This command probably cannot be used in converging scripts without an accompanying condition. See "rename" in perlfunc for details. See "move" for a command which can move a file between different filesystems.
$doit->rmdir($directory);
Make sure that the given $directory is removed. Fails if this directory is not empty. See "rmdir" in perlfunc for details.
$doit->setenv($key, $val);
Make sure that %ENV contains a key $key set to $value.
$doit->symlink($oldfile, $newfile);
Make sure that $newfile is a symlink pointing to $oldfile. Contrary to "ln_nsf" it does not change an existing symlink. See "symlink" in perlfunc for more details.
$doit->touch($file ...);
"Touches" the given files. Loosely modelled after the system command touch(1). Non-existent files are created as empty files, and for existent files the access and modification are updated. This command does not converge; for a converging command see "create_file_if_nonexisting".
Always returns the number of given files (unless there's an exception).
$doit->unlink($file ...);
Make sure that the given files are deleted. See "unlink" in perlfunc.
$doit->unsetenv($key);
Make sure that %ENV does not contain the key $key anymore.
$doit->utime($atime, $mtime, $file ...);
Make sure that access time and modification time of the listed files is set to the given values. Undefined time values are replaced by current time. Fails if not all files could be changed. See "utime" in perlfunc for details.
$doit->change_file({debug => $bool, check => $code}, $file, { change ... } ...);
Modify an existing $file using the set of change specifications. Return the number of changes made. Depending on the changes the command call can be converging or not. The following change specifications exist:
{ add_if_missing => $line }
Add the specified $line to the end of file if it is missing.
{ add_if_missing => $line, add_after => $rx }
Add the specified $line after a the last line mathing $rx. If no line matches, then an exception will be thrown.
{ add_if_missing => $line, add_after_first => $rx }
Add the specified $line after a the first line mathing $rx. If no line matches, then an exception will be thrown.
{ add_if_missing => $line, add_before => $rx }
Add the specified $line before a the first line mathing $rx. If no line matches, then an exception will be thrown.
{ add_if_missing => $line, add_before_last => $rx }
Add the specified $line before a the last line mathing $rx. If no line matches, then an exception will be thrown.
{ match => $rx_or_string, replace => $line }
Substitute all lines matching $rx_or_string with $line. $rx_or_string may be a regexp, or a string. In the latter case the complete line has to match.
{ match => $rx_or_string, delete => $bool }
All lines matching $rx_or_string will be deleted if delete is set to a true value.
delete
{ match => $rx, action => $code }
For all lines matching $rx_or_string call the specified $code reference. In the code reference $_[0] can be used to manipulate the line.
{ unless_match ... }
TBD
The following options may be set:
debug => $bool
Turn on debugging if set to a true value.
check => $code
Do a final check on the non-committed file. The $code reference will take the filename as parameter. Just throw an exception if the changes should not be committed.
Implementation details: a temporary copy is created first, which is then changed using Tie::File. If everything went right and changes had to be done, then the changed file is renamed to the final destination.
Using Tie::File has some consequences: it does not know anything about encodings, and it uses $/ to determine line endings, which by default is set to do the right thing for text files. In future these details may be streamlined or even changed.
Tie::File
$/
$doit->write_binary($filename, $content); $doit->write_binary({quiet=>$level, atomic=>$bool}, $filename, $content);
Make sure that the file $filename has the content $content. The given content should be unencoded (raw, binary, octets). If you need a specific encoding, then it has to be encoded before:
$doit->write_binary($filename, Encode::encode_utf8($character_content));
If quiet is set to 1, then no diffs are shown, otherwise a diff is shown if the file contents changed (and the diff(1) utility is available), or the complete contents are shown for a new file. If quiet is set to 2, then no logging at all is done.
quiet
2
By default, the file is written atomically by writing to a temporary file first, and then renamed. This can be changed by setting atomic=>0.
atomic=>0
The command is modelled after File::Slurper::write_binary, but implemented without any dependencies.
Note: currently there's no write_text, but maybe will be added in future.
write_text
A number of commands exist for executing system commands. All of these (except for cond_run) are non-converging and probably should be run conditionally.
cond_run
All of the commands may be used with list syntax to avoid usage of a shell. This is also especially true on Windows, where perl's system has problematic edge cases.
system
A quick overview:
$doit->cond_run(if => sub { ... }, cmd => ["command", "arg" ...]); $doit->cond_run(unless => sub { ... }, cmd => ["command", "arg" ...]); $doit->cond_run(creates => $file, cmd => ["command", "arg" ...]);
Conditionally run the command specified in cmd (an array reference with command and arguments). Conditions are expressed as code references which should return a true or false value (options if for a positive condition and unless for a negative condition), or with the option creates for checking the existence of the given $file which is expected to be created by the given command. Conditions may be combined.
cmd
if
unless
creates
The cmd option may also specify a IPC::Run-compatible list, for example:
$doit->cond_run(creates => $file, cmd => [["command", "arg" ...], ">", $file]);
Return 1 if the condition was true and the cmd executed, otherwise 0.
0
my $stdout = $doit->open2("command", "arg" ...); my $stdout = $doit->open2({quiet => $bool, info => $bool, instr => $input}, "command", "arg" ...);
Execute a command and return the produced stdout. If quiet is set to a true value, then no logging is done. If info is set to a true value, then command execution happens even in dry-run mode. If instr is specified, then it's send to the stdin of the command. Implementation is done with IPC::Open2.
info
instr
my $stdout = $doit->open3("command", "arg" ...); my $stdout = $doit->open3({quiet => $bool, info => $bool, instr => $input, errref => \$stderr, statusref => \%status, }, "command", "arg" ...);
Execute a command and return the produced stdout. If quiet is set to a true value, then no logging is done. If info is set to a true value, then command execution happens even in dry-run mode. If instr is specified, then it's send to the stdin of the command. If errref is set to a scalar reference, then this is filled with the stderr of the command; otherwise stderr won't show up. If statusref is set to a hash reference, then it is filled with the exit information of the command: msg, errno, exitcode, signalnum, coredump (except for msg fields may be missing).
errref
statusref
msg
errno
exitcode
signalnum
coredump
Implementation is done with IPC::Open3.
my $stdout = $doit->qx("command", "arg" ...); my $stdout = $doit->qx({quiet => $bool, info => $bool, statusref => \%status}, "command", "arg" ...);
Execute a command and return the produced stdout. If quiet is set to a true value, then no logging is done. If info is set to a true value, then command execution happens even in dry-run mode. statusref may be set to a hash reference for getting exit information, see "open3" for more information on it.
Implementation is done with Safe Pipe Opens.
$doit->run(...);
Execute a command using IPC::Run. The command specification may contain pipes, redirects and everything IPC::Run supports. Example:
IPC::Run
$doit->run([qw(grep pattern file)], '|', ['sort'], '|', [qw(uniq -c)], '>', 'outfile');
$doit->system("command", "arg" ...); $doit->system({show_cwd => $bool}, "command", "arg" ...);
Execute a command. If show_cwd is set to a true value, then logging shows also the current working directory. See "system" in perlfunc for more details.
show_cwd
Commands starting with info_ also run in dry-run mode. It is expected that the user only runs system commands which are not doing any changes to the system, but just return some kind of "information".
info_
Note: currently the info_* commands fail on non-zero exit code. This behavior is probably not very useful (just think of running a non-matching grep) and may change in future. Currently these invocations have to be wrapped in an eval { ... } if non-zero exit may happen.
info_*
eval { ... }
my $stdout = $doit->info_open2("command", "arg" ...); my $stdout = $doit->info_open2({quiet => $bool, instr => $input}, "command", "arg" ...);
Like "open2", but with the option info=>1 set.
info=>1
my $stdout = $doit->info_open3("command", "arg" ...); my $stdout = $doit->info_open3({...}, "command", "arg" ...);
Like "open3", but with the option info=>1 set.
my $stdout = $doit->info_qx("command", "arg" ...); my $stdout = $doit->info_qx({quiet => $bool, statusref => \%status}, "command", "arg" ...);
Like "qx", but with the option info=>1 set.
It's possible to run Perl code or Doit functionality on remote servers, or as different users. Two commands exist to create a Doit runner-like object: "do_ssh_connect" for running code over a ssh connection, and "do_sudo" for running code as a different user using sudo.
sudo
This Doit runner-like object may execute all Doit commands. Additionally it's possible to call functions defined in the Doit script itself using "call_with_runner" and "call". The latter two may be preferable in some cases preferable, as every call involves some serialization and communication overhead, also the current serialization method (Storable) limits the possible parameter and result types (i.e. regexps cannot be transferred, which is needed for some Doit commands like "change_file").
For remote and sudo operation the active Doit script together with Doit.pm and required components are sent to the destination system. The Doit script is loaded using require there — which is the reason why the script has to be written like a modulino: the "main" part of the script must not re-run again.
Remote and sudo operation work best if things are setup to be password-less. Using a ssh agent helps here, and if you may, define NOPASSWD in the /etc/sudoers rules. It's still possible to run scripts which require manual password input, but some setups like combining ssh and sudo may be tricky.
NOPASSWD
dry-run mode, if defined, is passed to the other system or user.
my $ssh = $doit->do_ssh_connect('user@host', options ...);
Create a Doit runner-like object which runs commands over a ssh connection to user@host. The connection and communication is created and done using Net::OpenSSH. The following options are available:
Turn communication-level debugging on.
as => $username
Run as a different user on remote side. Switching user is done with sudo. If the switch is not password-less, then probably something like tty should be passed in the options.
tty
forward_agent => $bool
Enable ssh agent forwardning if set to true.
tty => $bool
Allocate a pseudo terminal. May be useful if the script requires interactive input (e.g. password input).
port => $port
Use a different ssh port.
master_opts => [ ... ]
Additional options to pass. See "master_opts" in Net::OpenSSH.
put_to_remote => $method
Method to copy files to the remote side. Possible options are rsync_put (default) and scp_put.
rsync_put
scp_put
perl => $path
Use a different perl for running the commands. Defaults to $^X.
$^X
my $sudo = $doit->do_sudo(options);
Create a Doit runner-like object which runs in a different user context. By default it runs as root. The following options are available:
root
sudo_opts => [ ... ]
An array reference of options passed to the sudo command. For example, to run as another user than root, try:
..., sudo_opts => ['-u', $username], ...
my $result = $ssh->call_with_runner('function', 'arg' ...); my $result = $sudo->call_with_runner('function', 'arg' ...);
Call a function in a remote or switched user context. The function will get the Doit runner object as the first argument. Other arguments are serialized and sent to the function. The function's result is also serialized and sent back. Context is preserved.
Example:
sub print_hostname { my($doit) = @_; $doit->system('hostname'); # will print the remote hostname } ... return 1 if caller; ... my $ssh = $doit->do_ssh_connect('user@host'); $ssh->call_with_runner('print_hostname');
my $result = $ssh->call('function', 'arg' ...); my $result = $sudo->call('function', 'arg' ...);
Like "call_with_runner", but don't pass a Doit runner object.
Doit components are Perl modules which may define additional commands mixed into the Doit runner object. A component is added by calling the "add_component" method:
$doit->add_component('git');
The component name is written lowercase without the Doit:: prefix of the implementing module, that is, for Doit::Git one uses the name git.
Doit::
Doit::Git
git
The component commands are typically prefixed with the component name. For example, the fbsdpkg component defined the fbsdpkg_install_packages and fbsdpkg_missing_packages commands.
fbsdpkg
fbsdpkg_install_packages
fbsdpkg_missing_packages
Components added with add_component are also synced to remote systems for subsequent connections.
add_component
The following components are available:
Doit::Brew - handle homebrew packages for Mac OS X
Doit::Deb - handle Debian packages
Doit::Fbsdpkg - handle FreeBSD packages
Doit::Gem - handle Ruby gem packages
Doit::Rpm - handle RPM packages
Doit::Macsecurity - handle certificate management on Mac OS X
Doit::Ssl - handle certificate management for OpenSSL systems
Doit::File - additional file commands
Doit::Git - handle git repositories
Doit::Locale - handle locale installation
Doit::Lwp - WWW access
Doit::User - user management
Slaven Rezic <srezic@cpan.org>
Copyright (c) 2017 Slaven Rezic. All rights reserved. This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself.
perlfunc - most core commands are modelled after perl builtin functions
Doit::Log, Doit::Exception, Doit::ScopeCleanups, Doit::Util, Doit::Win32Util - packages embeded in Doit.pm
make(1), slaymake(1), Slay::Makefile, Commands::Guarded
To install Doit, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Doit
CPAN shell
perl -MCPAN -e shell install Doit
For more information on module installation, please visit the detailed CPAN module installation guide.