++ed by:
Marc Lehmann

NAME

Urlader - installer-less single-file independent executables

SYNOPSIS

 use Urlader;

DESCRIPTION

Urlader (that's german for "bootloader" btw.) was created out of frustration over PAR always being horribly slow, again not working, again not being flexible enough for simple things such as software upgrades, and again causing mysterious missing file issues on various platforms.

That doesn't mean this module replaces PAR, in fact, you should stay with PAR for many reasons right now, user-friendlyness is one of them.

However, if you want to make single-file distributions out of your perl programs (or python, or C or whatever), and you are prepared to fiddle a LOT, this module might provide a faster and more versatile deployment technique then PAR. Well, if it ever gets finished.

Also, nothing in this module is considered very stable yet, and it's far from feature-complete.

Having said all that, Urlader basically provides three services:

A simple archiver that packs a directory tree into a single file.
A small C program that works on windows and unix, which unpacks an attached archive and runs a program (perl, python, whatever...).
A perl module support module (this one), that can be used to query the runtime environment, find out where to install updates and so on.

EXAMPLE

How can it be used to provide single-file executables?

So simple, create a directory with everything that's needed, e.g.:

   # find bintree
   bintree/perl
   bintree/libperl.so.5.10
   bintree/run
   bintree/pm/Guard.pm
   bintree/pm/auto/Guard/Guard.so
   bintree/pm/XSLoader.pm
   bintree/pm/DynaLoader.pm
   bintree/pm/Config.pm
   bintree/pm/strict.pm
   bintree/pm/vars.pm
   bintree/pm/warnings.pm

   # cat bintree/run
   @INC = ("pm", "."); # "." works around buggy AutoLoader
   use Guard;
   guard { warn "hello, world!\n" }; # just to show off
   exit 0; # tell the urlader that everything was fine

Then pack it:

   # wget http://urlader.schmorp.de/prebuilt/1.0/linux-x86
   # urlader-util --urlader linux-x86 --pack myprog ver1_000 bintree \
                  LD_LIBRARY_PATH=. ./perl run \
                  >myprog
   # chmod 755 myprog

The packing step takes a binary loader and appends all the files in the directory tree, plus some meta information.

The resulting file is an executable that, when run, will unpack all the files and run the embedded program.

CONCEPTS

urlader

A small (hopefully) and relatively portable (hopefully) binary that is prepended to a pack file to make it executable.

You can build it yourself from sources (see prebuilt/Makefile in the distribution) or use one of the precompiled ones at:

   http://urlader.schmorp.de/prebuilt/1.0/

The README there has further information on the binaries provided.

exe_id

A string that uniquely identifies your program - all branches of it. It must consist of the characters A-Za-z0-9_- only and should be a valid directory name on all systems you want to deploy on.

exe_ver

A string the uniquely identifies the contents of the archive, i.e. the version. It has the same restrictions as the exe_id, and should be fixed-length, as Urlader assumes lexicographically higher versions are newer, and thus preferable.

pack file (archive)

This contains the exe_id, the exe_ver, a number of environment variable assignments, the program name to execute, the initial arguments it receives, and finally, a list of files (with contents :) and directories.

override

When the urlader starts, it first finds out what exe_id is embedded in it. It then looks for an override file for this id ($URLADER_EXE_DIR/override) and verifies that it is for the same exe_id, and the version is newer. If this is the case, then it will unpack and run the override file instead of unpacking the files attched to itself.

This way one can implement software upgrades - download a new executable, write it safely to disk and move it to the override path.

ENVIRONMENT VARIABLES

The urlader sets and maintains the following environment variables, in addition to any variables specified on the commandline. The values in parentheses are typical (but not gauranteed) values for unix - on windows, ~/.urlader is replaced by %AppData%/urlader.

URLADER_VERSION (1.0)

Set to the version of the urlader binary itself. All versions with the same major number should be compatible to older versions with the same major number.

URLADER_DATADIR (~/.urlader)

The data directory used to store whatever urlader needs to store.

URLADER_CURRDIR

This is set to the full path of the current working directory where the urlader was started. Atfer unpacking, the urlader changes to the URLADER_EXECDIR, so any relative paths should be resolved via this path.

URLADER_EXEPATH

This is set to the path of the urlader executable itself, usually relative to $URLADER_CURRDIR.

URLADER_EXE_ID

This stores the executable id of the pack file attached to the urlader.

URLADER_EXE_VER

This is the executable version of the pack file attached to the urlader, or the override, whichever was newer. Or in other words, this is the version of the application running at the moment.

URLADER_EXE_DIR (~/.urlader/$URLADER_EXE_ID>

The directory where urlader stores files related to the executable with the given id.

URLADER_EXECDIR (~/.urlader/$URLADER_EXE_ID/i-$URLADER_EXE_VER)

The directory where the files from the pack file are unpacked and the program is being run. Also the working directory of the program when it is run.

URLADER_OVERRIDE (empty or override)

The override file used, if any, relative to $URLADER_EXECDIR. This is either missing, when no override was used, or the string override, as thta is currently the only override file urlader is looking for.

FUNCTIONS AND VARIABLES IN THIS MODULE

$Urlader::URLADER_VERSION

Set to the urlader version (URLADER_VERSION) when the program is running form within urlader, undef otherwise.

$Urlader::DATADIR, $Urlader::EXE_ID, $Urlader::EXE_VER, $Urlader::EXE_DIR, $Urlader::EXECDIR

Contain the same value as the environment variable of the (almost) same name. You should prefer these, though, as these might even be set to correct values when not running form within an urlader environment.

Urlader::set_exe_info $exe_id, $exe_ver

Sets up the paths and variables as if running the given executable and version from within urlader.

$lock = Urlader::lock $path, $exclusive, $wait

Tries to acquire a lock on the given path (which must specify a file which will be created if necessary). If $exclusive is true, then it tries to acquire an exclusive lock, otherwise the lock will be shared. If $wait is true, then it will wait until the lock can be acquired, otherwise it only attempts to acquire it and returns immediately if it can't.

If successful it returns a lock object - the lock will be given up when the lock object is destroyed or when the process exits (even on a crash) and has a good chance of working on network drives as well.

If the lock could not be acquired, undef is returned.

This function is provided to assist applications that want to clean up old versions, see "TIPS AND TRICKS", below.

TIPS AND TRICKS

Gathering files

Gathering all the files needed for distribution can be a big problem. Right now, Urlader does not assist you in this task in any way, however, just like perl source stripping, it is planned to unbundle the relevant technology from staticperl (http://staticperl.plan9.de) for use with this module.

You could always use par to find all library files, unpack the bundle and add perl, libperl and other support libraries (e.g. libgcc_s).

Software update

Updating the software can be done by downloading a new packfile (with the same exe_id but a higher exe_ver - this can simply be the executable you create when making a release) and replacing the override file in the $URLADER_EXE_DIR.

When looking for updates, you should include $URLADER_VERSION, $URLADER_EXE_ID and $URLADER_EXE_VER - the first two must be identical for update and currently running program, while the last one should be lexicographically higher.

Replacing the override file can be done like this:

   rename "new-override.tmp", "$Urlader::EXE_DIR/override"
      or die "could not replace override";

This can fail on windows when another urlader currently reads it, but should work on all platforms even when other urlader programs execute concurrently.

Cleaning up old directories

Urlader only packs executables once and then caches them in the $URLADER_EXECDIR. After upgrades there will be old versions in there that are not being used anymore. Or are they?

Each instance directory (i-*) in the $URLADER_EXE_DIR) has an associated lock file (i-*.lck) - while urlader executes an app it keeps a shared lock on this file.

To detect whether a version is in use or not, you must try to acquire an exclusive lock, i.e.:

  my $lock = Urlader::lock "~/.urlader/myexe/i-ver01.lck", 1, 0;
  if (!$lock) {
     # instance dir is not in use and can be safely deleted
  }

If an older urlader wants to use an instance that was deleted or is currently being deleted it will wait until it's gone and simply recreate it, so while less efficient, deleting instance directories should always be safe.

The lockfile itself can be deleted as long as you have an exclusive lock on it (if your platform allows this).

A real world project

The only real world project using this that I know of at the moment is the deliantra client (http://www.deliantra.net for more info).

It uses some scary scripts to build the client and some dependnet modules (build.*), to gather perl source files into a distribution tree, shared objects and system shared libraries (some of which have to be patched or, due to the horrible dll hell on OS X, even renamed), called gatherer, and a script called gendist to build executable distributions.

These can be found at http://cvs.schmorp.de/deliantra/Deliantra-Client/util/, but looking at them can lead to premature blindless.

Shared Libraries

It is often desirable to package shared libraries - for example the Deliantra client packages SD>, Berkely DB, Pango and amny other libraries that are unlikely to be available on the target system.

This usually requires some fiddling (see below), and additionally some environment variables to be set.

For example, on ELF systems you usually want LD_LIBRARY_PATH=. and on OS X, you want DYLD_LIBRARY_PATH=. (these are effectively the default on windows).

These can most easily be specified when building the packfile:

   urlader-util ... LD_LIBRARY_PATH=. ./perl run
Portability: RPATH

Often perl is linked against a shared libperl.so - and might be so using an rpath. Perl extensikns likewise might use an rpath, which means the binary will mostly ignore LD_LIBRARY_PATH, which leads to trouble.

There is an utility called chrpath, whose -d option can remove the rpath from binaries, shared library and shared objects.

Portability: OS X DLL HELL

OS X has the most severe form of DLL hell I have seen - if you link against system libraries, which is practically unavoidable, you get libraries of well-known names (e.g. libjpeg) that have nothing to do with what you normally expect libjpeg to be, and there is no way to get your version of libjpeg into your program.

Moreover, even if apple ships well-known libraries (e.g. libiconv), they often ship patched versions which have a different ABI or even API then the real releases.

The only way aorund this I found was to change all library names in my releases (libjpeg.dylib becomes libdeliantra-jpeg.dylin and so on), by patching the paths in the share dlibraries and shared objects. install-name-tool (with -id and -change) works in many cases, but often paths are embedded indirectly, so you might have to use a dirty string replacement.

SECURITY CONSIDERATIONS

The urlader executable itself does not support setuig/setgid operation, or running with elevated privileges - it does no input sanitisation, and is trivially exploitable.

AUTHOR

 Marc Lehmann <schmorp@schmorp.de>
 http://home.schmorp.de/



Hosting generously
sponsored by Bytemark