The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
VM-EC2-1.11
River stage one • 1 direct dependent • 5 total dependents
5 ++
/ VM::EC2::Staging::Volume

NAME

VM::EC2::Staging::Volume - High level functions for provisioning and populating EC2 volumes

SYNOPSIS

 use VM::EC2::Staging::manager;

 # get a new staging manager
 my $ec2     = VM::EC2->new;
 my $staging = $ec2->staging_manager();                                         );

 my $vol1 = $staging->get_volume(-name => 'Backup',
                                 -fstype => 'ext4',
                                 -size   => 11,
                                 -zone   => 'us-east-1a');

 # make a couple of directories in new volume
 $vol1->mkdir('pictures');
 $vol1->mkdir('videos');

 # use rsync to copy local files onto a subdirectory of this volume
 $vol1->put('/usr/local/my_pictures/' =>'pictures');
 $vol1->put('/usr/local/my_videos/'   =>'videos');

 # use rsync to to copy a set of files on the volume to a local directory 
 mkdir('/tmp/jpegs');
 $vol1->get('pictures/*.jpg','/tmp/jpegs');

 # note that these commands are executed on the remote server as root!
 @listing = $vol1->ls('-r','pictures');
 $vol1->chown('fred','pictures');
 $vol1->chgrp('nobody','pictures');
 $vol1->chmod('0700','pictures');
 $vol1->rm('-rf','pictures/*');
 $vol1->rmdir('pictures');

 # get some information about the volume
 my $mtpt     = $vol->mtpt;
 my $mtdev    = $vol->mtdev;
 my $mounted  = $vol->mounted;
 my $server   = $vol->server;

 # detach the volume
 $vol->detach;

 # delete the volume entirely
 $vol->delete;

DESCRIPTION

This is a high-level interface to EBS volumes which is used in conjunction with VM::EC2::Staging::Manager and VM::EC2::Staging::Server. It is intended to ease the process of allocating and managing EBS volumes, and provides for completely automated filesystem creation, directory management, and data transfer to and from the volume.

You can use staging volumes without having to manually create and manage the instances needed to manipulate the volumes. As needed, the staging manager will create the server(s) needed to execute the desired actions on the volumes.

Staging volumes are wrappers around VM::EC2::Volume, and have all the methods associated with those objects. In addition to the standard EC2 volume characteristics, each staging volume in an EC2 region has a symbolic name, which can be used to retrieve previously-created volumes without remembering their volume ID. This symbolic name is stored in the tag StagingName. Volumes also have a filesystem type (stored in the tag StagingFsType). When a volume is mounted on a staging server, it will also have a mount point on the file system, and a mounting device (e.g. /dev/sdf1).

Staging Volume Creation

Staging volumes are created via a staging manager's get_volume() or provision_volume() methods. See VM::EC2::Staging::Manager. One typical invocation is:

 my $ec2     = VM::EC2->new;
 my $manager = $ec2->staging_manager();                                         );
 my $vol = $manager->get_volume(-name => 'Backup',
                                -fstype => 'ext4',
                                -size   => 5,
                                -zone   => 'us-east-1a');

This will either retrieve an existing volume named "Backup", or, if none exists, create a new one using the provided specification. Behind the scenes, a staging server will be allocated to mount the volume. The manager tries to conserve resources, and so will reuse a suitable running staging server if one is available.

The other typical invocation is:

 my $vol = $manager->provision_volume(-name => 'Backup',
                                      -fstype => 'ext4',
                                      -size   => 5,
                                      -zone   => 'us-east-1a');

This forces creation of a new volume with the indicated characteristics. If a volume of the same name already exists, this method will die with a fatal error (to avoid this, either wrap in an eval, or leave off the -name argument and let the manager pick a unique name for you).

Volume Information

The methods in this section return status information about the staging volume.

$name = $vol->name([$newname])

Get/set the symbolic name associated with this volume.

$mounted = $vol->mounted

Returns true if the volume is currently mounted on a server.

$type = $vol->fstype

Return the filesystem type requested at volume creation time.

$server = $vol->server

Get the server associated with this volume, if any.

$device = $vol->mtdev

Get the device that the volume is attached to, e.g. /dev/sdf1. If the volume is not attached to a server, returns undef.

$device = $vol->mtpt

Get the mount point for this volume on the attached server. If the volume is not mounted, returns undef.

$ebs_vol = $vol->ebs

Get the underlying EBS volume associated with the staging volume object.

$manager = $vol->manager

Return the VM::EC2::Staging::Manager which manages this volume.

$string = $vol->fstab_line();

This method returns the line in /etc/fstab that would be necessary to mount this volume on the server to which it is currently attached at boot time. For example:

 /dev/sdf1 /mnt/staging/Backups ext4 defaults,nobootwait 0 2

You can add this to the current server's fstab using the following code fragment:

 my $server = $vol->server;
 my $fh = $server->scmd_write('sudo -s "cat >>/etc/fstab"');
 print $fh $vol->fstab,"\n";
 close $fh;

$type = $vol->get_fstype

Return the volume's actual filesystem type. This can be different from the requested type if it was later altered by running mkfs on the volume, or the contents of the disk were overwritten by a block-level dd command. As a side effect, this method sets fstype() to the current correct value.

Lifecycle Methods

The methods in this section control the state of the volume.

$snapshot = $vol->create_snapshot('description')

Create a VM::EC2::Snapshot of the volume with an optional description. This differs from the VM::EC2::Volume method of the same name in that it is aware of the mount state of the volume and will first try to unmount it so that the snapshot is clean. After the snapshot is started, the volume is remounted.

$snapshot = $vol->snapshot('description')

Identical to create_snapshot(), but the method name is shorter.

$vol->mount($server [,$mtpt])

$vol->mount()

Mount the volume on the indicated VM::EC2::Staging::Server, optionally at a named mount point on the file system. If the volume is already attached to a different server, it will be detached first. If any of these step fails, the method will die with a fatal error.

When called with no arguments, the volume is automounted on a staging server, creating or starting the server if necessary.

$vol->unmount()

Unmount the volume from wherever it is, but leave it attached to the staging server. If the volume is not already mounted, nothing happens.

Note that it is possible for a volume to be mounted on a stopped server, in which case the server will be started and the volume only unmounted when it is up and running.

$vol->detach()

Unmount and detach the volume from its current server, if any.

Note that it is possible for a volume to be mounted on a stopped server, in which case the server will be started and the volume only unmounted when it is up and running.

$vol->delete()

Delete the volume entirely. If it is mounted and/or attached to a server, it will be unmounted/detached first. If any steps fail, the method will die with a fatal error.

Data Operations

The methods in this section operate on the contents of the volume. By and large, they operate with root privileges on the server machine via judicious use of sudo. Elevated permissions on the local machine (on which the script is running) are not needed.

$vol->get($source_on_vol_1,$source_on_vol_2,...,$dest)

Invoke rsync() on the server to copy files & directories from the indicated source locations on the staging volume to the destination. Source paths can be relative paths, such as "media/photos/vacation", in which case they are relative to the top level of the mounted volume, or absolute paths, such as "/usr/local/media/photos/vacation", in which case they are treated as absolute paths on the server on which the volume is mounted.

The destination can be a path on the local machine, a host:/path on a remote machine, a staging server and path in the form $server:/path, or a staging volume and path in the form "$volume/path". See "Instance Methods for Managing Staging Volumes" in VM::EC2::Staging::Manager for more formats you can use.

As a special case, if you invoke get() with a single argument:

 $vol->get('/tmp/foo')

Then the entire volume will be rsynced into the destination directory /tmp/foo.

$vol->copy($source_on_vol_1,$source_on_vol_2,...,$dest)

This is an alias for get(). It is intended to make it easier to read the intent of this command:

 $source_volume->copy($destination_volume);

Which basically makes a copy of $source_volume onto $destination_volume.

$vol->put($source1,$source2,$source3,...,$dest_on_volume)

Invoke rsync() on the server to copy files & directories from the indicated source locations a destination located on the staging volume. The rules for paths are the same as for the get() method and as described in "Instance Methods for Managing Staging Volumes" in VM::EC2::Staging::Manager .

As a special case, if you invoke put() with a single argument:

 $vol->put('/tmp/foo')

Then the local directory /tmp/foo will be copied onto the top level of the staging volume. To do something similar with multiple source directories, use '/' or '.' as the destination:

 $vol->put('/tmp/pictures','/tmp/audio' => '/');

$vol->dd($destination_volume)

The dd() method performs a block level copy of the volume's disk onto the destination. The destination must be another staging volume.

$output = $vol->cmd($cmd,@args)

This method runs command $cmd on the server that is mounting the volume using ssh. Before the command is run, the working directory is changed to the top level of the volume's mount point. Any arguments, switches, etc you wish to pass to the command can be provided as @args. The output of the command is returned as a string in a scalar context, or an array of lines in a list context.

Example:

 @log = $volume->cmd('tar cvf /tmp/archive.tar .');

$result = $vol->ssh($cmd,@args)

This is similar to cmd(), except that the output of the command is sent to STDOUT and the method returns true if the command executed succcessfully on the remote machine. The cmd() and ssh() methods are equivalent to backticks are system() respectively.

Example:

 $volume->ssh('gzip /tmp/archive.tar') or die "couldn't compress archive";

$output = $vol->df(@args)

$output = $vol->ls(@args)

$success = $vol->mkdir(@args)

$success = $vol->chown(@args)

$success = $vol->chgrp(@args)

$success = $vol->chmod(@args)

$success = $vol->cp(@args)

$success = $vol->mv(@args)

$success = $vol->rm(@args)

$success = $vol->rmdir(@args)

Each of these methods performs the same function as the like-named command-line function, after first changing the working directory to the top level of the volume. They behave as shown in the pseudocode below:

 chdir $vol->mtpt;
 sudo  $method @args

The df() and ls() methods return the output of their corresponding commands. In a scalar context each method returns a string corresponding to the output of running the command on the server to which the volume is attached. In a list context, the methods return one element per line of output.

For example:

 my $free      = $volume->df('.');  # free on current directory
 my ($percent) = $free =~ /(\d+)%/;
 warn "almost out of space" if $percent > 90;

The other methods return a boolean value indicating successful execution of the command on the remote machine.

Command line switches can be passed along with other arguments:

 $volume->mkdir('-p','media/photos/vacation');
 $volume->chown('-R','fred','media');

With the exception of df, each of these commands runs as the superuser, so be careful how you call them.

You may run your own commands using the cmd() and ssh() methods. The former returns the output of the command. The latter returns a success code:

 @log = $volume->cmd('tar cvf /tmp/archive.tar .');
 $volume->ssh('gzip /tmp/archive.tar') or die "couldn't compress archive";

Before calling any of these methods, the volume must be mounted and its server running. A fatal error will occur otherwise.

SEE ALSO

VM::EC2 VM::EC2::Staging::Manager VM::EC2::Staging::Server VM::EC2::Instance VM::EC2::Volume VM::EC2::Snapshot

AUTHOR

Lincoln Stein <lincoln.stein@gmail.com>.

Copyright (c) 2012 Ontario Institute for Cancer Research

This package and its accompanying libraries is free software; you can redistribute it and/or modify it under the terms of the GPL (either version 1, or at your option, any later version) or the Artistic License 2.0. Refer to LICENSE for the full license text. In addition, please see DISCLAIMER.txt for disclaimers of warranty.