ccp - cascading copy
ccp [OPTIONS]
This script implements cascading copies using Net::CascadeCopy.
This module efficiently distributes a file or directory across a large number of servers in multiple datacenters via rsync or scp.
A frequent solution to distributing a file or directory to a large number of servers is to copy it from a central file server to all other servers. To speed this up, multiple file servers may be used, or files may be copied in parallel until the inevitable bottleneck in network/disk/cpu is reached. These approaches run in O(n) time.
This module and the included script, ccp, take a much more efficient approach, i.e. O(log n). Once the file(s) are been copied to a remote server, that server will be promoted to be used as source server for copying to remaining servers. Thus, the rate of transfer increases exponentially rather than linearly. Needless to say, when transferring files to a large number of remote servers (e.g. over 40), this can make a ginormous difference.
Servers can be specified in groups (e.g. datacenter) to prevent copying across groups. This maximizes the number of transfers done over a local high-speed connection (LAN) while minimizing the number of transfers over the WAN.
The following options are supported by this command
Specifies the path of the file to be transferrred.
Specified that the file should be copied to an alternate location on the remote host. Defaults to the same value as -path.
Add a group of servers named groupname containing three servers. Multiple groups may be specified. All copying will be performed within each defined group--no copying will be performed across groups.
Servers may not be listed in more than one group.
The initial transfer will be done from the current host to the first server in every group. After that, transfers will be performed in order by available servers.
Specify the command that will be executed to copy the file, e.g. scp or rsync.
Specify the arguments to be passed to the command specified. For example, "-p" might be used with scp to preserve permissions.
Specify that stdout/stderr of each child process should be written to a log file named ccp.hostname.log.
Specify how many times to allow a failed transfer to each target box. In the event of a failure, the failed target will be added back to the end of the list. Most likely each copy will be attempted from a different source machine. The default is 3.
Specify how many child processes should be spawned for each available source machine. The default is 2.
Specify how to invoke the ssh command locally if it can't be found in your path. ssh is used to log in to source servers to initiate copies to target servers.
Specify flags to be sent to ssh processes.
Verbose output.
Display usage. Displays full manual when combined with -v.
Display stdout from all child processes as it is received. This can get a bit crazy and is only recommended for debugging.
A list of every error and warning message that the module can generate (even the ones that will "never happen"), with a full explanation of each problem, one or more likely causes, and any suggested remedies. (See also "Documenting Errors" in Chapter 13.)
A full explanation of any configuration system(s) used by the module, including the names and locations of any configuration files, and the meaning of any environment variables or properties that can be set. These descriptions must also include details of any configuration language used. (See also "Configuration Files" in Chapter 19.)
A list of all the other modules that this module relies upon, including any restrictions on versions, and an indication of whether these required modules are part of the standard Perl distribution, part of the module's distribution, or must be installed separately.
A list of any modules that this module cannot be used in conjunction with. This may be due to name conflicts in the interface, or competition for system or program resources, or due to internal limitations of Perl (for example, many modules that use source code filters are mutually incompatible).
A list of known problems with the module, together with some indication of whether they are likely to be fixed in an upcoming release.
Also a list of restrictions on the features the module does provide: data types that cannot be handled, performance issues and the circumstances in which they may arise, practical limitations on the size of data sets, special cases that are not (yet) handled, etc.
Usually just contains:
There are no known bugs in this module. Please report problems to VVu@geekfarm.org
Patches are welcome.
VVu@geekfarm.org
Thanks to Russ and Robert for coming up with the idea of cascading deployments!
Copyright (c) 2006, VVu@geekfarm.org All rights reserved.
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
- Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
- Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
- Neither the name of the geekfarm.org nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
To install Net::CascadeCopy, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Net::CascadeCopy
CPAN shell
perl -MCPAN -e shell install Net::CascadeCopy
For more information on module installation, please visit the detailed CPAN module installation guide.