File::Repl - Perl module that provides file replication utilities
use File::Repl; %con = ( dira => 'C:/perl', dirb => 'M:/perl', verbose => '1', age => '10', ); $ref=File::Repl->New(\%con); $r1 = $ref->Update('\.p(l|m)','a<>b',1); $r2 = $ref->Update('\.t.*','\.tmp$','a<>b',1);
The File:Repl provides simple file replication and management utilities. Its main functions are
Allowing two directory structures to be maintained, ensuring files that meet selection logic criteria are mirrored and otherwise synchronized.
Allowing files in a directory structure to be renamed according to the selection logic.
Allowing files in a directory structure to be compressed according to a given logic.
Run a common perl process against files in a directory structure according to selection logic.
Allowing files in a directory structure to be deleted according to the selection logic.
The New method constructs a new File-Repl object. Options are passed in the form of a hash reference \%con which define the file directories to be operated on and other parameters. The directories are scanned and each file is stat'ed. The hash keys have the following definitions-
This identifies the first directory to be scanned (required).
This identifies the second directory to be scanned (required). If the object is only to have methods operate on it that operate on a single directory then dirb can be set to the same value as dira. This minimizes the directory structure to be sesarched.
The verbose flag has several valid values:
No verbosity (default mode).
All file copies and deletes are printed.
Tombstone file trunkations are printed, and any timestamp changes made. Any file copies or deletes that would have been made that failed the agelimit criteria are printed.
Configuration settings (from %con) and Files meeting the match criteria are printed.
Files identified in each directory that match the regex requirements (from the Update method) are printed.
This specifies the maximum age of a file in days. Files older than this will be ignored by Update, Rename, Compress and Delete methods.
If the age is specified as a negative number files newer than this age will be ignored.
A default value of zero causes no age limit to be tested - all files are accepted on age limits.
When set to FALSE only files at the top level of the dira and dirb are scanned. Default value is TRUE
This is the time to live (ttl for any tombstoned file, in days. Default value is 31.
Switches for case sensitivity - default is TRUE (case insensitive).
If either directory dira or dirb do not exist will attempt to create the directory if set TRUE. Default value is FALSE.
The Update method makes the file updates requested - determined by the %con hash (from the New method) and four associated arguments.
This method also allows files to be tombstoned (ie removed from the replicated file sets). A file is tombstoned by appending .remove to the file name. The first Update will cause the file to be set to zero size, and any replica files to be renamed (so that the original file does not return). The next update after the ttl has expired will cause deletion of all file replicas.
If a directory is tombstoned (by adding .remove to its name) the directory and contents are removed and a file with the directory name and the .remove suffix replaces it. The file is removed as a normally tombstoned file. Note that tombstoning ignores the Update action qualifiers.
The Update method returns a reference to data structures evaluated during the method call. This is based on the method arguments, and allows arrays and hash's of the file structure meeting the selection criteria to be returned. See "EXAMPLES". Note that the aonly, bonly, amatch and bmatch array references, and the common hash reference all refer to the file structure state BEFORE the Update method makes any changes.
A regular expression, used to match all file names that are to be maintained.
An optional regular expression used to match all files not to be maintained (ie excluded from the operation).
defines the action to be performed. Note that tombstoning activities ignore the action and assume the A<>B directive for those files and directories being tombstoned.
Files in the 'a' directory are to be replicated to the 'b' directory if a replica exists in 'b' directory and the timestamp is older than that of the file in the 'a' directory.
Files in the 'b' directory are to be replicated to the 'a' directory if a replica exists in 'a' directory and the timestamp is older than that of the file in the 'b' directory.
Files in the 'a' directory are to be replicated to the 'b' directory if a replica exists in 'b' directory and the timestamp is older than that of the file in the 'a' directory. Files in the 'b' directory are to be replicated to the 'a' directory if a replica exists in 'a' directory and the timestamp is older than that of the file in the 'b' directory.
Files in the 'a' directory are to be replicated to the 'b' directory - even if no replica exists in 'b' directory. If a replica already exists in the 'b' directory with a timestamp that is newer than that of the file in the 'a' directory it is not modified.
Files in the 'a' directory are to be replicated to the 'b' directory - even if no replica exists in 'b' directory. If a replica already exists in the 'b' directory with a timestamp that is newer than that of the file in the 'a' directory it is not modified. Orphan files in the 'b' directory are deleted.
Files in the 'b' directory are to be replicated to the 'a' directory - even if no replica exists in 'a' directory. If a replica already exists in the 'a' directory with a timestamp that is newer than that of the file in the 'b' directory it is not modified.
Files in the 'b' directory are to be replicated to the 'a' directory - even if no replica exists in 'a' directory. If a replica already exists in the 'a' directory with a timestamp that is newer than that of the file in the 'b' directory it is not modified. Orphan files in the 'a' directory are deleted.
Files in the 'a' directory are to be replicated to the 'b' directory - even if no replica exists in 'b' directory. If a replica already exists in the 'b' directory with a timestamp that is newer than that of the file in the 'a' directory it is not modified. Files in the 'b' directory are to be replicated to the 'a' directory - even if no replica exists in 'a' directory. If a replica already exists in the 'a' directory with a timestamp that is newer than that of the file in the 'b' directory it is not modified.
When set TRUE makes changes required - set FALSE to show potential changes (which are printed to STDOUT)
The Rename method is used to rename files in the dira directory structure in the object specified in the New method.
A regular expression, used to match all file names that are to be renamed.
An optional regular expression used to match all files not to be renamed (ie excluded from the operation).
The argument used for a perl substitution command is applied to the file name to create the file's new name.
e.g. /\.pl$/\.perl/
This examplewill rename all files (that meet regex and noregex criteria) from .pl to .perl
When set TRUE makes renames required - set FALSE to show potential changes (which are printed to STDOUT)
Not yet implemeneted
Not yet implemented
The Delete method removes files from the dira directory structure in the object specified in the New method.
A regular expression, used to match all file names that are to be deleted.
An optional regular expression used to match all files not to be deleted (ie excluded from the operation).
When set TRUE makes deletions required - set FALSE to show potential changes (which are printed to STDOUT)
The Version method returns the File::Repl module version. No calling argument is necessary.
File::Find; File::Copy; File::Basename; Win32::API (Win32 platforms only)
On FAT filesystems, mtime resolution is 1/30th of a second. A fudge of 2 seconds is used for synching FAT with other filesystems. Note that FAT filesystems save the local time in UTC (GMT).
On FAT filesystems, "stat" adds TZ_BIAS to the actual file times (atime, ctime and mtime) and conversley "utime" subtracts TZ_BIAS from the supplied parameters before setting file times. To maintain FAT at UTC time, we need to do the opposite.
If we don't maintain FAT filesystems at UTC time and the repl is between FAT and NON-FAT systems, then all files will get replicated whenever the TZ or Daylight Savings Time changes.
A simple example that retrieves and prints the working variables from the Update method
$ref=File::Repl->New(\%hash); $my=$ref->Update('.*','A>B',1); $sub = sub { # simple sub that determines the reference type and prints the associated values my ($ref) =$_[0]; if ( ref($ref) eq "SCALAR" ) { print " SCALAR $ref\n"; }elsif( ref($ref) eq "ARRAY" ) { print " ARRAY"; foreach (@$ref) { print "\t$_\n"; } }elsif( ref($ref) eq "HASH" ) { print " HASH "; foreach (keys %$ref) { print "\t$_ => $$ref{$_}\n"; } }elsif( ref($ref) eq "REF" ) { &$sub($$ref); }else{ print " VALUE\t$ref\n"; } print "\n"; }; foreach my $key (sort keys %$my) { print "$key:\n"; &$sub($$my{$key}); }
and a sample output
References and values of $my amatch: ARRAY /a/b/c/d/e/dummy.c /a/b /a/b/c/d/e/bar.pl /a/b/c/d/e/ABCDE.XYZ /a /a/b/c/d/e/foo.tst /a/b/c/d /a/b/c/d/e /a/b/c aonly: ARRAY /a/b/c/d/e/foo.tst /a/b/c/d/e/dummy.c /a/b/c/d/e/ABCDE.XYZ bmatch: ARRAY /a/b /a/b/c/d/e/bar.pl /a /a/b/c/d /a/b/c/d/e /a/b/c bonly: ARRAY common: HASH /a/b => /a/b /a/b/c/d/e/bar.pl => /a/b/c/d/e/bar.pl /a => /a /a/b/c/d => /a/b/c/d /a/b/c/d/e => /a/b/c/d/e /a/b/c => /a/b/c
The amatch and bmatch array references are those files and directory's in the adir and bdir structures that met the regex and negregex regular expression criteria. The aonly and bonly array references give those files and directories that exist only in that directory structure.
The common hash reference identifies those files and directories that exist in both dira and dirb directory structures. The key is for the dira, and value for dirb. Note that, depending on the nocase value the key and value may show differences in case on FAT and NTFS file systems.
A similar approach could be used to determine the referenced data from $ref. This would give access to
The following script can be called from a windows explorer prompt as the alternative to the windows delete function (the windows delete action might potentially be reversed by a replication). Obviously this will only function for file sand directories that are regularly synchronised using this module.
use strict; use warnings; my(@files) = @ARGV; END {print "\nDONE -- PRESS ENTER\n";<STDIN>}; print << "End_Of_Header"; ================================================================================ Executing $0\n This will mark files and/or directories for removal by the File::Repl file synchronisation utility. Files will be set to zero size when first processed by the File::Repl module, and finally removed after the tombstone period is expired. To reverse this process simply remove the added .remove file extension immediatly ================================================================================ End_Of_Header foreach my $file (@files){ print "$file\n"; if ($file =~ m/\.remove$/){ print "\t-is already marked for removal\n"; }elsif (-f $file){ unless (rename "$file","$file.remove"){ print "Unable to rename $file to $file.remove\n"; }else{ print "\t-marked for removal\n"; } } elsif (-d $file){ unless (rename "$file","$file.remove"){ print "Unable to rename $file to $file.remove\n"; }else{ print "\t-marked for removal\n"; } }else{ print "File $file not found !!\n"; } }
a hash of file names (the key) and values (mtime) of all files in the adir (or bdir) structure.
a hash of file names (the key) and values (file mode - from a stat operation) of all files in the adir (or bdir) structure.
In addition the scalar values of various settings determined when the New method is called can be determined.
Dave Roberts <droberts@cpan.org>
Thanks to Nigel Hodgson for his many contributions in developing this utility and helpin understanding fiel system specifics.
You can send bug reports and suggestions for improvements on this module to me at droberts@cpan.org. However, I can't promise to offer any other support for this package.
This module is Copyright � 2000 to 2010 Dave Roberts. All rights reserved.
This script is free software; you can redistribute it and/or modify it under the same terms as Perl itself. This script is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. The copyright holder of this script can not be held liable for any general, special, incidental or consequential damages arising out of the use of the script.
$Log: Repl.pm $ Revision 2.3 2015/11/03 18:07:45 Dave.Roberts interim release - corrections to help manage situations when a file and directory are compared
Revision 2.2 2015/11/01 22:36:38 Dave.Roberts removed Win32::Admin requirement from documentation
Revision 2.1 2015/07/15 20:51:29 Dave.Roberts added timestamp info for reading directories
Revision 2.0 2015/07/15 20:00:30 dave New major version, now with Win32::AdminMisc depandency removed as this module becomes more difficult to acquire and build for recent Perl releases
Revision 1.1 2015/07/15 19:58:06 Dave.Roberts Initial revision
Revision 1.31 2014/01/25 21:27:59 Dave.Roberts as advised from CPAN testing modified to include =encoding utf8 and escape the < and > characters in the pod with < and > respectively.
Revision 1.29 2010/05/04 15:02:05 Dave.Roberts corrected documentation - layout near Update method was incorrect
Revision 1.28 2010/04/27 14:55:00 Dave.Roberts minor code improvements in output messages for the Delete method
Revision 1.27 2010/04/13 08:36:52 Dave.Roberts added functionality for testing negative ages. This allows files older than the age specified to be selected (excluding all files younger)
Revision 1.26 2010/04/12 16:29:57 Dave.Roberts added Version method to return the File::Repl version corrected silly mistake in documentation - in definition of %con hash
Revision 1.25 2010/04/12 16:04:54 Dave.Roberts added example script for tombstoning removed windows linefeed characters from file
Revision 1.24 2010/04/07 02:00:11 Dave.Roberts modified code to remove the use of a hash as a reference - this was generating warnings as this use of a hash has beeen depreciated.
Revision 1.21 2002/02/07 10:37:39 Dave.Roberts corrected mode identified for Update method (the check used previously was invalid), and also synopsis for use of Update method (args incorrectly ordered)
Revision 1.20 2002/01/09 12:51:17 Dave.Roberts corrected errors in tombstoning of directories - subs $del and $deltree in particular
Revision 1.19 2001/11/21 21:28:19 Dave.Roberts resolved error in determining file age, especially when the 'a' file is missing evaluated the current time at start (set $runtime), and then removed many "time" calls
Revision 1.18 2001/08/22 07:10:41 Dave.Roberts logic change so that we don't use the Win32::API on win9x machines
Revision 1.17 2001/08/03 09:38:29 Dave.Roberts corrected code error (lines 572/3) where $$ was incorrectly used corrected code error (lines 572/3) where $$ was incorrectly used in truncation code
Revision 1.16 2001/08/02 22:09:02 Dave.Roberts corrected code for the Rename routine
Revision 1.15 2001/07/17 21:05:43 Dave.Roberts small changes to _arraysort - simplifying code
Revision 1.14 2001/07/12 21:51:50 jj768 additional documentation - and minor code changes
Revision 1.13 2001/07/12 15:18:43 Dave.Roberts code tidy up and reorganisation fixed logic errors (A>B! mode in Update method was not copying new files from A to B), also for A<B! removed several local variables and used referred object directly
Revision 1.12 2001/07/11 10:30:16 Dave.Roberts resolved various errors introduced in 1.11 - mainly associsated with reference errors rehacked fc subroutine - to give more logical messages still in need of more documentation - esp of object reference returned and associated variables
Revision 1.11 2001/07/06 14:52:53 jj768 double referencing of blessed object removed (from New method) and subsequent methods updated. Requires Testing. Update and other methods now return reference to data arrays and hashs evaluated during method call
Revision 1.10 2001/07/06 08:23:48 Dave.Roberts code changes to allow the colume info to be detected correctly using Win32::AdminMisc when a drive letter is specified (was only working with UNC names)
Revision 1.9 2001/06/27 13:35:53 Dave.Roberts minor presentation changes
Revision 1.8 2001/06/27 12:59:22 jj768 logic to prevent "Use of uninitialized value in pattern match (m//)" errors on use of $vol{FileSystemName}
Revision 1.6 2001/06/21 12:32:15 jj768
*** empty log message ***
Revision 1.5 2001/06/20 20:39:21 Dave.Roberts minor header changes
Revision 1.4 2001/06/20 19:55:21 jj768 re-built module source files as per perlmodnew manpage
Revision 1.1 2001/06/20 19:53:03 Dave.Roberts Initial revision
Revision 1.3.5.0 2001/06/19 10:34:11 jj768 Revised calling of the New method to use a hash reference, rather than a hash directly
Revision 1.3.4.0 2001/06/19 09:48:38 jj768 intermediate development revision. Introduced Delete method and the _generic subroutine (used for all methods except New) this is preparatory to the hash being passed as a reference
Revision 1.3.3.0 2001/06/14 15:42:48 jj768 minor code changes in constructing hash and improvement in documentation -still need more docs on Timezones.
To install File::Repl, copy and paste the appropriate command in to your terminal.
cpanm
cpanm File::Repl
CPAN shell
perl -MCPAN -e shell install File::Repl
For more information on module installation, please visit the detailed CPAN module installation guide.