upfiles -- upload files to FTP or SFTP server, for push mirroring
upfiles [--options] [filename...]
Upfiles uploads changed files from your local disk to an FTP or SFTP server, for a simple kind of "push" mirroring.
Create files locally with the same directory structure as the target, and in a ~/.upfiles.conf file give the locations,
upfiles (local => '/my/directory', remote => 'ftp://fred@example.com/pub/fred');
This is actually Perl code, so you can put comment lines with #, write some conditionals, use $ENV{HOME}, etc. Then to upload run
#
$ENV{HOME}
upfiles
Or to upload just some selected files,
upfiles /my/directory/foo.txt upfiles bar.txt ../something/quux.pl
Your username on the remote system is in the ftp:// remote URL. A password is taken from file ~/.netrc the same as for the ftp program and similar programs. See netrc(5) or Net::Netrc for the format of that file.
ftp://
ftp
Note that FTP transmits username and password in clear text over the network. See "FTP Security" below for secure connections (or "SSH File Transfer (SFTP)").
upfiles records files sent in an SQLite3 database file .upfiles.sqdb in each local toplevel directory, for example
/my/directory/.upfiles.sqdb
Local changes are identified by traversing the local tree and comparing file modtimes and sizes against what the database records was last sent. This is faster than asking the remote server what files etc it's got.
For convenience, some local files are always excluded from the upload. Currently these are
.upfiles.sqdb which is upfiles itself foo~ Emacs backups #foo# Emacs autosaves .#foo Emacs lockfiles
Files are uploaded one by one. The upload goes first to a temporary filename and is then renamed. This means an incomplete file isn't left if the connection is lost or upfiles is killed during transfer. Temporary files are noted in the database and leftovers are deleted on the next upfiles run if necessary.
File modification times are copied to an FTP server if it has the draft standard MFMT command or the common SITE UTIME extension (either 2-arg or 5-arg).
MFMT
SITE UTIME
Plain RFC959 FTP doesn't have a notion of symlinks or hard links so upfiles follows any local links to actual content to upload. Perhaps in the future the SITE SYMLINK extension, if available, could be used for links within the uploaded tree. (Or SFTP below has symlinks.)
SITE SYMLINK
Filenames containing CR or LF characters (\r, \n) cannot be sent by FTP protocol and Upfiles will refuse to operate on them.
Option passive => 1 selects "passive" mode FTP,
passive => 1
upfiles (local => '/my/directory', remote => 'ftp://fred@example.com/pub/fred', passive => 1);
This has the usual FTP meaning and is likely necessary when you're behind a firewall or IPv4 NAT which prevents you receiving incoming connections. The symptom under NAT might be a message from the server like
I won't open a connection to 192.168.0.1 (only to ...)
The hint is that 192.168... is a local private range address, unknown and unreachable in the outside world.
FTP can be used with SSL encryption and end-to-end destination verification in one of two ways. Either is highly recommended when supported by the server. Recent Net::FTP (version 1.28 circa 2014) and its SSL dependencies (IO::Socket::SSL) are required.
Net::FTP
ftps is immediate SSL on both command and data channels. The server listens on a different port 990 than plain FTP (port 21). Give ftps in the remote URL,
ftps
upfiles (local => '/my/directory', remote => 'ftps://fred@example.com/pub/fred');
Or, on the usual FTP connection a TLS command is given to encrypt the command channel (and data channels) from that point onwards. For this, give remote URL with plain ftp, then use_TLS option,
TLS
use_TLS
upfiles (local => '/my/directory', remote => 'ftp://fred@example.com/pub/fred', use_TLS => 1);
Upfiles gives TLS immediately on connecting to the server, so both username and password are only sent after the connection is encrypted with SSL.
See "DEBUGGING" below on protocol tracing.
The ssh "secure shell" program and protocol includes an SFTP file transfer. Upfiles can use this by setting the remote URL to sftp.
ssh
sftp
upfiles (local => '/my/directory', remote => 'sftp://fred@example.com/home/fred/mydir');
This requires Perl Net::SFTP::Foreign and suitable ssh(1) program. The remote URL target directory is a full filesystem path, in the form seen in the ssh shell on that system.
remote
ssh has various user authentication methods. No-password operation can be had by putting your public key on the remote machine. For password operation, upfiles looks in ~/.netrc for a password by username and host similar to plain FTP. If not found there then the ssh program will query. (Not sure whether querying is a good idea, might prefer to immediately abort.)
Each upfiles call in ~/.upfiles.conf takes the following key/value parameters,
local
The local directory to upload from.
The remote FTP or SFTP server to upload to, as a URL. The path in the URL is the target directory, and your username on the remote machine is included with "@" syntax, like
remote => 'ftp://fred@example.com/pub/fredsdir',
The "scheme" part can be ftp, ftps or sftp. A password is sought in ~/.netrc by username and machine name.
passive
For FTP (including ftps), if set to 0 then use "active" mode, or if 1 then use "passive" mode. Active might not work behind a firewall or IPv4 NAT. Passive should work in all cases.
For ftp remote, if set to 1 then use the TLS command to give an SSL secure protocol with the server. This is highly recommended when supported by the server (or equally good ftps or sftp).
For ftps, this option is ignored, as the connection is secured before any commands at all.
exclude_regexps
Additional filenames to exclude. For example to exclude a local Makefile
upfiles (local => '/my/directory', remote => 'ftp://example.com/pub/fred', exclude_regexps => [ qr{/(^|/)[Mm]akefile$} ]);
copy_utime
"if_possible"
Whether to copy file modification times to the server. Usually this is desirable. The default "if_possible" means do so if the server supports it. 0 means don't try. 1 means must copy times.
For FTP, this uses the MFMT or SITE UTIME commands, which are widely available extensions to the basic FTP protocol. For SFTP, copying times is always possible.
The command line options are
Show what would be uploaded or deleted on the server, but don't actually do anything.
upfiles -n
Print some brief help information.
Print some diagnostics about what's being done. With --verbose=2 or --verbose=3 print some technical details too.
upfiles --verbose
Print the upfiles program version number. With --verbose also print the version numbers of some modules used.
--verbose
The --verbose option above shows what upfiles is considering for uploads, and also enables various levels of debug or verbosity from Net::FTP (its Debug option), IO::Socket::SSL (its $DEBUG), and Net::SFTP::Foreign (-v to the ssh program). SSL and SSH diagnostics include information about certificates checked and encryption negotiated.
Debug
IO::Socket::SSL
$DEBUG
Net::SFTP::Foreign
-v
It can be helpful to check first that a connection to the remote works. For FTP this can be the usual ftp program, or even telnet (quit to exit).
telnet
quit
ftp example.com telnet example.com 21
Server support for TLS command can be checked in the usual way by the FEAT command (which in an ftp client program might have to be sent by quot feat) and looking for AUTH TLS in its response. The initial text banner message often says something for human readers too. The use_TLS option should then work, or not, according to server support.
FEAT
quot feat
AUTH TLS
For ftps (FTP with SSL on port 990), if no server is listening on that port then the host might either immediately say "connection refused", or might ignore so that upfiles times-out. If a suitable SSL-enabled ftp program is not at hand then raw telnet equivalent though SSL can be had with either
openssl s_client -connect example.com:990 gnutls-cli --verbose -p 990 example.com
For SSH SFTP, check ssh works by running it directly (give remote username with -l). Its -v can be repeated for ever greater verbosity.
-l
ssh -v -l fred example.com
On first connection to a new remote host, ssh will usually ask whether to accept the host certificate. This happens when run through upfiles too. Ideally you should have some separate independent way to validate or get a certificate so you know who you're talking to. Likewise SSL setups.
HOME
For ~/.upfiles.conf directory, and Net::Netrc for ~/.netrc.
Net::Netrc
Configuration file.
FTP password file.
SQLite3 database of information about what has been sent so far from the tree /my/local/dir.
Upfiles determines the home directory ~ using File::HomeDir. Net::Netrc has something similar for the .netrc file, and looks also for _netrc.
Changing a local file from a file to a directory or vice versa probably doesn't work very well. Remove it locally and upload to remove it at the remote, then create the new locally and upload again.
FTP and SFTP make a couple of round trip command/responses to the server for every file. When uploading many small files something streaming or parallel would be faster. The temp file and rename adds a round trip too, but is desirable so anyone looking doesn't see half a file. Perhaps an option could turn this off if not needed (such as upfiles to a remote backup). For reference, Net::SFTP::Foreign has its own atomic option in put(), but think it's still just a client side action, so recording temporaries in the database is a better cleanup if interrupted.
put()
The temporary files are named using the local $$ process ID added to the target filename. This is enough to protect against simultaneous uploads from the same source machine, but potentially not if you're networked and are foolish enough to upfiles the same tree simultaneously from two different source machines. For FTP, havoc probably ensues. For SFTP, no-overwrite is used so should make an error. FTP STOU would guarantee uniqueness, but does it have a window while the name comes back which if interrupted could leave the file created but unknown to upfiles? Net::FTP put_unique() doesn't return the name until the transfer completes too.
$$
STOU
put_unique()
For FTP, there's a small chance of a 2-arg SITE UTIME attempt looking like 5-arg to a pure-ftpd pre-1.0.33 server (circa 2011), if the filename uploaded happens to be dates and "UTC". Perhaps more care could be taken to identify the SITE UTIME style the server expects. In practice such filenames should be unlikely, and there's no problem with new enough pure-ftpd which has MDTM.
pure-ftpd
MDTM
There's no support for a proxy connection to the remote system. The squid CONNECT style which makes a pass-through socket wouldn't be hard, and ssh can be made to speak through such things too. The HTTP-only style, where HTTP PUT must be given for FTP upload, is irritating and makes the proxy a man-in-the-middle which you have to trust. If a HTTP PUT upload method was of interest then such proxying might follow from that easily enough.
File sizes and totals are reported in kbytes (of 1024 bytes). Probably should jump to mbytes or even gbytes at some sensible threshold.
For a given local directory, if the remote URL is changed then upfiles will want to send everything again. This is good if the remote is genuinely somewhere new, or if uploading a local directory to multiple remote places. But if it's just a change of hostname or protocol then would want no resending. Hopefully this is rare. There's a secret undocumented maybe-working --catchup for marking everything already sent. Some sort of force option to resend particular files could be useful too.
--catchup
The .upfiles.sqdb database records fake timestamps for directories just to indicate whether it has been created on the remote yet. Currently this is time "1 second" in the system epoch (1 Jan 1970 for Unix, whatever 1 Jan 1904 for MacOS). Alas this makes the database contents system-dependent. Hopefully cross-system operation will be rare and the effect on moving systems no worse than some repeated MKDs. Directory stamps are only actually for whether the directory has been created yet or not, so the intention is to treat any stamp value as meaning created.
MKD
Net::FTP, Net::SFTP::Foreign, ssh(1), netrc(5), Net::Netrc, DBD::SQLite, sqlite3(1)
sitecopy(1), ftpmirror(1), ftp-upload(1), rsync(1)
http://user42.tuxfamily.org/upfiles/index.html
(Upfiles is good for uploading to tuxfamily, either FTP or SFTP.)
Copyright 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016, 2017, 2020, 2023, 2024 Kevin Ryde
Upfiles is free software; you can redistribute it and/or modify it under the terms of the GNU General Public License as published by the Free Software Foundation; either version 3, or (at your option) any later version.
Upfiles is distributed in the hope that it will be useful, but WITHOUT ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU General Public License for more details.
You should have received a copy of the GNU General Public License along with Upfiles. If not, see http://www.gnu.org/licenses/.
To install App::Upfiles, copy and paste the appropriate command in to your terminal.
cpanm
cpanm App::Upfiles
CPAN shell
perl -MCPAN -e shell install App::Upfiles
For more information on module installation, please visit the detailed CPAN module installation guide.