Revision History -- ECS, the EMDIS Communication System
Version 0.01, 2003-03-28
========================
- Initial release (internal).
Version 0.02, 2003-04-30
========================
- Various tweaks, enhancements, and bug fixes.
- Added ecs_setup script.
Version 0.03, 2003-05-09
========================
- small tweaks and bug fixes
- more automated unit/regression tests
- more documentation
- outgoing message processor example (see "perldoc ecstool")
Version 0.04, 2003-10-09
========================
- Added ecs_pid_chk script
- If decryption fails on an incoming message, ecs_scan_mail now
executes process_message() using message body as cleartext
- Fix problem whereby ecs_chk_com and ecs_scan_mail scripts would
mistakenly delete pid file if daemon was already running
Version 0.05, 2004-01-19
========================
- minor adjustments to POD documentation
- improve error messages generated by ecs_proc_meta script
- limit number of iterations through email inbox performed by scan_mail()
subroutine of ecs_scan_mail script
Version 0.06, 2004-03-02
========================
- ecstool: modify --meta option, adding capability to generate a series of
multiple MSG_ACK or RE_SEND meta-messages
- ecs_scan_mail: modify process_store() to log_info() when generating RE_SEND
- ecs_scan_mail: modify process_store() to move_to_trash() duplicate seq_num
message files encountered while scanning ECS_MBX_STORE_DIR directory
Version 0.07, 2004-03-19
========================
- add MS Windows support for Windows 2000 and Windows XP
(code modifications provided by Canadian Blood Services - Tony Wai)
- add DIRECTORY inbox protocol
(code modifications provided by Canadian Blood Services - Tony Wai)
Version 0.08, 2004-03-22
========================
- make adjustments to fix problems running under Windows
- modify ECS::timelimit_cmd_win32() to make it a bit more robust
Version 0.09, 2004-03-25
========================
- more adjustments to ECS::timelimit_cmd_win32()
Version 0.10, 2004-04-02
========================
- ecstool: modify usage message
- LockedHash: revise _lock() to use blocking for unix, polling for win32
Version 0.11, 2004-05-14
========================
- modify formatting of result returned by ECS::timelimit_cmd_unix()
- add T_ADM_DELAY configuration parameter
Version 0.12, 2004-05-19
========================
- ecs_scan_mail: add quick hack to help prevent single node from monopolizing
message processing loop
Version 0.13, 2004-05-24
========================
- ecs_scan_mail: fix minor problem with hack introduced in version 0.12
Version 0.14, 2004-05-24
========================
- ecs_scan_mail: improve SIGINT & SIGTERM signal handling
Version 0.15, 2004-06-02
========================
- ecs_chk_com: don't check comm status with self
- ecs_scan_mail: add runtime status reporting features (write status info
to NODE_TBL while running)
Version 0.16, 2004-06-09
========================
- ECS.pm: add POD documentation for q_first_file, q_max_seq, q_min_seq,
q_size, proc_file, proc_node, proc_seq
- ecs_scan_mail: minor adjustment to "fair queueing" system, to make it
a little more fair (reverse nodelist on even numbered scan interval)
Version 0.17, 2004-06-21
========================
- ecstool: add --prune option, for cleaning out mboxes/store directory
- ecs_chk_com: send MSG_ACK if node has in_seq_ack < in_seq,
disable chk_com() if node_disabled status is YES or TRUE
- ecs_scan_mail: only send MSG_ACK if $ECS_CFG->ALWAYS_ACK is YES or TRUE,
disable process_store() if node_disabled status is YES or TRUE
Version 0.18, 2004-07-07
========================
- add generate_pdf.sh script
- remove ECS.pm dependency from LockedHash.pm, so LockedHash.pm loads faster
when used separately (e.g. ecs_status.pl CGI script)
Version 0.19, 2004-07-12
========================
- ecstool: add --maildrop command
Version 0.20, 2004-07-13
========================
- ecstool and ECS.pm: use file locking to avoid potential undetected race
condition when handling maildrop files
Version 0.21, 2004-07-14
========================
- ecstool: initialize all node properties during --add,
modify --send sanity check regex to be more permissive
- ecs_proc_meta: set $node->{last_in} when processing RE_SEND message
- add web_status/ecs_status.pl to MANIFEST
Version 0.22, 2004-08-02
========================
- generate_pdf.sh: adjust regex used to massage "<email>" addresses
(these had been improperly handled by pod2latex on the NMDP's system)
- ecs_status.pl: for disabled node(s), add visual indication of
node_disabled status; modify generated HTML to use document-level
style sheet instead of <b>, <u>, <font>, <h2>, <tt>, bgcolor, align, etc.
- ecstool: add --archive command
Version 0.23, 2004-08-03
========================
- ecstool: modify --archive command -- adjust error handling, add progress
output, and fix a minor bug
Version 0.24, 2005-03-08
========================
- Config.pm: change default GNU_TAR value to /usr/bin/tar
- config.t: add a few more tests for Config.pm
- revise method for automatic generation of RE_SEND requests; add
T_RESEND_DELAY feature (modified files: Config.pm, ecs_scan_mail,
ecs_setup, ecstool, config.t)
Version 0.25, 2005-03-11
========================
- ecs_scan_mail: fix some broken error messages
- ecs_scan_mail: send_admin_email() when issuing RE_SEND requests
- ecstool: define q_gap_seq, q_gap_time, and contact when adding new node
- ECS.pm: document NODE_TBL fields q_gap_seq, q_gap_time, and contact;
revise RE_SEND protocol description
- ecs_chk_com: document sending of MSG_ACK meta-messages
Version 0.26, 2005-07-13
========================
- ecstool: adjust some POD comments
- README: add example for running Perl ECS under a unix home directory
- ECS.pm: modify subject line generated by send_admin_email() to show
configured MAIL_MRK instead of hard-coded "EMDIS"
- Config.pm: allow config file to omit either PGP2 or OpenPGP configuration
Version 0.27, 2005-09-01
========================
- Message.pm: fix problem parsing messages containing multiple \r characters
- ecs_scan_mail: retain message for later retry if "Unable to open SMTP
connection" error encountered while attempting to send message from
maildrop directory; otherwise, notify admin more explicitly when a message
is discarded
Version 0.28, 2005-11-09
========================
- ecstool: add optional node parameter for --send command
- ECS.pm: generate error message in send_ecsmsg_email if node addr not defined
Version 0.29, 2007-06-13
========================
- ECS.pm: modify embedded documentation to mention DIRECTORY method for
retrieval of incoming email
- ECS.pm: modify embedded documentation to mention new NODE_TBL field:
ready_num_disabled -- for EMDIS IP4
- ecs_chk_com: modify READY meta-message to include last_recv_num and
last_sent_num (in_seq, out_seq) -- for EMDIS IP4
- ecstool: modify READY meta-message to include last_recv_num and
last_sent_num (in_seq, out_seq) -- for EMDIS IP4; also, modify --view
command to accept optional node parameter and modify --tweak command
to display settings for only the affected node
- ecs_scan_mail: add information to "Message ... exceeds maximum size
limit" error message
- add LICENSE file
- add and modify copyright notices
Version 0.30, 2008-04-21
========================
- Change perl path from /usr/local/bin to /usr/bin.
- Change position of 'use strict' to the end of the block
- MANIFEST: add public domain image files for flags and status LEDs;
also add ECS logo image, ecs_ctl script, ecs_status wrapper script,
and several new data files for automated tests
- README: modify installation example to mention use of sudo for make
install, add note about original sources for image files
- generate_pdf.sh: add ecs_ctl
- ECS.pm: revise method for handling log messages, so it also sends admin
email as configured via the MAIL_LEVEL configuration parameter; this
eliminates the need for other code to directly invoke the
send_admin_email subroutine
- ECS.pm: revise send_ecsmsg_email subroutine to update node_tbl after
sending email, instead of before
- ECS.pm: modify regular expressions for parsing the email subject line,
to accept a subject line containing spam flags preceding the ECS mail_mrk
- Config.pm: add new configuration parameters: LOG_LEVEL, MAIL_LEVEL,
ECS_TO_DIR, ECS_FROM_DIR
- LockedHash.pm: use $ECS::VERSION, instead of a separate local $VERSION
- Message.pm: modify regular expressions for parsing the email subject line,
to accept a subject line containing spam flags preceding the ECS mail_mrk
- ecs_chk_com: optionally use ECS_CONFIG_FILE environment variable to
specify location of ECS configuration file
- ecs_chk_com: use log_info and log_error instead of log_admin_email
- ecs_chk_com: remove unnecessary (obsolete) send_admin_email calls
- ecs_chk_com: remove code that inserted a pseudo-random number in meta
messages
- ecs_ctl: new tool (ecs control) to ease administration; features are
start | stop | restart | status (PID)
- ecs_pid_check: modify script to support both Solaris and Linux
- ecs_proc_meta: log_info when processing RE_SEND message
- ecs_scan_mail: optionally use ECS_CONFIG_FILE environment variable to
specify location of ECS configuration file
- ecs_scan_mail: if the ECS_TO_DIR configuration parameter is defined,
use the new process_to_XX procedure instead of process_maildrop
- ecs_scan_mail: for non-ECS message, write message content to file and
email only the file location, not the complete message, to ECS admins;
log this occurrence as info instead of error
- ecs_scan_mail: use log_info and log_error instead of log_admin_email
- ecs_scan_mail: remove unnecessary (obsolete) send_admin_email calls
- ecs_scan_mail: in process_maildrop, exit processing loop ("last" instead
of "next") after encountering an error, to avoid potential problems
- ecs_scan_mail: in process_store, change log level for duplicate message
from error to warn
- ecs_scan_mail: in process_store, if decryption fails, don't automatically
attempt to process the message as clear text
- ecs_scan_mail: in process_message, use ECS_FROM_DIR, if configured
- ecs_setup: prompt for LOG_LEVEL and MAIL_LEVEL configuration parameters
- ecs_setup: if using optional to_XX and from_XX feature, prompt for
ECS_TO_DIR and ECS_FROM_DIR
- ecs_setup: mention ECS_CONFIG_FILE environment variable
- ecs_setup: fix some typos in generated configuration file
- ecstool: optionally use ECS_CONFIG_FILE environment variable to specify
location of ECS configuration file
- ecstool: add --export command to export node_table contents to text file
- ecstool: add --get command to view a property across one or all nodes
- ecstool: modify --add to create new node's ECS_TO_DIR and ECS_FROM_DIR
subdirectories as needed
- ecstool: modify --archive to include log files and ecs config file
- ecstool: modify --delete to delete node specific FROM_DIR and TO_DIR
- ecstool: modify --view; if keys specified on command line, don't sort them
- ecstool: modify --tweak; call check_prop() to check property
- ecs_status: new wrapper script to set environment variables and
execute ecs_status.pl
- ecs_status.pl: add Info column with node specific status details
- ecs_status.pl: add node specific (green/yellow/red/black) status LED
- ecs_status.pl: add optional green/yellow/red background colors
- ecs_status.pl: add optional status fields for the ECS daemons
- ecs_status.pl: add ECS logo favicon
- ecs_status.pl: revise script to be configurable via environment variables
Version 0.31, 2011-01-11
========================
- ECS.pm: add code to process comma-separated list of email recipients
- ECS.pm: add mention of msg_part_size in POD documentation
- ECS.pm: add EOL subroutine to return platform-dependent EOL string
- ECS.pm: modify read_ecs_message_id to return part_num and num_parts
- ECS.pm: add $FILEMODE to EXPORT_TAGS
- ECS.pm: guard against possibility of infinite recursion between log_error
and send_admin_email
- Config.pm: revise value assigned to ECS_DRP_DIR if ECS_TO_DIR is being used
- Config.pm: add MSG_PART_SIZE_DFLT configuration parameter
- Message.pm: add part_num and num_parts, modify code that parses Subject line
- ecs_chk_com: add check for missing command line arguments
- ecs_chk_com: add --help option to show embedded POD documentation
- ecs_ctl: add check for missing command line arguments
- ecs_proc_meta: add --help option to show embedded POD documentation
- ecs_proc_meta: send email to admin if error encountered during RE_SEND
- ecs_proc_meta: modify RE_SEND processing to handle part_num, if specified
- ecs_proc_meta: check for seq_num > node->out_seq when processing RE_SEND
- ecs_scan_mail: eliminate error warning when ECS_FROM_DIR not configured
- ecs_scan_mail: add --help option to show embedded POD documentation
- ecs_scan_mail: modify scan_mail subroutine to delete inbound messages where
sender is this node or is not really a node
- ecs_scan_mail: use FileBackedMessage instead of send_fmlfile_email
- ecs_scan_mail: refactor code to define update_statistics subroutine, use
this to update statistics at beginning and end of process_store
- ecs_scan_mail: assemble message parts for inbound messages, as described
by EMDISCORD RFC
- ecs_setup: add MSG_PART_SIZE_DFLT configuration parameter
- ecs_status.pl: add all_red feature to visibly indicate complete
communication loss
- ecs_status.pl: adjust and fix some error warnings
- ecs_status.pl: use ECS_DAEMON_USER as file name prefix for copied node_tbl
files
- ecstool: for --archive function, include ecs_proc_meta.log, don't leave
rotated log files in ECS_DAT_DIR (only archive them), and send admin email
if unable to archive a log file
- ecstool: modify --send option to use FileBackedMessage, to support very
large messages required by EMDISCORD RFC
- ecstool: modify --meta RE_SEND processing to handle seq_num:part_num format
- add 22px and 60px flags for Slovakia (SK)
- add FileBackedMessage.pm: new message object for working with large messages
- generate_pdf.sh: add lib/ECS/FileBackedMessage.pm
- flag_GL_logo.png: updated image, now based on favicon.ico
Version 0.32, 2012-11-30
========================
- correct error in ZKRD URL: change all instances of http://zkrd.de/ to
http://www.zkrd.de/
- Changes: revise version 0.31 date; change from 2010-09-03 to 2011-01-11
- LICENSE: update copyright notice for 2012
- ecs_chk_com: enable program to be run without any command-line parameters
- ecs_scan_mail: add explicit "undef $msg" steps in process_maildrop() and
process_to_XX(), to avoid problem deleting message file under Windows
- ecs_scan_mail: revise process_maildrop() subroutine to remove code that
automatically discards maildrop files
- LockedHash.pm: revise lock() subroutine to add retry loop
- FileBackedMessage.pm: add send_email to ECS import list, to fix problem
using "ecstool --send" to send META message file
- ecs_status.pl: fix some bad HTML; change </td> to </th> in table header
Version 0.33, 2015-07-09
========================
- FileBackedMessage.pm: to support proxy RFC, remove code from
send_via_email method that double-checks HUB_SND and HUB_RCV (fields
populated by inspect_fml method)
- ecs_chk_com: to capture unusual (externally caused) runtime errors,
enclose main processing loop in eval{} block
- ecs_scan_mail: to capture unusual (externally caused) runtime errors,
enclose main processing loop in eval{} block
- ecs_scan_mail: in process_maildrop subroutine, add code to log a
warning saying that maildrop feature is deprecated
- ecs_scan_mail: in process_maildrop subroutine, don't send message if
receiving node is disabled
- ecs_scan_mail: in process_to_XX subroutine, don't process folder if
receiving node is disabled
- ecs_scan_mail: in process_message subroutine, add workaround for
potential race condition involving files in ECS_FROM_DIR
- ecs_scan_mail: revise mbox_open subroutine to enable use of SSL on
the POP3 or IMAP server connection
- ECS.pm: remove code that creates unnecessary IO::Handle instances
- ECS.pm: revise send_email subroutine to enable use of SSL on the
SMTP server connection
- LockedHash.pm: tweak error messages
- Config.pm: add support for new configuration settings:
INBOX_USE_SSL, SMTP_PASSWORD, SMTP_PORT, SMTP_USE_SSL, SMTP_USERNAME
- ecs_setup: add support for new configuration settings:
INBOX_USE_SSL, SMTP_PASSWORD, SMTP_PORT, SMTP_USE_SSL, SMTP_USERNAME
- ecs_setup: change default outbound mail storage option from [m] (maildrop)
to [t] (to_XX and from_XX)
- Dockerfile.ubuntu: Dockerfile for docker image based on Ubuntu 14.04
- Dockerfile.centos: Dockerfile for docker image based on CentOS 7
Version 0.34, 2015-08-31
========================
- ecstool: add --nodedata command, for importing and exporting node table
- ecs_setup: fix problem that could cause ECS_TO_DIR and ECS_FROM_DIR to be
omitted from output file when default mail storage option is selected
- Changes.pm: remove ADAPTER_CMD from the list of required attributes
Version 0.35, 2016-02-05
========================
- ecs_scan_mail: revise process_message subroutine to enable simultaneous
usage of FROM_DIR and ADAPTER_CMD configuration options
- ecs_scan_mail: in mbox_open subroutine, set $mbox->Ignoresizeerrors(1)
when using IMAP protocol
- MANIFEST: under web_status/images, add 22px and 60px flag images for CE,
DL, GR, HR, IN, LT, MK, RO, SI, UK, and WB
- Makefile.PL: add CPAN::Version to PREREQ_PM
Version 0.36, 2016-02-29
========================
- Change namespace from ECS to EMDIS::ECS, so this module can later be
added to CPAN without needing to occupy a top-level namespace.
- ecs_setup: revise descriptions of INBOX_PASSWORD and SMTP_PASSWORD
to say "password" instead of "user name"
- ecs_setup: if available, take ECS_TO_DIR and ECS_FROM_DIR from existing
configuration file
- Config.pm: add brief ADAPTER_CMD example to embedded documentation
Version 0.37, 2016-09-28
========================
- Dockerfile.centos: install EMDIS::ECS from CPAN instead of using tarball
- Dockerfile.ubuntu: install EMDIS::ECS from CPAN instead of using tarball
- ECS.pm: in send_mail subroutine, add MIME and encoding headers, so MTAs
like Microsoft Exchange should not change the encoding
- Message.pm: use MIME::QuotedPrint to automatically decode messages which
were encoded during transmission
- ecs_scan_mail: fix some log messages which were not correctly
interpolated or produced errors, and adjust usage of ECS::Message
Version 0.38, 2016-11-30
========================
- ECS.pm: in openpgp_decrypt, openpgp_encrypt, pgp2_decrypt, and pgp2_encrypt
subroutines, revise code that composes encryption commands to handle multiple
occurrences of special symbols such as __INPUT__, __OUTPUT__, __RECIPIENT__,
and __SELF__
- Config.pm: add "--local-user __SELF__" to OPENPGP_CMD_ENCRYPT default value
- LICENSE: revise copyright notice for 2016
Version 0.39, 2016-12-05
========================
- ecs_setup: change GPG_HOMEDIR default value to /home/username/gpg and
PGP_HOMEDIR default value to /home/username/pgp
- Config.pm: add "-u __SELF__" to PGP2_CMD_ENCRYPT default value
- ECS.pm: add encr_out_keyid and encr_out_passphrase properties to NODE_TBL;
add encr_out_keyid and encr_out_passphrase parameters to
send_encrypted_email, openpgp_encrypt, and pgp2_encrypt subroutines; add
encr_out_passphrase parameter to openpgp_decrypt and pgp2_decrypt subroutines
- FileBackedMessage.pm: revise usage of send_encrypted_email subroutine
- Message.pm: revise usage of openpgp_decrypt and pgp2_decrypt subroutines
- ecstool: add encr_out_keyid and encr_out_passphrase properties to NODE_TBL;
revise usage of openpgp_decrypt and pgp2_decrypt subroutines; adjust
formatting of ecstool --view output
Version 0.40, 2018-03-19
========================
- ecstool: revise regex to avoid "Unescaped left brace in regex is illegal"
error
- Config.pm: add INBOX_USE_STARTTLS and SMTP_USE_STARTTLS settings;
assign default INBOX_PORT and SMTP_PORT
- ecs_setup: add INBOX_USE_STARTTLS and SMTP_USE_STARTTLS settings; allow
GPG_PASSPHRASE to be the empty string (don't initialize with 'gpgpg')
- ECS.pm: add subroutines is_yes and is_no; add support for
SMTP_USE_STARTTLS; perform SMTP authentication only if both
SMTP_USERNAME and SMTP_PASSWORD are configured; eliminate compile-time
circular dependency between ECS.pm and other modules (Config.pm,
LockedHash.pm); revise openpgp_decrypt and openpgp_encrypt subroutines
to handle empty GPG_PASSPHRASE (avoid broken pipe error)
- ecs_scan_mail: add support for INBOX_USE_STARTTLS
- Dockerfile.centos: enable step that installs modules from CPAN into
local-lib directory; various other tweaks and additions; move to
docker/centos/Dockerfile
- Dockerfile.ubuntu: update to use Ubuntu 16.04; move to
docker/ubuntu/Dockerfile
- docker/dist/Dockerfile: new Dockerfile to build distribution artifacts
and to facilitate testing of email communications
- docker/greenmail/Dockerfile: new Dockerfile to provide SMTP, POP3, and
IMAP email services for testing of email communications
Version 0.41, 2020-12-08
========================
- add Python scripts for AMQP 1.0 messaging using Apache Qpid Proton:
ecs_amqp_recv.py, ecs_amqp_send.py
- docker/ubuntu: update to use Ubuntu 18.04; add Python 3 and Python
Qpid Proton to enable use of AMQP 1.0 scripts
- docker/centos: add Python 3 and Python Qpid Proton to enable use of
AMQP 1.0 scripts
- docker/qpid-broker-cpp: add Docker setup for AMQP broker using
Apache Qpid Broker-CPP (via RHEL/CentOS packages from EPEL repo)
- docker/dist: add test nodes DD and EE for testing AMQP communications,
along with AMQP configuration for test node AA, and supporting resources
- Config.pm: add AMQP-related configuration settings; enable use of
INBOX_PROTOCOL "NONE"
- FileBackedMessage.pm: enable use of either email or AMQP to send message,
deprecate inspect_fml subroutine, make modifications to support document
exchange
- Message.pm: make modifications to support document exchange
- ECS.pm: enable use of either email or AMQP to send messages, enable
use of custom properties/headers in AMQP/email messages, make
modifications to support document exchange
- ecs_scan_mail: enable use of either email or AMQP to send and receive
messages, add process_document subroutine, make modifications to support
document exchange
- ecs_proc_meta: enable use of either email or AMQP to send messages,
add support for DOC_MSG_ACK and DOC_RE_SEND meta messages
- ecs_chk_com: enable use of either email or AMQP to send messages
- ecstool: enable use of either email or AMQP to send messages; define
AMQP-related properties usable in node table, to enable node-level
customization of AMQP parameters, add support for DOC_MSG_ACK and
DOC_RE_SEND meta messages
- ecs_setup: add support for AMQP-related configuration settings
Version 0.42, 2020-12-19
========================
- ECS.pm: revise code to avoid "Experimental keys on scalar is now forbidden"
error using newer versions of Perl
Version 0.43, 2021-04-21
========================
- ECS.pm: in send_encrypted_message(), fix defect causing AMQP document to be
sent via regular AMQP message queue
- FileBackedMessage.pm: in send_this_message(), when sending unencrypted META
message, guard against sending via email if node is configured for amqp_only
- MANIFEST: add docker/ubuntu/README_GPG
- greenmail/Dockerfile: update to use greenmail/standalone:1.6.3
- generate updated certfiles.tar.gz files for dockerized test setup
- adjust various Dockerfile comments
Ideas for Future Enhancements
=============================
- send admin message when slow message status > 24 hours
- add --reprocess feature(s) to ecstool
- build PPM package
- add more automated tests, especially for complex operations involving
encryption and email
- add option to use MIME (RFCs 2045-2049) for encrypted email instead of the
existing basic ASCII Armor format (use Email::MIME?)
- refactor and clean up code
- improve method for handling encr_typ; allow for use of GnuPG with PGP2
protocol
- modify handling of incoming messages to send admin email and not
process message if subject line indicates message is from THIS_NODE
- add option to tweak scan_mail loop in ecs_scan_mail, to insert 5 second
delay and increase maximum loop count to 60 (i.e. what Australia did
to accommodate Microsoft Exchange)
- add option to tweak mbox_get_message function in ecs_scan_mail, to use
MIME::QuotedPrint decode_qp function (another Microsoft Exchange
workaround)
- add ecstool option to process a cleartext file
- allow for automatic reprocessing of self-signed message files
- add ecstool option to decrypt and assemble a logical message consisting
of multiple physical messages