CGI::UploadEngine - File Upload Engine for Multi-App Web Server
This document describes CGI::UploadEngine version 0.9.3
The main design goal of CGI::UploadEngine (CGI::UE) has been to enable developers to use file upload select boxes in a regular HTML form without needing to handle multi-part forms (or making any number of rookie mistakes). This is accomplished by injecting the necessary code into the developers template to submit the form twice; once to upload the file, and a second time to submit the rest of the data along with an acknowledgement token. This means that the "Upload Engine" (UE) handles all of the html and javascript necessary to upload, validate (and optionally limit) file type, size, and destination directory.
The UE includes the four main methods of CGI::UE and a mysql table. The two middle methods are meant to be implemented in a "Core" controller/script that handles the actual file upload. This package currently only includes working code for a Catalyst supported Core. If you do not have Catalyst, please get it, wait for an update, or send us your working CGI Core to include in the next update.
CGI::UE can be installed with make, but the UE Core should be installed by a server administrator (currently with Catalyst) for the benefit of developers on their server (either developing with Catalyst or "doing it old school"). The developers will use both the first and last CGI::UE methods and the installed UE Core to painlessly create forms that can handle both file and field data, even in "old school" CGI scripts.
Normally, file select boxes have to be in a form with a special multi-part attribute which can significantly complicate handling other values in the same form. By handling the file upload with AJAX, we can replace the file select box with a simple (hidden) text field (with our "success token" pre-filled) that reduces file uploading to just another dynamic field to be filled in a template.
<form action="[% action_url %]" method="POST"> <table cellspacing="2" cellpadding="0" width="600"> <tr> <td> Select file </td> <td> [% file_upload %] </td> </tr> <tr> <td> Your fields here </td> <td> <input type="text" name="other" id="other" value=""> </td> </tr> <tr> <td> </td> <td> <input type="submit"> </td> </tr> </table> </form>
With the template in hand, we only need to inject the UE "Package". The Package contains all of the HTML and most of the JavaScript necessary to upload the file to your specified directory. (Additional JavaScript is used from the Ext library.)
However, additional work is required to make a controller/script. Below is text of form_eg.cgi, which is included in the CGI::UE package. Notice it uses Template Toolkit (TT). It also requires a script level configuration file, template file, and library file.
use CGI::UploadEngine; use Template; require "upload_cfg.pl"; require "upload_html.pl"; # Create an upload object my $upload = CGI::UploadEngine->new(); # Set the view template my $tmpl = "upload/form_eg.tt2"; # Set the template variables my $vars = { action_url => $root_url . "cgi-bin/handler_eg.cgi", file_upload => $upload->upload_prepare({ file_path => "/tmp" }) }; # Process the template my $result; my $template = Template->new({ INCLUDE_PATH => $tmpl_incl }); $template->process( $tmpl, $vars, \$result ); # Print the page header(); print $result; footer();
In the above script, the functions header() and footer() are from the library file upload_html.pl (as is parse_form_data() in the script below). (Try reading the short configuration file.)
Note: Page layout could be handled through TT. You could grab the CGI form data another way. Database passwords should be in file X. Bleep blurp bleep. In short, these CGI scripts are working samples of developer software, not final software.
Once the end user selects a file, it is submitted via AJAX. On success, the form is altered. The file upload box is removed and replaced with a regular text box containing the returned success token. As a developer, you will need to retrieve the file info using the token to discover the file-path-name and other pertinant facts.
Below is text of handler_eg.cgi, which is also included in the package.
use CGI::UploadEngine; use Template; require "upload_cfg.pl"; require "upload_html.pl"; # Handle CGI variables parse_form_data(); my $token = $FORM_DATA{token}; my $other = $FORM_DATA{other}; # Create an upload object my $upload = CGI::UploadEngine->new(); # Retrieve file hash_ref my $file = $upload->upload_retrieve({ token => $token }); # Set the view template my $tmpl = "upload/handler_eg.tt2"; # Set the template variables my $vars = { path_name => $file->{file_path} . "/" . $file->{file_name}, other => $other }; # Process the template my $result; my $template = Template->new({ INCLUDE_PATH => $tmpl_incl }); $template->process( $tmpl, $vars, \$result ); # Print the page header(); print $result; footer();
The sample script prints the values on a page. That is probably not what you will do with it. Notice however that "other" form values are passed along with the token, which is used to retrieve file information in a perl hash reference.
CGI::UE has four public methods (besides new()). The first and last are "developer methods", and are used by the person who wants to create a form mixing a file select box with other more common elements like text input, textareas, radio buttons, or a normal select box. The middle methods are "Core methods", and are accessed via a running URL pre-installed by the server administrator.
Besides mastering these calls, developers must include the JavaScript library reference in the page header. CGI::UE uses the Ext library for cross-browser support.
new()
Create a new CGI::UE object.
# Create an upload object my $upload = CGI::UploadEngine->new();
The returned object is used to access the next four methods.
upload_prepare()
The destination directory and file limits are passed to the engine using upload_prepare(), which returns a text string containing two HTML form elements and javascript.
my $injection = $upload->upload_prepare({ file_path => '/tmp' }) };
If you have a template with your caption and a cell for the file upload box:
<tr> <td> Select file </td> <td> [% file_upload %] </td> </tr>
You would fill it by assigning the results of upload_prepare() to the file_upload key:
# Set the template variables my $vars = { action_url => $root_url . "cgi-bin/handler_eg.cgi", file_upload => $upload->upload_prepare({ file_path => "/tmp" }) };
Example results of a generated Package are shown below.
upload_retrieve()
However you retrieve the CGI variable, it is named "token".
# Handle CGI variables parse_form_data(); my $token = $FORM_DATA{token};
Use the token to retrieve a hash reference with relevant results.
# Retrieve file hash_ref my $file = $upload->upload_retrieve({ token => $token }); my $vars = { path_name => $file->{file_path} . '/' . $file->{file_name} };
Use the file information to further your nefarious schemes.
So far unmentioned is the fact that the Package injected into the template includes an "attempt token". When a file is selected by an end user, the JavaScript submits the attempt token along with the file to the server UE Core via AJAX. The Package includes the correct URL for UE Core, and swaps the Core URL with your URL before submitting the form for the first time. On success, your URL is replaced along with new HTML code that includes the success token.
upload_validate()
$file_obj = $upload->upload_validate({ token => $token });
A Core implementation first calls upload_validate() with the attempt token. The returned file object has relevant details for the Core to validate the file and save it to the appropriate directory.
upload_success()
$success = $upload->upload_success({ token => $token, file_name => $file_name, file_size => $file_size});
If the file is valid and uploads without fail, the Core updates the name and size, and returns a success token.
Tokens are generated simply using alphanumerics and underscore, through rand().
our $token_length = 60; our @token_chars = ("a".."z","A".."Z","0".."9","_"); # Random string created from global package variables foreach (1..$token_length) { $token .= $token_chars[rand @token_chars]; }
The following is included as an accessible reference, and is not meant to be copied or used directly. The code below was generated by the private method _generate_html().
<tr> <td> Select file </td> <td> <script type="text/javascript"> Ext.onReady(function() { var file_upload_box = Ext.get("auto_file"); var auto_file_form = document.forms[0]; var action = document.forms[0].action; var encoding = document.forms[0].encoding; var file_id; document.forms[0].action = "http://DOMAIN/upload"; document.forms[0].encoding = "multipart/form-data"; file_upload_box.addListener("change", function(el) { Ext.Ajax.request({ form: auto_file_form, success: function(response, opts) { if(!response.responseText.match(/ERROR: /)){ file_id = Ext.util.JSON.decode(response.responseText).success; Ext.fly(file_upload_box).remove(); Ext.fly(file_control).insertHtml("afterBegin", "<img src="http://DOMAIN/images/checked.jpg"> File Uploaded"); var token = document.getElementById("token"); token.value = file_id; document.forms[0].action = action; document.forms[0].encoding = encoding; }else{ //we have an error. Ext.fly(file_upload_box).remove(); //if possible decode the JSON try{ file_id = Ext.util.JSON.decode(response.responseText).success; Ext.fly(file_control).insertHtml("afterBegin", "<img src="http://DOMAIN/images/alert.jpg">"+file_id); }catch(e){ //JSON decode failed so use responseText Ext.fly(file_control).insertHtml("afterBegin", "<img src="http://DOMAIN/images/alert.jpg">"+response.responseText); } } }, failure: function(response, opts) { alert("server-side failure with status code " + response.status); } }); }); }); </script> <div name="file_control" id="file_control"> <input type="file" size="50" name="auto_file" id="auto_file"> <input type="hidden" name="token" id="token" value="xGgbwIKjcM_SJ75uAvwwg4f3OtpQjcrULSmiUuXnSqYaFeZ8LoVxJUgrVg9a"> </div> </td> </tr>
The elements are wrapped in a div named "file_control", which is used as the anchor to replace the elements with the success token.
These errors are meant to appear on the upload page if an upload was not successful. If you get one of these as an exception, then it was not caught correctly.
<"ERROR: Failed to copy '%s1' to '%s2': %s3">
<"ERROR: Failed to copy '%s1' to '%s2': %s3"
This error occurs when the uploaded file could not be copied. It is most likely a file system issue.
%s1 - file name %s2 - target location %s3 - error string
<"ERROR: file size $file_size is smaller than min size $min_size">
<"ERROR: file size $file_size is smaller than min size $min_size"
The size of the file that was to be uploaded was smaller than the minimum size set in the configuration file or passed to IOSea::Upload::upload_prepare.
<"ERROR: file size $file_size is larger than max size $max_size">
<"ERROR: file size $file_size is larger than max size $max_size"
The size of the file that was to be uploaded was larger than the minimum size set in the configuration file or passed to IOSea::Upload::upload_prepare.
<"ERROR: file type $type not allowed. Allowed types are: $allowed_types">
<"ERROR: file type $type not allowed. Allowed types are: $allowed_types"
The type of the file that was to be uploaded was not listed in the allowed types set in the configuration file or passed to IOSea::Upload::upload_prepare.
<"ERROR: file type $type is forbidden. Forbidden types are: $disallowed_types">
<"ERROR: file type $type is forbidden. Forbidden types are: $disallowed_types"
The type of the file that was to be uploaded was listed in the forbidden types set in the configuration file or passed to IOSea::Upload::upload_prepare.
<"ERROR: request upload failed">
<"ERROR: request upload failed"
The catalyst request failed
<"could not determine db name from arguments">
<"could not determine db name from arguments"
The database name was not in the parameter hash used to initialize an IOSea::Upload object or in the config file.
<"could not determine host from arguments">
<"could not determine host from arguments"
The host was not in the parameter hash used to initialize an IOSea::Upload object or in the config file.
<"could not determine user from arguments">
<"could not determine user from arguments"
The user was not in the parameter hash used to initialize an IOSea::Upload object or in the config file.
<"could not determine pass from arguments">
<"could not determine pass from arguments"
The password was not in the parameter hash used to initialize an IOSea::Upload object or in the config file.
<"no file_path could be determined for upload_prepare">
<"no file_path could be determined for upload_prepare"
The file path was not in the parameter hash passed to IOSea::Upload::upload_prepare or in the config file.
<"no max_size could be determined for upload_prepare">
<"no max_size could be determined for upload_prepare"
The max size was not in the parameter hash passed to IOSea::Upload::upload_prepare or in the config file.
<"no min_size could be determined for upload_prepare">
<"no min_size could be determined for upload_prepare"
The min size was not in the parameter hash passed to IOSea::Upload::upload_prepare or in the config file.
<"failed to write file parameters to database">
<"failed to write file parameters to database"
The SQL call to insert file parameters into the database failed.
<"ERROR: failed to exec sql statement">
<"ERROR: failed to exec sql statement"
An SQL call failed or returned no results.
<"ERROR: file_path is blank">
<"ERROR: file_path is blank"
The file path retrieved from the database in IOSea::Upload::upload_validate was blank.
<"ERROR: max_size is blank">
<"ERROR: max_size is blank"
The max size retrieved from the database in IOSea::Upload::upload_validate was blank.
<"ERROR: min_size is blank">
<"ERROR: min_size is blank"
The min size retrieved from the database in IOSea::Upload::upload_validate was blank.
<"ERROR: created is blank">
<"ERROR: created is blank"
The created field retrieved from the database in IOSea::Upload::upload_validate was blank.
<"ERROR: success_token already exists for attempt_token: $attempt_token">
<"ERROR: success_token already exists for attempt_token: $attempt_token"
In IOSea::Upload::upload_success, there is already a success token corresponding to the current attempt token. This indicates that this is a duplicate attempt at uploading. Possibly caused by dubiously altered HTML code.
<"could not determine root_url from config hash">
<"could not determine root_url from config hash"
In IOSea::Upload::_generate_html, the root url was not in the config hash. Likely because it was not correctly specified in the config file.
<"config hash not defined in _generate_html()">
<"config hash not defined in _generate_html()"
In IOSea::Upload::_generate_html, the config hash was not set. This is likely because it is missing, incorrectly formatted, or the path to it is wrong in the controller Upload.pm or UploadEG.pm =back
CGI::UploadEngine is not CPAN "install" friendly and requires significant configuration, but all files and directions are included in this section.
There is a custom perl install script (with bash execs). It might not work for your system, so the steps required are described below. Please read the directions first so you will understand what the script is asking from you.
Download, unzip, and expand the archve. All the commands below should be run from the root package directory, ls -lh should look (mostly) like this:
drwxr-xr-x 2 root root 4096 Oct 28 04:05 cgi-bin -rw-r--r-- 1 root root 96 Oct 26 02:11 Changes drwxr-xr-x 2 root root 4096 Oct 28 04:28 Controller drwxr-xr-x 2 root root 4096 Oct 26 15:43 images -rwxr-xr-x 1 root root 5677 Oct 26 15:58 install drwxr-xr-x 3 root root 4096 Oct 26 02:11 lib -rw-r--r-- 1 root root 535 Oct 26 02:11 Makefile.PL -rw-r--r-- 1 root root 72 Oct 26 22:10 MANIFEST -rw-r--r-- 1 root root 1096 Oct 26 02:55 README drwxr-xr-x 3 root root 4096 Oct 26 02:47 root drwxr-xr-x 2 root root 4096 Oct 26 02:53 scripts drwxr-xr-x 2 root root 4096 Oct 26 02:47 sql drwxr-xr-x 2 root root 4096 Oct 26 02:47 t -rw-r--r-- 1 root root 162 Oct 26 02:51 uploadengine
The following commands are what the install script attempts to accomplish. After you read through the steps, try the install script before trying to follow the steps manually.
If you are attempting manual installation, note that these commands were developed in bash shell. If you are unsure what shell you are using, go ahead and start bash now:
bash
Initialize Variables
It will save a few steps if we get this out of the way first. The variables in question here are all related to target directories and server information. Values below are typical (for a Catalyst project MBC), but yours may be different.
UE_APACHE_ROOT=/var/www; export UE_APACHE_ROOT; UE_CGI_DIR=$APACHE_ROOT/cgi-bin; export UE_CGI_DIR; UE_IMAGE_DIR=$APACHE_ROOT/html/images; export UE_IMAGE_DIR; UE_EXT_PATH=$APACHE_ROOT/html/js/ext-core/ext-core-debug.js; export UE_EXT_PATH; UE_CAT_PROJ=MBC; export UE_DOMAIN; UE_CONTROLLER_DIR=/var/www/catalyst/MBC/lib/MBC/Controller; export UE_CONTROLLER_DIR; UE_TMPL_DIR=/var/www/catalyst/MBC/root/src; export UE_TMPL_DIR; UE_DOMAIN=bioinformatics.ualr.edu; export UE_DOMAIN; UE_ROOT_URL=http://$UE_DOMAIN/catalyst; export UE_ROOT_URL; UE_EXT_URL=http://$UE_DOMAIN/js/ext-core/ext-core-debug.js; export UE_EXT_URL;
Install CGI::UE config
This configuration file allows server level settings for failure icon, success message, root URL, allowed file types, and future features. It has to be in a directory that is readable by Apache, and the path is currently hard-coded in CGI::UE.
Copy script to apache root directory
The script name is prepended with a period to "hide" it.
cp uploadengine $UE_APACHE_ROOT/.uploadengine
Update config path in CGI::UE module
sed 's#APACHE_ROOT#$UE_APACHE_ROOT#g';
Install CGI::UE Module
After the apache root directory is updated, install the module:
perl Makefile.PL make install clean rm -f Makefile.old perl t/00.load.t
Install UE Database
The UE only requires a single table, but a new, separate database is recommended. The installation script will ask for your password in order to create the necessary mysql resource. To install manually, first create the database, here named files:
mysqladmin -p create files
The create the table:
mysql -p files <sql/upload.sql
The -p switch makes each program prompt you for the password. You will also need to create a user and password on the database for CGI::UE to use. The database name, user, and pass will need to match the values in cgi-bin/upload_cfg.pl.
Note: Do *NOT* use the example password "tmie"below. *DO* use an new, independent user such as "files".
mysql -p files -e "grant all privileges on files.* to 'files'\@'localhost' identified by 'tmie'";
Install CGI::UE Core
As mentioned above, the only Core currently provided is a Catalyst module, and therefore requires a working Catalyst project.
Copy template
This template defines the response from the Core to the AJAX file upload.
cp root/src/upload.tt2 $UE_TMPL_DIR/upload.tt2
Rename controller namespace
sed 's#UE_CAT_PROJ#$UE_CAT_PROJ#g';
Copy controller
cp Controller/Upload.pm $UE_CONTROLLER_DIR/Upload.pm
Restart apache
The installation script does not restart apache at all, but if you are installing manually, we recommend a restart at this point to test that CGI::UE is installed well enough to use.
Install Sample Developer App
Two types of sample apps are included: Catalyst controllers and "old school" CGI scripts. You might install both, but you should install at least one to test the installation.
In each case, there are two actions: form_eg and handler_eg. The first creates a sample form with a file select box and one other text box using the upload_prepare() method. The second uses upload_retrieve() to get the results from the engine. The scripts are shown above in the Synopsis. The controllers perform equivilant actions under Catalyst.
Install CGI scripts
The CGI scripts need several support files: templates for each action, a configuration file with web server details, and a library file that handles HTTP responses and HTML layout.
Install the javascript file
For security reasons, the jasvascript path needs to be from your server.
Update javascript path in config file
The javascript file is
Update URLs in config file
The domain
Install Catalyst controllers
The installation script does not restart apache at all, but if you are installing manually and chose to install the sample controllers, we recommend a restart at this point to test that the Controllers will load.
To install, type:
bash perl installUE
Class::Std
None reported.
No bugs have been reported.
Please report any bugs or feature requests to bug-cgi-uploadengine@rt.cpan.org, or through the web interface at http://rt.cpan.org.
bug-cgi-uploadengine@rt.cpan.org
Roger A Hall C<< <rogerhall@cpan.org> >> Michael A Bauer C<< <mbkodos@cpan.org> >> Kyle D Bolton C<< <kdbolton@ualr.edu> >> Aleksandra A Markovets C<< <aamarkovets@ualr.edu> >>
Copyleft (c) 2011, Roger A Hall <rogerhall@cpan.org>. All rights pre-served.
<rogerhall@cpan.org>
This module is free software; you can redistribute it and/or modify it under the same terms as Perl itself. See perlartistic.
BECAUSE THIS SOFTWARE IS LICENSED FREE OF CHARGE, THERE IS NO WARRANTY FOR THE SOFTWARE, TO THE EXTENT PERMITTED BY APPLICABLE LAW. EXCEPT WHEN OTHERWISE STATED IN WRITING THE COPYRIGHT HOLDERS AND/OR OTHER PARTIES PROVIDE THE SOFTWARE "AS IS" WITHOUT WARRANTY OF ANY KIND, EITHER EXPRESSED OR IMPLIED, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE. THE ENTIRE RISK AS TO THE QUALITY AND PERFORMANCE OF THE SOFTWARE IS WITH YOU. SHOULD THE SOFTWARE PROVE DEFECTIVE, YOU ASSUME THE COST OF ALL NECESSARY SERVICING, REPAIR, OR CORRECTION.
IN NO EVENT UNLESS REQUIRED BY APPLICABLE LAW OR AGREED TO IN WRITING WILL ANY COPYRIGHT HOLDER, OR ANY OTHER PARTY WHO MAY MODIFY AND/OR REDISTRIBUTE THE SOFTWARE AS PERMITTED BY THE ABOVE LICENSE, BE LIABLE TO YOU FOR DAMAGES, INCLUDING ANY GENERAL, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES ARISING OUT OF THE USE OR INABILITY TO USE THE SOFTWARE (INCLUDING BUT NOT LIMITED TO LOSS OF DATA OR DATA BEING RENDERED INACCURATE OR LOSSES SUSTAINED BY YOU OR THIRD PARTIES OR A FAILURE OF THE SOFTWARE TO OPERATE WITH ANY OTHER SOFTWARE), EVEN IF SUCH HOLDER OR OTHER PARTY HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.
2 POD Errors
The following errors were encountered while parsing the POD:
You can't have =items (as at line 569) unless the first thing after the =over is an =item
You forgot a '=back' before '=head1'
To install CGI::UploadEngine, copy and paste the appropriate command in to your terminal.
cpanm
cpanm CGI::UploadEngine
CPAN shell
perl -MCPAN -e shell install CGI::UploadEngine
For more information on module installation, please visit the detailed CPAN module installation guide.