CSV_LOADER

The csv_loader plugin provides functionality to load delimited files into a database table. It relies on a control file to define the fields in the incoming data file. The following parameters must be set in the config section:

  • table_name. The name table that the loader inserts the file data into.

  • controlfile_dir. The directory that contains the control file.

  • control_file. The name of the control file

The item parameters are:

  • name. The name of this item.

  • ignore_field_count. whether to continue if the number fields in the data (CSV) file don't match the number of fields in the control file.

  • skip. The number of rows to skip in the file before loading begins. This allows header records to be ignored.

  • localize. A boolean setting that instructs the loader to localize the end-of-line markers for the current file system. Defaults to 0.

  • csv_options. A section containing additional options for processing. See http://search.cpan.org/dist/Text-CSV/lib/Text/CSV.pm#new_%28\%attr%29.

  • on_error. Override the setting for the job.

Table Audit Column

The csv_loader plugin requires the table that stores the data have a column called file_id of type integer. This is used to link the file to its audit record.

Example

    <item>
        name = load customer file
        type = csv_loader
        # Skip one line
        skip = 1
        <csv_options>
            allow_whitespace = 1 
            sep_char         = | # use pipes not commas for delimiting data
        </csv_options>
    </item>