NAME

dbpipeline - allow db commands to be assembled as pipelines in Perl

SYNOPSIS

    use Fsdb::Filter::dbpipeline qw(:all);
    dbpipeline(
        dbrow(qw(name test1)),
        dbroweval('_test1 += 5;')
    );

DESCRIPTION

This module makes it easy to create threaded pipelines in Perl.

By default (as with all Fsdb modules), input is from STDIN and output to STDOUT. Two helper functions, fromfile and tofile can grab data from files.

Dbpipeline differs in several ways from all other Fsdb::Filter modules: it does not have a corresponding Unix command (it is used only from within Perl). It does not log its presence to the output stream (this is arguably a bug, but it doesn't actually do anything).

OPTIONS

Unlike most Fsdb modules, dbpipeline defaults to --autorun.

This module also supports the standard fsdb options:

-d

Enable debugging output.

-i or --input InputSource

Read from InputSource, typically a file name, or - for standard input, or (if in Perl) a IO::Handle, Fsdb::IO or Fsdb::BoundedQueue objects.

-o or --output OutputDestination

Write to OutputDestination, typically a file name, or - for standard output, or (if in Perl) a IO::Handle, Fsdb::IO or Fsdb::BoundedQueue objects.

--autorun or --noautorun

By default, programs process automatically, but Fsdb::Filter objects in Perl do not run until you invoke the run() method. The --(no)autorun option controls that behavior within Perl.

--help

Show help.

--man

Show full manual.

SEE ALSO

Fsdb(3)

CLASS FUNCTIONS

dbpipeline

    dbpipeline(@modules);

This shorthand-routine creates a dbpipeline object and then immediately runs it.

Thus perl code becomes nearly as terse as shell code:

    dbpipeline(
        dbcol(qw(name test1)),
        dbroweval('_test1 += 5;'),
    );

The following commands currently have shorthand aliases:

cgi_to_db(1)

combined_log_format_to_db(1)

csv_to_db(1)

db_to_csv(1)

db_to_html_table(1)

dbcol(1)

dbcolcreate(1)

dbcoldefine(1)

dbcolhisto(1)

dbcolmerge(1)

dbcolmovingstats(1)

dbcolneaten(1)

dbcolpercentile(1)

dbcolrename(1)

dbcolscorrelate(1)

dbcolsplittocols(1)

dbcolsplittorows(1)

dbcolsregression(1)

dbcolstats(1)

dbcolstatscores(1)

dbfilealter(1)

dbfilestripcomments(1)

dbfilevalidate(1)

dbformmail(1)

dbjoin(1)

dbmapreduce(1)

dbmerge(1)

dbmerge2(1)

dbmultistats(1)

dbrow(1)

dbrowaccumulate(1)

dbrowcount(1)

dbrowdiff(1)

dbroweval(1)

dbrowuniq(1)

dbrvstatdiff(1)

dbsort(1)

html_table_to_db(1)

kitrace_to_db(1)

mysql_to_db(1)

tabdelim_to_db(1)

tcpdump_to_db(1)

and

dbsubprocess(3)

dbpipeline_filter

    my($result_reader, $thread) = dbpipeline_filter($source, $result_reader_aref, @modules);

Set up a pipeline of @MODULES that filters data pushed through it, where the data comes from $SOURCE (any Fsdb::Filter::parse_io_option object, like a Fsdb::IO::Reader object, queue, or filename).

Returns a $RESULT_READER Fsdb::IO::Reader object, created with $RESULT_READER_AREF as options. This reader will produce the filtered data, and a $THREAD that must be joined to guarantee output has completed.

dbpipeline_sink

    my($fsdb_writer, $thread) = dbpipeline_sink($writer_arguments_aref, @modules);

Set up a pipeline of @MODULES that is a data "sink", where the output is given by a --output argument, or goes to standard output (by default). The caller generates input into the pipeline by writing to a newly created $FSDB_WRITER, whose configuration is specified by the manditory first arugment $WRITER_ARGUMENTS_AREF. (These arguments should include the schema.) Returns this writer, and a $THREAD that must be joined to guarantee output has completed.

dbpipeline_open2

    my($fsdb_read_queue, $fsdb_writer, $thread) = dbpipeline_open2($writer_arguments_aref, @modules);

Set up a pipeline of @MODULES that is a data sink and source (both!). The caller generates input into the pipeline by writing to a newly created $FSDB_WRITER, whose configuration is specified by the manditory arugment $WRITER_ARGUMENTS_AREF. These arguments should include the schema.) The output of the pipeline comes out to the newly created $FSDB_READ_QUEUE. Returns this read queue and writer, and a $THREAD that must be joined to guarantee output has completed.

(Unfortunately the interface is assymetric with a read queue but a write Fsdb::IO object, because Fsdb::IO::Reader blocks on input of the header.)

Like IPC::Open2, with all of its pros and cons like potential deadlock.

dbpipeline_close2_hash

    my($href) = dbpipeline_close2_hash($fsdb_read_queue, $fsdb_writer, $thread);

Reads and returns one line of output from $FSDB_READER, after closing $FSDB_WRITER and joining the $THREAD.

Useful, for example, to get dbcolstats output cleanly.

new

    $filter = new Fsdb::Filter::dbpipeline(@arguments);

set_defaults

    $filter->set_defaults();

Internal: set up defaults.

parse_options

    $filter->parse_options(@ARGV);

Internal: parse options

setup

    $filter->setup();

Internal: setup, parse headers.

run

    $filter->run();

Internal: run over all IO

finish

    $filter->finish();

Internal: we would write a trailer, but we don't because we depend on the last command in the pipeline to do that. We don't actually have a valid output stream.

AUTHOR and COPYRIGHT

Copyright (C) 1991-2007 by John Heidemann <johnh@isi.edu>

This program is distributed under terms of the GNU general public license, version 2. See the file COPYING with the distribution for details.

cpanm Fsdb

perl -MCPAN -e shell
install Fsdb