Bio::ToolBox::Data::Stream - Read, Write, and Manipulate Data File Line by Line
use Bio::ToolBox::Data; ### Open a pre-existing file my $Stream = Bio::ToolBox::Data->new( in => 'regions.bed', stream => 1, ); # or directly my $Stream = Bio::ToolBox::Data::Stream->new( in => 'regions.bed', ); ### Open a new file for writing my $Stream = Bio::ToolBox::Data::Stream->new( out => 'output.txt', columns => [qw(chromosome start stop name)], ); ### Working line by line while (my $line = $Stream->next_line) { # get the positional information from the file data # assuming that the input file had these identifiable columns # each line is Bio::ToolBox::Data::Feature item my $seq_id = $line->seq_id; my $start = $line->start; my $stop = $line->end; # change values $line->value(1, 100); # index, new value } ### Working with two file streams my $inStream = Bio::ToolBox::Data::Stream->new( file => 'regions.bed', ); my $outStream = $inStream->duplicate('regions_ext100.bed'); my $sc = $inStream->start_column; my $ec = $inStream->end_column; while (my $line = $inStream->next_line) { # adjust positions by 100 bp my $s = $line->start; my $e = $line->end; $line->value($sc, $s - 100); $line->value($ec, $e + 100); $outStream->write_row($line); } ### Finishing # close your file handles when you are done $Stream->close_fh;
This module works similarly to the Bio::ToolBox::Data object, except that rows are read from a file handle rather than a memory structure. This allows very large files to be read, manipulated, and even written without slurping the entire contents into a memory.
For an introduction to the Bio::ToolBox::Data object and methods, refer to its documentation and the Bio::ToolBox::Data::Feature documentation.
Typically, manipulations are only performed on one row at a time, not on an entire table. Therefore, large scale table manipulations, such as sorting, is not possible.
A typical workflow consists of opening two Stream objects, one for reading and one for writing. Rows are read, one at a time, from the read Stream, manipulated as necessary, and then written to the write Stream. Each row is passed as a Bio::ToolBox::Data::Feature object. It can be manipulated as such, or the corresponding values may be dumped as an array. Working with the row data as an array is required when adding or deleting columns, since these manipulations are not allowed with a Feature object. The write Stream can then be passed either the Feature object or the array of values to be written.
A new Bio::ToolBox::Data::Stream object may be generated directly, or indirectly through the Bio::ToolBox::Data module.
my $Stream = Bio::ToolBox::Data::Stream->new( in => $filename, ); my $Stream = Bio::ToolBox::Data->new( stream => 1, in => $filename, );
Options to the new function are listed below. Streams are inherently either read or write mode, determined by the mode given through the options.
Provide the path of the file to open for reading. File types are recognized by the extension, and compressed files (.gz) are supported. File types supported include all those listed in Bio::ToolBox::file_helper.
Provide the path of the file to open for writing. No check is made for pre-existing files; if it exists it will be overwritten! A new data object is prepared, therefore column names must be provided.
Boolean option indicating that the input file does not have file headers, in which case dummy headers are provided. This is not necessary for defined file types that don't normally have file headers, such as BED, GFF, or UCSC files. Ignored for output files.
my $Stream = Bio::ToolBox::Data::Stream->new( out => $filename, columns => [qw(Column1 Column2 ...)], );
When a new file is written, provide the names of the columns as an anonymous array. If no columns are provided, then a completely empty data structure is made. Columns must be added with the add_column() method below.
When writing a GFF file, provide a GFF version. When this is given, the nine standard column names and metadata are automatically provided based on the file format specification. Note that the column names are not actually written in the file, but are maintained for internal use. Acceptable versions include 1, 2, 2.5 (GTF), and 3 (GFF3).
When writing a BED file, provide the number of bed columns that the file will have. When this is given, the standard column names and metadata will be automatically provided based on the standard file format specification. Note that column names are not actually written to the file, but are maintained for internal use. Acceptable values are integers from 3 to 12.
When writing a UCSC-style file format, provide the number of bed columns that the file will have. When this is given, the standard column names and metadata will be automatically provided based on the file format specification. Note that column names are not actually written to the file, but are maintained for internal use. Acceptable values include 10 (refFlat without gene names), 11 (refFlat with gene names), 12 (knownGene gene prediction table), and 15 (an extended gene prediction or genePredExt table).
Boolean value to change the compression status of the output file. If overwriting an input file, the default is maintain the compression status, otherwise no compression. Pass a 0 for no compression, 1 for standard gzip compression, or 2 for block gzip (bgzip) compression for tabix compatibility.
my $Out_Stream = $Stream->duplicate($new_filename);
For an opened-to-read Stream object, you may duplicate the object as a new opened-to_write Stream object that maintains the same columns and metadata. A new different filename must be provided.
There is a variety of general metadata regarding the Data structure that is available.
The following methods may be used to access or set these metadata properties. Note that metadata is only written at the beginning of the file, and so must be set prior to iterating through the file.
Returns or sets the name of the features used to collect the list of features. The actual feature types are listed in the table, so this metadata is merely descriptive.
Returns one of three specific values describing the contents of the data table inferred by the presence of specific column names. This provides a clue as to whether the table features represent genomic regions (defined by coordinate positions) or named database features. The return values include:
Returns or sets the name of the program generating the list.
Returns or sets the name or path of the database from which the features were derived.
Returns or sets the version of loaded GFF files. Supported versions included 1, 2, 2.5 (GTF), and 3.
Returns or sets the BED file version. Here, the BED version is simply the number of columns.
Returns or sets the UCSC file format version. Here, the version is simply the number of columns. Supported versions include 10 (gene prediction), 11 (refFlat, or gene prediction with gene name), 12 (knownGene table), 15 (extended gene prediction), or 16 (extended gene prediction with bin).
Returns or sets the VCF file version number. VCF support is limited.
These methods provide information about the file from which the data table was loaded. This does not include parsed annotation tables.
Returns the filename, full path, basename, and extension of the filename. Concatenating the last three values will reconstitute the first original filename.
$Data->add_file_metadata('/path/to/file.txt');
Add filename metadata. This will automatically parse the path, basename, and recognized extension from the passed filename and set the appropriate metadata attributes.
Comments are the other commented lines from a text file (lines beginning with a #) that were not parsed as metadata.
Returns a copy of the array containing commented lines.
Appends the text string to the comment array.
Deletes a comment. Provide the array index of the comment to delete. If an index is not provided, ALL comments will be deleted!
For VCF files, this will partially parse the VCF headers into a hash structure that can be queried or manipulated. Each header line is parsed for the primary key, being the first word after the ## prefix, e.g. INFO, FORMAT, FILTER, contig, etc. For the simple values, they are stored as the value. For complex entries, such as with INFO and FORMAT, a second level hash is created with the ID extracted and used as the second level key. The value is always the always the remainder of the string.
For example, the following would be a simple parsed vcf header in code representation.
$vcf_header = { FORMAT => { GT = q(ID=GT,Number=1,Type=String,Description="Genotype"), AD = q(ID=AD,Number=.,Type=Integer,Description="ref,alt Allelic depths"), }, fileDate => 20150715, }
If you have altered the vcf headers exported by the vcf_headers() method, then this method will rewrite the hash structure as new comment lines. Do this prior to writing the new file stream or else you will lose your changed VCF header metadata.
Information about the columns may be accessed. This includes the names of the column and shortcuts to specific identifiable columns, such as name and coordinates. In addition, each column may have additional metadata. Each metadata is a series of key => value pairs. The minimum keys are 'index' (the 0-based index of the column) and 'name' (the column header name). Additional keys and values may be queried or set as appropriate. When the file is written, these are stored as commented metadata lines at the beginning of the file. Setting metadata is futile after reading or writing has begun.
Returns an array or array reference of the column names in ascending (left to right) order.
Returns the number of columns in the Data table.
Returns the array index of the last (rightmost) column in the Data table.
$Stream->name($index, $new_name); my $name = $Stream->name($i);
Convenient method to return the name of the column given the index number. A column may also be renamed by passing a new name.
$Stream->metadata($index, $key, $new_value); my $value = $Stream->metadata($index, $key)
Returns or sets the metadata value for a specific $key for a specific column $index.
This may also be used to add a new metadata key. Simply provide the name of a new $key that is not present
If no key is provided, then a hash or hash reference is returned representing the entire metadata for that column.
$Stream->copy_metadata($source, $target);
This method will copy the metadata (everything except name and index) between the source column and target column. Returns 1 if successful.
$Stream->delete_metadata($index, $key);
Deletes a column-specific metadata $key and value for a specific column $index. If a $key is not provided, then all metadata keys for that index will be deleted.
my $i = $Stream->find_column('Gene'); my $i = $Stream->find_column('^Gene$')
Searches the column names for the specified column name. This employs a case-insensitive grep search, so simple substitutions may be made.
These methods will return the identified column best matching the description. Returns undef if that column is not present. These use the "find_column" method with a predefined list of aliases.
undef
These methods allow modification to the number and order of the columns in a Stream object. These methods can only be employed prior to opening a file handle for writing, i.e. before the first "write_row" method is called. This enables one, for example, to duplicate a read-only Stream object to create a write-only Stream, add or delete columns, and then begin the row iteration.
my $i = $Stream->add_column($name);
Appends a new column at the rightmost position (highest index). It adds the column header name and creates a new column metadata hash. Pass a text string representing the new column name. It returns the new column index if successful.
my $j = $Stream->copy_column($i);
This will copy a column, appending the duplicate column at the rightmost position (highest index). It will duplicate column metadata as well. It will return the new index position.
Deletes one or more specified columns. Any remaining columns rightwards will have their indices shifted down appropriately. If you had identified one of the shifted columns, you may need to re-find or calculate its new index.
$Data->reorder_column($c,$b,$a,$a);
Reorders columns into the specified order. Provide the new desired order of indices. Columns could be duplicated or deleted using this method. The columns will adopt their new index numbers.
Once a file Stream object has been opened, and metadata and/or columns adjusted as necessary, then the file contents can be iterated through, one row at a time. This is typically a one-way direction. If you need to go back or start over, the easiest thing to do is re-open the file as a new Stream object.
There are two main methods, "next_row" for reading and "write_row" for writing. They cannot and should not be used on the same Stream object.
This method reads the next line in the file handle and returns a Bio::ToolBox::Data::Feature object. This object represents the values in the current file row.
Note that strand values and 0-based start coordinates are automatically converted to BioPerl conventions if required by the file type.
$Data->add_row(\@values); $Data->add_row($Row); # Bio::ToolBox::Data::Feature object
This method writes a new row or line to a file handle. The first time this method is called the file handle is automatically opened for writing. Up to this point, columns may be manipulated. After this point, columns cannot be adjusted (otherwise the file structure becomes inconsistent).
This method may be implemented in one of three ways, based on the type data that is passed.
A Feature object
A Feature object representing a row from another Bio::ToolBox::Data data table or Stream. The values from this object will be automatically obtained. Modified strand and 0-based coordinates may be adjusted back as necessary.
An array reference of values
Pass an array reference of values. The number of elements should match the number of expected columns. The values will be automatically joined using tabs. This implementation should be used if you using values from another Stream and the number of columns have been modified.
Manipulation of strand and 0-based starts may be performed if the metadata indicates this should be done.
A string
Pass a text string. This assumes the column values are already tab concatenated. A new line character is appended if one is not included. No data manipulation (strand or 0-based starts) or sanity checking of the required number of columns is performed. Use with caution!
$Stream->iterate( sub { my $row = shift; my $number = $row->value($index); my $log_number = log($number); $row->value($index, $log_number); } );
A convenience method that will process a code reference for every line in the file. Pass a subroutine or code reference. The subroutine will receive the line as a Bio::ToolBox::Data::Feature object, just as with the "read_line" method.
The below methods work with the file handle. When you are finished with a Stream, you should be kind and close the file handle properly.
Returns the write mode of the Stream object. Read-only objects return false (0) and write-only Stream objects return true (1).
Closes the file handle.
Returns the IO::File compatible file handle object representing the file handle. Use with caution.
Bio::ToolBox::Data, Bio::ToolBox::Data::Feature
Timothy J. Parnell, PhD Dept of Oncological Sciences Huntsman Cancer Institute University of Utah Salt Lake City, UT, 84112
This package is free software; you can redistribute it and/or modify it under the terms of the Artistic License 2.0.
To install Bio::ToolBox, copy and paste the appropriate command in to your terminal.
cpanm
cpanm Bio::ToolBox
CPAN shell
perl -MCPAN -e shell install Bio::ToolBox
For more information on module installation, please visit the detailed CPAN module installation guide.