- REMOTE SEQUENCE RETRIEVAL (Bio::DB::*)
- SETTING UP BIOPERL INDICES (Bio::Index::*)
- USING BIOPERL WITH A RELATIONAL DATABASE (bioperl-db)
- BIO::DB::GFF: INDEXING AND RETRIEVAL OF SEQUENCE FEATURE DATA
Bioperl databases - how to use databases with Bioperl
Not really appropiate for a synopsis. Read on...
This document is designed to let you use Bioperl with databases. Bioperl can work with a number of sequence formats and allows users to interconvert easily amongst these formats. Bioperl can also index flat files in some of these standard formats, allowing very fast retrieval. Other layers above flat file indexing are provided, allowing users to retrieve sequences from remote databases via the Web or build and query in-house relational databases (RDBs).
Different scripts are provided to get you started with the Bioperl backend, for example:
Fetches sequences from a Database
Builds indexes for flat files databases which are easily accessible by bpfetch.pl. These scripts are discussed in more detail below.
The core of the backend system is found in following modules
Generic access to databases, whether flat file, web or RDB. At the moment, this provides random access retrieval, on the basis of ids or accession numbers, but does not provide the ability to loop over the entire database, nor does it provide any complex querying ability. For these advanced features see USING BIOPERL WITH A RELATIONAL DATABASE (bioperl-db).
Bio::DB::BioSeqI is the abstract interface for the databases (hence the I). Bio::DB::GenBank and Bio::DB::GenPept are concrete implementations for network access to the GenBank and GenPept databases at NCBI, and others, using HTTP as a protocol. See Bio::DB::GenBank, Bio::DB::GenPept, and "REMOTE SEQUENCE RETRIEVAL (Bio::DB::*)" for more information.
- Bio::Index::* and Bio::DB::Fasta
Flat file indexing system, for read-only, flat file distributions. These provide for format-specific, generic type access, but the underlying machinery can be customised for any number of different flat file systems. The indices allow extremely fast retrieval based on unique keys.
The Index modules EMBL and Fasta, as they are designed as sequence databases, conform to the Bio::DB::BioSeqI interface, meaning they can be used whereever the Bio::DB::BioSeqI is expected.
Flat file indexing of Fasta files is also provided by Bio::DB::Fasta, please see Bio::DB::Fasta for more information - this module has some useful features not contained in Bio::Index::Fasta.
Conversion systems for Bio::Seq objects, either to or from sequence streams. The movement of things into SeqIO prevents the Bio::Seq object bloating up with format code, and the SeqIO system has the benefit of being very easy to extend to new formats. Please see Bio::SeqIO for more information.
The bioperl-db package is installed separately from Bioperl but is tightly integrated with Bioperl's objects. The bioperl-db RDB contains a set of tables that map to Bioperl's Seq and SeqFeature objects, and sequences in familiar formats are easily loaded into the schema in relational form, and retrieved as Seq objects as well. The RDB is also designed to contain genetic marker and map information. Currently only mysql is supported, but other RDBMs are planned. See USING BIOPERL WITH A RELATIONAL DATABASE (bioperl-db) below.
Bioperl offers a number of modules for retrieving sequences over the network. The available remote databases include GenBank, GenPept, GDB, EMBL, SwissProt, XEMBL, and remote Ace servers. A typical method is
which returns a Seq object, or
If you want to use Bioperl indicies of Fasta, EMBL/SwissProt .dat files, SwissPfam, GenBank, or Blast files then the bpfetch.pl and bpindex.pl scripts are great ways to start off (and also reading the scripts shows you how to use the Bioperl indexing stuff). bpfetch.pl and bpindex.pl coordinate using two environment variables
BIOPERL_INDEX - directory where the indices are kept BIOPERL_INDEX_TYPE - type of DBM file to use for the index
The basic way of indexing a database, once BIOPERL_INDEX has been set up, is to go
bpindex.pl <index-name> <filenames as full path>
eg, for Fasta files
bpindex.pl est /nfs/somewhere/fastafiles/est*.fa
Or, for EMBL/Swissprot files
bpindex.pl -fmt=EMBL swiss /nfs/somewhere/swiss/swissprot.dat
To retrieve sequences from the index go
bpfetch.pl also has other options to connect to Genbank across the network.
or any other directory
setenv BIOPERL_INDEX /nfs/datadisk/bioperlindex/
in .cshrc or .tcshrc (or set and export in bash and its .bashrc)
bpindex.pl swissprot /nfs/datadisk/swiss/swissprot.dat
Flat file indexing of Fasta files is also provided by Bio::DB::Fasta, please see Bio::DB::Fasta for more information - this module provides some functionality absent from Bio::Index::Fasta.
First download and install the bioperl-db package (ftp://bioperl.org/pub/DIST/bioperl-db-0.1.tar.gz). The current version relies on the prior installation of the free mysql server, from http://www.mysql.com. The bioperl-db package integrates neatly with Bioperl's objects and contains tables for sequences, entries, sequence features, taxonomic information, references, keywords, a wide variety of genetic markers, and more. With a Seq object and a built-in database "adaptor" one can load the object's data in one step, as in
Similarly, one can query the database with id's or using query objects and retrieve arrays of Seq objects. For example
@seqobjs = $adaptor->fetch_by_query( -constraints => ["or", ["and", # find all human topoisomerases "species=Human", "keywords like *topoisomerase*"], ["and", # and Escherichia gyrases "species like Escherichia*", "keywords like *gyrase*"], ]);
An alternative view of sequence feature data is provided by the Bio::DB::GFF module, part of the core package. This module takes a set of sequence features stored in GFF (gene-finding format) format and loads them into a relational database optimized for positional queries. This allows a variety of data mining operations, such as finding sequence features that are within a certain distance of each other or which overlap. The module can also store large contiguous sequences and extract subsequences rapidly.