NAME

WARC::Record - one record from a WARC file

SYNOPSIS

  use WARC;             # or ...
  use WARC::Volume;     # or ...
  use WARC::Collection;

  # WARC::Record objects are returned from ->record_at and ->search methods

  # Construct a record, as when preparing a WARC file
  $warcinfo = new WARC::Record (type => 'warcinfo');

  # Accessors

  $value = $record->field($name);

  $version = $record->protocol; # analogous to HTTP::Message::protocol
  $volume = $record->volume;
  $offset = $record->offset;
  $record = $record->next;

  $fields = $record->fields;

...

DESCRIPTION

WARC::Record objects come in two flavors with a common interface. Records read from WARC files are read-only and have meaningful return values from the methods listed in "Methods on records from WARC files". Records constructed in memory can be updated and those same methods all return undef.

Common Methods

$record->fields

Get the internal WARC::Fields object that contains WARC record headers.

$record->field( $name )

Get the value of the WARC header named $name from the internal WARC::Fields object.

$record <=> $other_record

$record->compareTo( $other_record )

Compare two WARC::Record objects according to a simple total order: ordering by starting offset for two records in the same file, and by filename of the containing WARC::Volume objects for records in different files. Constructed WARC::Record objects are assumed to come from a volume named "" (the empty string) for this purpose, and are ordered in an arbitrary but stable manner amongst themselves. Constructed WARC::Record objects never compare as equal.

Perl constructs a == operator using this method, so WARC record objects will compare as equal iff they refer to the same physical record.

Convenience getters

$record->type

Alias for $record->field('WARC-Type').

$record->id

Alias for $record->field('WARC-Record-ID').

$record->date

Return the 'WARC-Date' field as a WARC::Date object.

Methods on records from WARC files

These methods all return undef if called on a WARC::Record object that does not represent a record in a WARC file.

$record->protocol

Return the format and version tag for this record. For WARC 1.0, this method returns 'WARC/1.0'.

$record->volume

Return the WARC::Volume object representing the file in which this record is located.

$record->offset

Return the file offset at which this record can be found.

$record->next

Return the next WARC::Record in the WARC file that contains this record. Returns an undefined value if called on the last record in a file.

$record->open_block

Return a tied filehandle that reads the WARC record block.

The WARC record block is the content of a WARC record, analogous to the entity body in an HTTP::Message.

$record->replay

$record->replay( as => $type )

Return a protocol-specific object representing the record contents.

This method returns undef if the library does not recognize the protocol message stored in the record and croaks if a requested conversion is not supported.

A record with Content-Type "application/http" with an appropriate "msgtype" parameter produces an HTTP::Request or HTTP::Response object. An unknown "msgtype" on "application/http" produces a generic HTTP::Message. The returned object may be a subclass to support deferred loading of entity bodies.

A request to replay a record "as => http" attempts to convert whatever is stored in the record to an HTTP exchange, analogous to the "everything is HTTP" interface that LWP provides.

$record->open_payload

Return a tied filehandle that reads the WARC record payload.

The WARC record payload is defined as the decoded content of the protocol response or other resource stored in the record. This method returns undef if called on a WARC record that has no payload or content that we do not recognize.

Methods on fresh WARC records

$record = new WARC::Record (key => value, ...)

Construct a fresh WARC record, suitable for use with WARC::Builder.

AUTHOR

Jacob Bachmeyer, <jcb@cpan.org>

SEE ALSO

WARC, HTTP::Message

"Extension subfield 'sl' in gzip header" in WARC::Builder

COPYRIGHT AND LICENSE

Copyright (C) 2019 by Jacob Bachmeyer

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself.