The Perl Toolchain Summit needs more sponsors. If your company depends on Perl, please support this very important event.
 / 
Queue-Q-0.15
River stage zero No dependents
8 ++
/ Queue::Q::ReliableFIFO

NAME

Queue::Q::ReliableFIFO - FIFO queue keeping track of claimed items

SYNOPSIS

  use Queue::Q::ReliableFIFO::Redis; # or ::Perl or ...
  my $q = ... create object of chosen ReliableFIFO implementation...
  
  # producer:
  $q->enqueue_item([qw(my data structure)]); # rinse repeat...
  
  # consumer:
  my $item = $q->claim_item;
  # work with data...
  $q->mark_item_as_done($item);
  
  # Implementation dependent: somewhere in a recovery cron job
  # - Fetch claimed items older than $n minutes/seconds
  # - Requeue or log&drop those timed-out items

DESCRIPTION

Abstract interface class for a FIFO queue that keeps track of all in-flight ("claimed") items. Implementations are required to provide strict ordering and adhere to the run-time complexities listed below (or better).

The general workflow with Queue::Q::ReliableFIFO based queue is:

  • Producer enqueues one or multiple items.

  • Consumer claims one or multiple items and works on them.

  • Consumer marks its claimed items as done.

To recover from failed consumers, one may apply in any one of many application specific recovery strategies such as periodically re-enqueuing all claimed items that are older than a threshold or possible simply clearing them out and logging the fact.

Since the actual recovery strategy is application-dependent and the support by the queue implementation may vary, there's no API for this in this abstract base class. That may change in a future release.

METHODS

enqueue_item

Given a data structure, that data structure is added to the queue. Items enqueued with enqueue_item in order must be returned by claim_item in the same order.

Complexity: O(1)

enqueue_items

Given a number data structures, enqueues them in order. This is conceptually the same as calling enqueue_item multiple times, but may help save on network round-trips.

Complexity: O(n) where n is the number of items to enqueue.

claim_item

Returns the oldest item in the queue. Returns undef if there is none left.

Complexity: O(1)

Implementations may (but should not) deviate from the strict O(1) complexity for the number of claimed items at any given time. That is acceptable as high as O(log(n)). This is likely acceptable because the number of items being worked on is likely not to be extremely large. Implementations that make use of this relaxed requirement must document that clearly. The number of queued items must not affect the complexity, however.

claim_items

As enqueue_items is to enqueue_item, claim_items is to claim_item. Takes one optional parameter (defaults to 1): The number of items to fetch and return:

  my @items = $q->claim_items(20); # returns a batch of 20 items

If there are less than the desired number of items to be claimed, returns a correspondingly shorter list.

Complexity: O(n) where n is the number of items claimed.

See the documentation of claim_item for a potential relaxation on the complexity bounds with respect to the number of in-flight, claimed items.

mark_item_as_done

Given a item that was previously claimed from this queue, it is removed from the claimed-items tracking and thus removed from the queue altogether.

Complexity: Generally O(1), but O(log(n)) under relaxed requirements (see above).

mark_items_as_done

Given any number of items that were previously claimed from this queue, they are removed from the claimed-items tracking and thus removed from the queue altogether.

Complexity: Generally O(n), but O(n*log(n)) under relaxed requirements (see above) with n understood to be the number of items to mark as done.

queue_length

Returns the number of items available in the queue.

Complexity: O(1)

flush_queue

Removes all content in the queue.

Complexity: O(n) where n is the number of items in the queue.

AUTHOR

Herald van der Breggen, <herald.vanderbreggen@booking.com>

COPYRIGHT AND LICENSE

Copyright (C) 2012 by Herald van der Breggen

This library is free software; you can redistribute it and/or modify it under the same terms as Perl itself, either Perl version 5.8.1 or, at your option, any later version of Perl 5 you may have available.