RePrec::Collection - Parse relevance judgements for evaluation purposes
To do an evaluation of effectiveness of information retrieval methods one needs relevance judgements for queries and a collection under consideration. These need to be parsed for doing the evaluation. Class RePrec::Collection provides for means to do so which should suit for most formats of relevance judgments. In case it doesn't suit one can subclass this class. From a list of relevance judgements one needs to filter the query ID (QID), the document ID (DOCID) and a judgement (JUDGE) wether DOCID is relevant with respect to QID. As an additional parameter the number of documents in the collection under consideration is needed.
- new %parms
Constructor which does the parsing of a given judgements file. The constructor calls the private method
_init(with %parms as argument) in order to do the parsing. The argument %parms is described within the documentation of that method.
- _init %parms
The file parsing method, which should be the only method to replace in subclasses of RePrec::Collection. Within this baseclass it is assumed that the data in $file comes as an table, with each row containing a QID, a DOCID and the judgement (JUDGE) itself. A document is marked relevant if the value of JUDGE equals 1. Argument %parms keep the following parameters (defaults are given in parens):
- separator (' +')
perl regular expression separating columns
column which holds the QIDs
column which holds the DOCIDs
column which holds the JUDGEs
- ignore (undef)
perl regular expression; matching rows are ignored
- numdocs (undef)
number of documents in the collection under consideration.
- relevant $qid, $docid
returns 1 if document with ID $docid is relevant with respect to query with ID $qid. Else returns
returns number of documents with respect to the collection under consideration.
- get_numrels $qid
returns number of relevant documents for query with ID $qid with respect to the collection under consideration.
Yes. Please let me know!
Norbert Gövert <email@example.com>