, 
), according to the formula for
wkd specified by Okapi. You can consider that k1 = 1.
Now replot your two (Conservative and Weak) 11-point averages,
showing the original curves for comparison.
Due: 3 June 99
No groups for this project
You have now described the documents you want to index, indexed them, and written a matching algorithm that compares queries against the index and returns the ``best'' match. You must be aware of the many, often arbitrary design decision you've made in the process; how can you evaluate how well the system retrieves?
To answer normatively, we must identify a set of ``typical'' queries and, for each query, the documents that are ``relevant'' to it. Once again, many issues arise as we attempt to characterize typical query patterns. For a large corpus, exhaustively searching for all documents that someone might reasonably find relevant to each of these is a huge problem in and of itself (cf. Section 4.3).
In this exercise we will focus on a set of 11 queries against the AIT corpus. These 11 are divided into eight ``short'' and three ``long'' queries. The former are sets of two to seven keywords. Long queries are each defined with respect to a particular document within the AIT corpus; the idea is that a user has expressed (via relevance feedback) interest in this document and want more like it. (N.B., you must be careful to do the same morphological and noise word processing to these queries as you did to your corpora!)
For each of these queries, we have asked others (previous students, faculty and graduate students in AI, etc.) to assess a number of documents' relevance. Their collective opinions are collected in .../public/mp/combined-relfbk.l. For each query, and each document receiving at least some reaction by some evaluator, a list of relevance assessments (RelAss) is encoded as:
| Score | Interpretation |
| 3 | Critically relevant |
| 2 | Relevant |
| 1 | Somewhat relevant |
| 0 | No reaction |
It is not immediately obvious just how the strong/weak opinions of variable numbers of people should be combined to form the Boolean notion of ``relevant'' required by most IR system assesments. We will consider two particular predicates: A Conservative notion of relevance requires assessments by more than one person, and the average of these assesments to be at least ``somewhat relevant." A Weak notion of relevant simply requires at least some reaction by some evaluator; i.e., .../public/mp/combined-relfbk.l contains information about a document iff it's relevant.
These two predicates have both been applied .../public/mp/combined-relfbk.l and .../public/mp/rel.d contains the results. Each line of this file contains a query/document pair, and either 2 if it satisfies the conservative notion of relevance, and 1 otherwise:
| QueryNo | DocumentNo | (1|2) |
Use this data for the following purposes:
Create a ``batch'' routine that applies
all 11 queries against AIT and produces a file match.d giving, in descending
order all nonzero MatchScores (not only the first 100):
| QueryNo | DocumentNo | MatchScore |
Create two ``Precision/Recall curves,'' as described in Section 4.3, corresponding to the Conservative and Weak RelAss sets, resp. Do this using a match_rel2pr(hitlist,relass) routine which combines, for each query, the matcher's hitlist with the corresponding relevance assesments. It should produce output that can be used directly by a scatter plotting package (e.g., gnuplot).
Aggregate across queries, averaging their precision at fixed recall levels ranging from 0%-100% (use at least ten levels, one every ten percent). Check how this is done in Figure 4.16 (FOA p.111). Do this for both the Conservative and Weak RelAss sets and plot these two average curves on an additional plot. Recall that this will require interpolation for recall/precision points at these specified recall values.
Perturb the search engine you have built to implement the Okapi
weighting formula (FOA Eq 3.22, p. 75) in place of the TF-IDF formula
used in MP4. You use again cosine similarity, but you have to recalculate
all variables (Len(d), Len(q), average doc length
, ), according to the formula for
wkd specified by Okapi. You can consider that k1 = 1.
Now replot your two (Conservative and Weak) 11-point averages,
showing the original curves for comparison.
Create a subdirectory eval in your home directory, and place all files named below in it.
To turn these in, you should make a compressed tar file of the directory and then run turnin to submit it for grading, before the due date for the assignment. Assuming you're in your home directory, which should contain your eval directory, the appropriate sequence is:
tar cvf - eval | compress > eval.tar.Z turnin -c cs133w eval.tar.Z
Turning in after the first time overwrites your previously turned-in version.
To make grading easier for the TA, please also print out hardcopies of:
This document was generated using the LaTeX2HTML translator Version 97.1 (release) (July 13th, 1997)
Copyright © 1993, 1994, 1995, 1996, 1997, Nikos Drakos, Computer Based Learning Unit, University of Leeds.
The command line arguments were:
latex2html -split 0 -dir /cse/grad/mpetropo/mp5/mp5/ /cse/grad/mpetropo/mp5/mp5.tex.
The translation was initiated by Michail PETROPOULOS on 5/25/1999