Modifications are in RED.
o The terms in the splay tree should be built up from
the kw-inv.d file provided in the
/public/mp/
directory. The
terms are the set of items stored in
the doclist data structure (see
MP3
Notes for description of doclist).
o To calculate LogBaseTwo(someNumber),
use the <math.h> function log10(someNumber),
and the
equation loga(x) =
logb(x) / logb(a) . Also, compile with the
-lm
option to include the math libs.
o Be sure to use normalization formulas (3.34) and (3.35) for short and long queries respectively (pg 80).
o Do not use the tinsert or tadd
thresholds (as shown on pg 83) when selecting matching documents; simply
collect all
documents which partially match the
query, then return the top 100.
o Define a SHORT_QUERY as
(for the purposes of MP4) the length of the longest short query (the short
queries are the first eight queries)
and a LONG_QUERY as, well, longer.
o Here is a pseudocode sketch of the algorithm.
NOTE: this is just a HINT. Your algorithm may be more efficient,
or less (although it behooves
you to make it at least as efficient).
// build up splay tree.
for (each keyword ki in kw-inv.d) {
for each document di that ki appears in { // between the -1 and -2 markers
if Doclist[di] != NULL {
add ki to Doclist[di];
}
else {
create Doclist[di];
add ki to Doclist[di];
}
}
add
ki, ki's stats to splay tree.
}
// weight words in splay tree.
for(each word w in the splay tree) {
for( each document di that
w appears in) {
calculate wkd = weight of w in di using formula (3.33)
store wkd with the stats about w in di.
//zzz
If I were to implement a threshold on weights, sorting
//zzz of the docs would be necessary here.
//zzz sort the di's in descending order, based on weight.
}
}
// compute length of each document vector.
for each document di in Doclist {
for each keyword ki in di {
get ki out of splay tree, extract wkd, fkd.
total += wkd^2;
}
di.length = sqrt(total);
}
// Perform queries.
for( each query vector Q in /public/mp/queries.txt)
{
// Weigh the queries.
Q = query vector {q1, ...,
qn}
for(each qi in Q) {
if( this is a SHORT_QUERY)
calculate weight_qi using weighting formula (3.34)
else
calculate weight_qi using weighting formula (3.35)
Q.len = sqrt( S qi (weight_qi)^2)
}
//zzz
If I were to implement a threshold on weights, sorting would be
//zzz necessary here.
sort Q
in descending order, based on weight_qi
// Match queries with docs.
M = list of docs matching
query Q
for( each qi in Q) {
for ( each doc di that qi appears in ) {
if ( di is in M )
m_di.score += (weight_qi * wkd) / (Doclist[di].length * Q.len)
else {
add doc di to M;
m_di.score = (weight_qi * wkd ) / (Doclist[di].length * Q.len)
}
}
}
// return results.
sort M in descending order
of m_di.score
print top 100 in M,
according to format in the MP4 spec.
}//end for each query.