Notes on MP4
last updated 5/19/99 11:50 am.

Modifications are in RED.

o    The terms in the splay tree should be built up from the kw-inv.d file provided in the /public/mp/ directory.  The
      terms are the set of items stored in the doclist data structure (see MP3 Notes for description of doclist).

o    To calculate LogBaseTwo(someNumber), use the <math.h> function log10(someNumber), and the
      equation  loga(x) = logb(x) / logb(a) .  Also, compile with the -lm option to include the math libs.

o    Be sure to use normalization formulas (3.34) and (3.35) for short and long queries respectively (pg 80).

o    Do not use the tinsert or tadd thresholds (as shown on pg 83) when selecting matching documents; simply collect all
      documents which partially match the query, then return the top 100.

o     Define a SHORT_QUERY as (for the purposes of MP4) the length of the longest short query (the short queries are the first eight queries)
       and a LONG_QUERY as, well, longer.

o     Here is a pseudocode sketch of the algorithm. NOTE: this is just a HINT. Your algorithm may be more efficient,
        or less (although it behooves you to make it at least as efficient).

    // build up splay tree.
    for (each keyword ki in kw-inv.d)  {

            for each document di that ki appears in {  // between the -1 and -2 markers

                        if  Doclist[di] != NULL  {
                                add ki to Doclist[di];
                         }
                        else {
                                create Doclist[di];
                                add ki to Doclist[di];
                        }

              }

            add ki, ki's stats to splay tree.
     }

    // weight words in splay tree.
    for(each word w in the splay tree) {
        for( each document di that w appears in) {
            calculate  wkd = weight of w in di using formula (3.33)
            store wkd with the stats about  w in di.

            //zzz If I were to implement a threshold on weights, sorting
            //zzz of the docs would be necessary here.
            //zzz sort the di's in descending order, based on weight.
        }
    }

   // compute length of each document vector.

    for each document di  in Doclist {

            for each keyword ki in di {
                       get ki out of splay tree, extract wkd, fkd.
                       total += wkd^2;
            }

            di.length = sqrt(total);

    }

    // Perform queries.
    for( each query vector Q in /public/mp/queries.txt) {

        // Weigh the queries.
        Q = query vector {q1, ..., qn}
        for(each qi in Q) {
            if( this is a SHORT_QUERY)
                calculate weight_qi using weighting formula (3.34)
            else
                calculate weight_qi using weighting formula (3.35)

            Q.len = sqrt( S qi (weight_qi)^2)
         }

            //zzz If I were to implement a threshold on weights, sorting would be
            //zzz necessary here.
           sort Q in descending order, based on weight_qi

        // Match queries with docs.
        M = list of docs matching query Q
        for( each qi in Q) {
            for ( each doc di that qi appears in ) {
                      if  ( di is in M )
                            m_di.score += (weight_qi * wkd) / (Doclist[di].length * Q.len)
                      else {
                            add doc di to M;
                            m_di.score = (weight_qi * wkd ) / (Doclist[di].length * Q.len)
                     }
              }
        }

        // return results.
        sort M in descending order of m_di.score

         print top 100 in M, according to format in the MP4 spec.
 }//end for each query.