FOA Home
A central question remains: how many dimensions $k$ should be used? To
date, the only answers are empirical. Early experiments suggest using
too few dimensions dramatically degrades performance. {Deerwester et al.
suggest that earlier attempts to apply factor analytic techniques to IR
[Koll79] [Borko63] may have foundered because they
considered too few dimensions (7 and 21 , resp.). } The original
Top of Page
How many dimensions $k$ to reduce to?
MED corpus used in early LSI experiments [Deerwester90] [Dumais91] happens to be distinguished
by a particulary topically-focused vocabulary. While a few hundred
dimensions may suffice to discriminate this small set of documents it
might well be necessary to use more dimensions when describing a broader
domain of discourse. While a few hundred dimensions may suffice to
discriminate this small set of documents it might well be necessary to
use more dimensions when describing a broader domain of discourse.
Empirically, however, reduction to around 500 dimensions seems to
provide significant improvements to even very large corpora. For
example, Figure (figure) shows one experiment [Landauer97] where $k$ is varied over
four orders of magnitude. Some attempts to select $k$ theoretically,
borrowing a RANK-PLUS-SHIFT method borrowed from signal
processing, have also been made [Zha98]
. The most satisfying answer may come from probabilistic models
which relate the raw frequency statistics to an underlying distribution
[Papadimitriou98] [Hofmann99] .