Data mining and applications in science and technology

Homework project: Text mining and information retrieval

1. Experiments in text-mining and information retrieval

   Perform some or all of the investigations given below.

   • Perform a clustering of the term-document matrix in, say, 4 clusters (try some more values). Represent the documents in terms of the basis of centroids. Compare the results with those obtained with LSI. Also compare the amount of memory needed.
   • Instead of using centroids as basis vectors, compress each cluster using SVD (how should you then determine which clusters to look in?). Discuss the memory requirements, and check if this approach has any advantages over the other two. (This is similar to what we used for handwritten digit classification; local bases are popular also in other applications).
   • The performance of the vector space model depends critically on the weighting scheme, according to some papers in the area. Test some different schemes mentioned in the book by Berry.
   • You can also try other document data bases, e.g. those that are available at   Glasgow IR test collection.   Note that it may be necessary to edit the data (especially with respect to document separator and Windows format; the latter is non-trivial). Also check the filters available e.g. for  removing html.

2. Spam filter Take a file consisting of a number of a number of mail message that have been characterized as spam, and parse it using GTP (it is necessary first to filter away mail and html tags). Compress the term-document matrix using the SVD (a clustering approach is also possible). It may be of some interest to see what words the most significant entries of the dominant singular vectors correspond to ("dollar" is a rather certain guess). Using the dictionary produced by GTP, find the vector of an incoming message, and determine if it is close enough to any document in the compressed data base to be characterized as spam. Test the performance of the spam filter for different values of the parameters in the procedure. e.g. distance measure, SVD rank.
This project requires some familiarity with script programming.
3. FAQ answering machine This assignment assumes that you have an application where some users ask questions by e-mail. Starting out from a database of answers, parse it producing a term-document matrix. Then, given a question in natural language, parse it as in assignment 2 above, and return the 5 closest previous answers.