Dataless Classification with Descartes

The following example shows how to generate the Explicit Semantic Analysis representation for a given file. To compile this, all the jars in the bin directory need to be in the classpath.

import java.io.BufferedReader;
import java.io.FileReader;
import java.util.ArrayList;

import edu.illinois.cs.cogcomp.descartes.retrieval.IResult;
import edu.illinois.cs.cogcomp.descartes.retrieval.ISearcher;
import edu.illinois.cs.cogcomp.descartes.retrieval.SearcherFactory;

/**
 * Generates Explicit Semantic Analysis. Uses the Wikipedia index from index-dir
 * and generates the specified number of concepts for the input file.
 * 
 * Usage: java ESA index-dir num-concepts file-input file
 * 
 * This assumes that descartes-0.1.jar and the other dependencies are in the
 * classpath.
 * 
 */
public class ESA {

    public static void main(String[] args) throws Exception {
        if (args.length != 2) {
            System.err.println("Usage: java ESA index-dir num-concepts file");
            System.exit(-1);
        }

        String indexDir = args[0];
        int numConcepts = Integer.parseInt(args[1]);
        String file = args[2];

        // Create a new searcher to search the index
        ISearcher searcher = SearcherFactory.getStandardSearcher(indexDir);

        // Grab the input text
        BufferedReader reader = new BufferedReader(new FileReader(file));
        StringBuffer sb = new StringBuffer();
        String line = null;
        while ((line = reader.readLine()) != null) {
            sb.append(line + " ");
        }
        String text = sb.toString();

        // Clean up the input to make sure that nothing unexpected happens.
        text = text.replaceAll("[^a-zA-Z0-9 ]", "");

        // Get the concepts.
        ArrayList<IResult> results = searcher.search(text, numConcepts);

        // Print them
        for (IResult result : results) {
            // IResult has getters for id, title, document and score.
            System.out.println(result.getTitle());
        }
    }
}

For Dataless classification, we measure the similarity between the text and class labels in the concept space and predict the label that is most similar to the text. Here is example code that shows how to perform dataless classification:

public String dataless() {

    // The two classes that we want to classify into
    String prototype1 = "science electronics";
    String prototype2 = "science crypt";
    List<String> classPrototypes = Arrays.asList(prototype1, prototype2);

    // Create a searcher. This is similar to how it is done in the ESA
    // example.
    ISearcher searcher = SearcherFactory
            .getStandardSearcher(indexDirectory);

    // Specify the number of concepts to be used. This is a parameter that
    // can be tuned.
    int numConcepts = 1000;

    // Create a classifier
    DatalessClassifier classifier = new DatalessClassifier(searcher,
            numConcepts, classPrototypes);

    // The input text that needs to be classified
    String inputText = "Decryption is the reverse, in other words, "
            + "moving from the unintelligible ciphertext "
            + "back to plaintext.";

    // Perform the classification. This should return one of the prototypes.
    String label = classifier.getLabel(inputText);

    return label;
}

Notes

1. Creating a searcher can be a potentially expensive operation. So it would probably be a good idea to keep one searcher for the dataset.
2. The ISearcher is thread safe. So it can be used in a multi-threaded context.
3. Once a searcher is created and the class prototypes are fixed, it is a good idea to create one instance of the classifier (again, for efficiency).
4. Like the ISearcher, the DatalessClassifier is also thread safe.

Suppose we have an XML-RPC server running at http://server-address:8142, as described above. Here is some sample Java code that describes how to access the services. This code uses the Apache XML-RPC client.

import java.net.MalformedURLException;
import java.net.URL;

import org.apache.xmlrpc.XmlRpcException;
import org.apache.xmlrpc.client.XmlRpcClient;
import org.apache.xmlrpc.client.XmlRpcClientConfigImpl;

public class DescartesClient {
        public static void main(String[] args) throws MalformedURLException,
                        XmlRpcException {
                XmlRpcClientConfigImpl config = new XmlRpcClientConfigImpl();
                config.setServerURL(new URL("http://greedy.cs.uiuc.edu:8412"));
                XmlRpcClient client = new XmlRpcClient();
                client.setConfig(config);

                String s1 = "Bill Clinton";
                String s2 = "Barack Obama ";

                // Getting the dataless similarity between the two
                // strings using 1000 concepts.

                Object[] params = new Object[] { s1, s2, new Integer(1000) };
                Double result = (Double) client.execute("DescartesServer.similarity",
                                params);
                System.out.println(result);


                // Printing the ESA representation of the two strings
                params = new Object[] { s1, Integer.valueOf(10) };

                Object[] esa = (Object[]) client.execute("DescartesServer.esa", params);

                for (int i = 0; i < esa.length; i++) {
                        System.out.println((String)esa[i]);
                }

                System.out.println();
                params = new Object[] { s2, Integer.valueOf(10) };

                esa = (Object[]) client.execute("DescartesServer.esa", params);

                for (int i = 0; i < esa.length; i++) {
                        System.out.println((String)esa[i]);
                }

        }
}