Topic models

• US 8,645,298 B2
• Filed: 10/26/2010
• Issued: 02/04/2014
• Est. Priority Date: 10/26/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Hennig, Philipp, Stern, David, Graepel, Thore, Herbrich, Ralf
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Sugent, James F
• Application Number:
  US12/912,428
• Publication Number:
  US 20120101965A1
• Time in Patent Office: 1,197 Days
• Field of Search: 706/12, 706/45, 707/708, 707/727, 707/731, 707/738, 707/810
• US Class Current: 706/45
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

Machine learning techniques may be used to train computing devices to understand a variety of documents (e.g., text files, web pages, articles, spreadsheets, etc.). Machine learning techniques may be used to address the issue that computing devices may lack the human intellect used to understand such documents, such as their semantic meaning. Accordingly, a topic model may be trained by sequentially processing documents and/or their features (e.g., document author, geographical location of author, creation date, social network information of author, and/or document metadata). Additionally, as provided herein, the topic model may be used to predict probabilities that words, features, documents, and/or document corpora, for example, are indicative of particular topics.

Claims

1. A method for training a topic model, comprising:

for a document within a document corpus;

receiving a document representation of the document and features of the document, the document representation comprising a frequency of word occurrences within the document;

processing the document representation and the features using a topic model, the processing comprising;

specifying a feature/topic parameter for a feature of the document, the feature/topic parameter specifying a probability of the feature being indicative of a first topic, the feature/topic parameter based upon a first uncertainty measure that is associated with a first determination of a first deviation of the feature/topic parameter from a current feature/topic parameter for the topic model;

updating the first uncertainty measure based upon a first difference measure between the feature/topic parameter and one or more previously specified feature/topic parameters for the topic model, the first difference measure representing a first range of deviation for the first uncertainty measure;

specifying a document/word/topic parameter for a word within the document, the document/word/topic parameter specifying a probability of the word being indicative of a second topic, the document/word/topic parameter based upon a second uncertainty measure that is associated with a second determination of a second deviation of the document/word/topic parameter from a current document/word/topic parameter for the topic model; and

updating the second uncertainty measure based upon a second difference measure between the document/word/topic parameter and one or more previously specified document/word/topic parameters for the topic model, the second difference measure representing a second range of deviation for the second uncertainty measure; and

training the topic model based upon the feature/topic parameter and the document/word/topic parameter.

2. The method of claim 1, comprising:

determining a word/topic distribution prediction for a second word within a second document based upon processing the second document using the topic model, the word/topic distribution prediction specifying a probability of the second word being indicative of a third topic.

3. The method of claim 1, comprising:

receiving a reference document representation of a reference document and reference features of the reference document; and

determining a document/topic distribution prediction for the reference document based upon processing the reference document representation and the reference features using the topic model, the document/topic distribution prediction specifying a probability of a second word within the reference document stemming from a third topic.

4. The method of claim 3, comprising:

specifying an input document as comprising similar topics as the reference document.

5. The method of claim 1, comprising:

determining a feature/topic distribution prediction for a second feature based upon the second feature corresponding to one or more feature/topic parameters within the topic model.

6. The method of claim 1, comprising:

determining a corpus/topic distribution prediction for the document corpus based upon the topic model, the corpus/topic distribution prediction specifying a probability of the document corpus being indicative of a third topic.

7. The method of claim 6, comprising:

specifying an input document as comprising similar topics as the document corpus.

8. The method of claim 1, comprising:

processing the document corpus within a single iteration.

9. The method of claim 1, comprising:

detecting a new document added to the document corpus; and

processing a new document representation of the new document and new features of the new document using the topic model.

10. The method of claim 9, comprising updating the topic model based upon the processing of the new document representation and the new features.

11. The method of claim 1, comprising:

grouping one or more documents within the document corpus having semantically similar topics as determined by the topic model.

12. The method of claim 1, the features of the document comprising at least one of:

author designation;

reader designation;

geographical location;

creation date designation;

creation time designation;

document length;

document metadata;

source of the document;

document type;

social network membership of author;

web browsing history of author;

or

previous search queries of author.

13. A system for training a topic model, comprising:

one or more processing units; and

memory comprising instructions that when executed by at least one of the one or more processing units, perform a method comprising;

for a document within a document corpus;

receiving a document representation of the document and features of the document, the document representation comprising a frequency of word occurrences within the document;

processing the document representation and the features using a topic model, the processing comprising;

specifying a feature/topic parameter for a feature of the document, the feature/topic parameter specifying a probability of the feature being indicative of a first topic, the feature/topic parameter based upon a first uncertainty measure that is associated with a first determination of a first deviation of the feature/topic parameter from a current feature/topic parameter for the topic model;

updating the first uncertainty measure based upon a first difference measure between the feature/topic parameter and one or more previously specified feature/topic parameters for the topic model, the first difference measure representing a first range of deviation for the first uncertainty measure;

specifying a document/word/topic parameter for a word within the document, the document/word/topic parameter specifying a probability of the word being indicative of a second topic, the document/word/topic parameter based upon a second uncertainty measure that is associated with a second determination of a second deviation of the document/word/topic parameter from a current document/word/topic parameter for the topic model; and

updating the second uncertainty measure based upon a second difference measure between the document/word/topic parameter and one or more previously specified document/word/topic parameters for the topic model, the second difference measure representing a second range of deviation for the second uncertainty measure; and

training the topic model based upon the feature/topic parameter and the document/word/topic parameter.

14. The system of claim 13, the features of the document comprising at least one of author designation or reader designation.

15. The system of claim 13, the features of the document comprising geographical location.

16. The system of claim 13, the features of the document comprising at least one of creation date designation or creation time designation.

17. The system of claim 13, the features of the document comprising at least one of document metadata or source of the document.

18. The system of claim 13, the features of the document comprising at least one of document length or document type.

19. The system of claim 13, the features of the document comprising at least one of social network membership or web browsing history.

20. A computer readable device comprising instructions that when executed by a processing unit perform a method for training a topic model, the method comprising:

for a document within a document corpus;

receiving a document representation of the document and features of the document, the document representation comprising a frequency of word occurrences within the document;

processing the document representation and the features using a topic model, the processing comprising;

specifying a feature/topic parameter for a feature of the document, the feature/topic parameter specifying a probability of the feature being indicative of a first topic, the feature/topic parameter based upon a first uncertainty measure that is associated with a first determination of a first deviation of the feature/topic parameter from a current feature/topic parameter for the topic model;

updating the first uncertainty measure based upon a first difference measure between the feature/topic parameter and one or more previously specified feature/topic parameters for the topic model, the first difference measure representing a first range of deviation for the first uncertainty measure;

specifying a document/word/topic parameter for a word within the document, the document/word/topic parameter specifying a probability of the word being indicative of a second topic, the document/word/topic parameter based upon a second uncertainty measure that is associated with a second determination of a second deviation of the document/word/topic parameter from a current document/word/topic parameter for the topic model; and

updating the second uncertainty measure based upon a second difference measure between the document/word/topic parameter and one or more previously specified document/word/topic parameters for the topic model, the second difference measure representing a second range of deviation for the second uncertainty measure; and

training the topic model based upon the feature/topic parameter and the document/word/topic parameter.