Text mining apparatus and associated methods

• US 20060206306A1
• Filed: 02/09/2005
• Published: 09/14/2006
• Est. Priority Date: 02/09/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Li, Hang, Cao, Yunbo, Martin, Benjamin, Ribet, Olivier
• Granted Patent: US 7,461,056 B2
• Time in Patent Office: Days
• US Class Current: 704/4
• CPC Class Codes: G06F 16/313 Selection or weighting of terms for indexing
• CPC Class Codes: G06F 16/3338 Query expansion
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: Y10S 707/99934 Query formulation, input preparation, or translation

Abstract

A method for extracting key terms and associated key terms for use in text mining is provided. The method includes receiving unstructured text documents, such as emails over a customer service system. Term candidates are extracted based on identifying consecutive word strings satisfying a context independency threshold. Term candidates are weighted using mutual information to generate a list of weighted terms. The weighted terms are then recounted. Terms are associated based on Chi-square values. Associated terms can then be used for information retrieval. A user interface can be personalized with individual user profiles.

Claims

1. A method of extracting key terms from text comprising:

receiving unstructured text documents;

detecting boundaries between sentences in the unstructured text documents to generate a sentence list;

extracting key terms from the unstructured text documents to generate key terms with weightings;

re-counting the extracted key terms to generate key terms with more accurate weightings.

2. The method of claim 1, wherein detecting boundaries comprises accessing a list of special symbols, wherein the special symbols are indicative of sentence boundaries.

3. The method of claim 1, and further comprising generating term candidates comprising:

identifying consecutive word strings using a suffix array; and

selecting term candidates from among the identified consecutive word strings based on context independency values.

4. The method of claim 3, wherein selecting term candidates comprises calculating entropy values for left and right context words and associated consecutive word strings.

5. The method of claim 3, wherein generating term candidates further comprises:

identifying consecutive word strings comprises using an LSP array in conjunction with the suffix array; and

selecting consecutive word strings having a frequency of at least 2.

6. The method of claim 3, wherein extracting key terms further comprises calculating mutual information values for the term candidates.

7. The method of claim 6, wherein calculating mutual information values comprises calculating mutual information between constituent word strings of the term candidates.

8. The method of claim 3, wherein extracting key terms comprises accessing at least one of boundary stop words and stop terms to filter out term candidates.

9. The method of claim 3, wherein extracting key terms comprises accessing a list of notable terms.

10. The method of claim 1, wherein re-counting the key terms comprises accessing at least one of a synonym list and a cluster list.

11. A method of performing text mining comprising:

receiving unstructured text documents;

generating a list of key terms from the unstructured text documents with weighting and count information;

receiving a query comprising over a user interface; and

calculating Chi-square values between the query and at least some of the key terms to identify associated terms from among the key terms.

12. The method of claim 11, and further comprising retrieving content from the unstructured text documents, wherein the content is associated with the associated terms.

13. The method of claim 12, and further comprising accessing at least one of clusters and synonyms to retrieve content from the unstructured text documents.

14. The method of claim 11, and further comprising maintaining a user profile from each user, wherein the user profile comprises at least one of notable terms, stop terms, synonyms, clusters, and categories.

15. A computer readable medium including instructions which, when implemented, cause a computer to perform text mining, the instructions comprising:

a key term extraction module adapted to identify a list of key terms in documents of unstructured text; and

a text mining module adapted to receive a query and associate at least a portion of the query with some of the key terms based on Chi-square values to generate associated terms.

16. The computer readable medium of claim 15, wherein the key term extraction module identifies consecutive word strings in the unstructured text using a suffix array and generates a list of term candidates based on context independency values.

17. The computer readable medium of claim 16, wherein the term extraction module calculates context independency values based on entropy of left context and right context word strings surrounding the consecutive word strings to generate term candidates.

18. The computer readable medium of claim 17, wherein the term extraction module calculates mutual information values of constituent word strings in term candidates to generate a list of key terms, and wherein the term extraction module re-counts at least some of the weighted term in the unstructured text.

19. The computer readable medium of claim 15, wherein the text mining module accesses at least one of synonyms and clusters to retrieve information from the unstructured documents associated with the synonyms or clusters.

20. The computer readable medium of claim 15, and further comprising a synonym construction module adapted to generate all morphological forms of key terms or their constituent word strings.

21. The computer readable medium of claim 15, wherein the text mining module retrieves information from the unstructured text that is associated with the associated terms.