APPARATUS FOR AUTOMATIC THEME DETECTION FROM UNSTRUCTURED DATA

• US 20130268534A1
• Filed: 03/01/2013
• Published: 10/10/2013
• Est. Priority Date: 03/02/2012
• Status: Active Grant

• Current Assignee: Clarabridge Incorporated
• Original Assignee: Clarabridge Incorporated
• Inventors: Mathew, Thomas, Voorhees, Kenneth Robert
• Granted Patent: US 10,372,741 B2
• Time in Patent Office: Days
• US Class Current: 707/740
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 40/237 Lexical tools
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/40 Processing or translation of natural language natural language analysis G06F40/20; semantic analysis G06F40/30

Abstract

This apparatus provides a system and method of determining significant repeating themes in a collection of documents. The apparatus operates unsupervised and leverages a natural language processing mechanism supported with lexicon, synonym and taxonomy dictionaries to determine themes and establish their relevance using a two-level hierarchical structure. The apparatus also assigns meaningful names to identified themes and determines a set of rules that describe the theme such that it can be applied to categorize other documents outside of the collection as well.

Claims

1. A system comprising:

a repository of unstructured documents;

a theme detection component configured to;

process the unstructured documents;

discover themes;

assign labels to each discovered theme;

identify patterns that describe each theme; and

organize the themes in a hierarchy; and

a user interface configured to;

allow an operator to initiate theme detection by the theme detection component; and

allow an operator to view and interact with the results of the theme detection, wherein the results comprise at least one of the assigned labels, the patterns, and the hierarchy.

2. The system of claim 1, further comprising:

a component configured to;

acquire additional unstructured documents; and

add the additional unstructured documents to the repository of unstructured documents.

3. The system of claim 1, further comprising:

a natural language processing component configured to extract features from unstructured text,

wherein the theme detection component uses the extracted features in the theme detection process.

4. The system of claim 1, further comprising:

a component configured to filter out frequently occurring noise terms from any discovered themes.

5. The system of claim 1, further comprising:

an unstructured data categorization component configured to classify unstructured text using a set of text patterns.

6. The system of claim 1, further comprising:

a sentiment computation component configured to determine sentiment associated with collected unstructured data,

wherein the theme detection component uses the determined sentiment to weight the significance of a discovered theme.

7. A method of determining themes from a collection of unstructured text documents, the method comprising:

receiving a set of unstructured text documents to process;

determining, by a computing system, frequently occurring terms within the set of unstructured text documents;

determining, by the computing system, a label for each term in the frequently occurring terms;

determining, by the computing system, one or more text patterns, wherein the one or more text patterns are used to identify if the term is contained within a document; and

creating, by the computing system, a category model to organize the identified terms as themes of top level themes.

8. The method of claim 7, further comprising:

determining, by the computing system, noise terms of the themes; and

removing the noise terms from the category model.

9. The method of claim 7, further comprising:

creating second level themes based on terms that co-occur with the top level discovered themes; and

incorporating the second level themes into the category model.

10. The method of claim 7, further comprising:

selecting nouns and verbs as top level themes using parts of speech of terms.

11. The method of claim 10, further comprising:

selecting a noun as top level themes if the noun demonstrates a tendency to appear more in sentences expressing sentiment.

12. The method of claim 7, further comprising:

using verbs as a top level theme only if the verb is associated with sentiment.

13. The method of claim 10, wherein creating second level themes based on terms that co-occur with the top level discovered themes further comprises:

creating the second level of themes using parts of speech of terms, wherein the second level themes comprise frequently co-occurring nouns and adjectives if the top level theme is a noun, and wherein the second level themes comprise frequently co-occurring nouns if the top level theme is a verb.

14. The method of claim 7, wherein the set of unstructured text documents comprises categorized documents and uncategorized documents;

and wherein receiving the set of unstructured text documents to process further comprises;

removing the categorized documents from the set of unstructured text documents prior to identifying themes.

15. The method of claim 7, further comprising:

isolating uncategorized documents to process for theme detection from a current unstructured document harvest, wherein only uncategorized documents from a document categorization step are used in the current unstructured document harvest.

16. The method of claim 7, further comprising:

grouping the top level themes using an ontology to produce one or more groups, wherein each of the one or more groups contains themes that are related according to the ontology.

17. The method of claim 7, further comprising:

grouping top level themes using a synonym dictionary to produce one or more groups, wherein each of the one or more groups contains themes that are related according to the synonym dictionary.

18. The method of claim 9, further comprising:

grouping second level themes using ontology to produce one or more groups, wherein each of the one or more groups contains themes that are related according to the ontology.

19. The method of claim 9, further comprising:

grouping second level themes using a synonym dictionary to produce one or more groups, wherein each of the one or more groups contains themes that are related according to the synonym dictionary.

20. A computer readable storage medium comprising instructions that if executed enables a computing system to:

receive a set of unstructured text documents to process;

determine frequently occurring terms within the set of unstructured text documents;

determine a label for each term in the frequently occurring terms;

determine one or more text patterns, wherein the one or more text patterns are used to identify if the term is contained within a document; and

create a category model to organize the identified terms as themes of top level themes.

21. The computer readable storage medium of claim 20, further comprising instructions that if executed enable the computing system to:

determine noise terms of the themes; and

remove the noise terms from the category model.

22. The computer readable storage medium of claim 20, further comprising instructions that if executed enable the computing system to:

create second level themes based on terms that co-occur with the top level discovered themes; and

incorporate the second level themes into the category model.

23. The computer readable storage medium of claim 20, further comprising instructions that if executed enable the computing system to:

select nouns and verbs as top level themes using parts of speech of terms.

24. The computer readable storage medium of claim 23, further comprising instructions that if executed enable the computing system to:

select a noun as top level themes if the noun demonstrates a tendency to appear more in sentences expressing sentiment.

25. The computer readable storage medium of claim 20, further comprising instructions that if executed enable the computing system to:

use verbs as a top level theme only if the verb is associated with sentiment.

26. The computer readable storage medium of claim 23, wherein the instructions for creating second level themes based on terms that co-occur with the top level discovered themes further comprise instructions that if executed enable the computing system to:

create the second level of themes using parts of speech of terms, wherein the second level themes comprise frequently co-occurring nouns and adjectives if the top level theme is a noun, and wherein the second level themes comprise frequently co-occurring nouns if the top level theme is a verb.

27. The computer readable storage medium of claim 20, wherein the set of unstructured text documents comprises categorized documents and uncategorized documents;

and wherein the instructions for receiving the set of unstructured text documents to process further comprise instructions that if executed enable the computing system to;

remove the categorized documents from the set of unstructured text documents prior to identifying themes.

28. The computer readable storage medium of claim 20, further comprising instructions that if executed enable the computing system to:

isolate uncategorized documents to process for theme detection from a current unstructured document harvest, wherein only uncategorized documents from a document categorization step are used in the current unstructured document harvest.

29. The computer readable storage medium of claim 20, further comprising instructions that if executed enable the computing system to:

group the top level themes using an ontology to produce one or more groups, wherein each of the one or more groups contains themes that are related according to the ontology.

30. The computer readable storage medium of claim 20, further comprising instructions that if executed enable the computing system to:

group top level themes using a synonym dictionary to produce one or more groups, wherein each of the one or more groups contains themes that are related according to the synonym dictionary.

31. The computer readable storage medium of claim 22, further comprising instructions that if executed enable the computing system to:

group second level themes using ontology to produce one or more groups, wherein each of the one or more groups contains themes that are related according to the ontology.

32. The computer readable storage medium of claim 22, further comprising instructions that if executed enable the computing system to:

group second level themes using a synonym dictionary to produce one or more groups, wherein each of the one or more groups contains themes that are related according to the synonym dictionary.