System and methods for automated document topic discovery, browsable search and document categorization

• US 8,843,476 B1
• Filed: 05/18/2010
• Issued: 09/23/2014
• Est. Priority Date: 03/16/2009
• Status: Expired due to Fees

• Current Assignee: Linfo IP LLC (Pueblo Nuevo LLC)
• Original Assignee: Guangsheng Zhang
• Inventors: Zhang, Guangsheng
• Primary Examiner(s): Alam, Hosain
• Assistant Examiner(s): Ahluwalia, Navneet K
• Application Number:
  US12/782,545
• Time in Patent Office: 1,589 Days
• Field of Search: 707/723
• US Class Current: 707/723
• CPC Class Codes: G06F 16/345 Summarisation for human users
• CPC Class Codes: G06F 16/35 Clustering; Classification

Abstract

A computer-assisted method for discovering topics in a document collection is disclosed. The method includes obtaining a group of text units in the document collection, tokenizing the words in the group of text units to produce a plurality of tokens that include a jth token, and adding a weighting coefficient to a parameter token_j_count for each text unit in the first group that includes the jth token. The weighting coefficient is dependent on the grammatical role of the jth token. The method includes calculating an internal term prominence value (ITP) using token_j_count, selecting one or more tokens from the tokens based on the ITP values of the respective tokens, and outputting the one or more selected tokens as topic terms associated with the document collection.

Claims

1. A computer-assisted method for automatically discovering topics in a document collection comprising one or more documents that comprise sentences and terms, comprising:

automatically dividing each of the sentences into the grammatical components of a subject and a predicate by a computer system, wherein the predicate is defined as the portion of a sentence other than the subject;

assigning a first weighting coefficient to subjects in the sentences;

assigning a second weighting coefficient to predicates in the sentences;

tokenizing the sentences to produce a plurality of tokens, wherein each of the terms is associated with one or more of the plurality of tokens;

for each one of the terms,

calculating a first token count in the plurality of tokens in which the one of the terms matches a subject in the sentences;

multiplying the first weighting coefficient with the first token count to produce a first weighted token count;

calculating a second token count in the plurality of tokens in which the one of the terms matches a predicate in the sentences;

multiplying the second weighting coefficient with the second token count to produce a second weighted token count;

producing a first score value for the one of the terms based on the first weighted token count and the second weighted token count, wherein the first score can be referred to as an internal term prominence (ITP) value in comparison to a second value, wherein the second value can be referred to as an external term prominence (ETP) value, wherein the ETP value represents the prominence of the term outside the document collection;

selecting one or more terms from the terms at least in part based on the first score values of the terms; and

outputting the one or more terms representing one or more topics associated with the document collection.

2. The computer-assisted method of claim 1, wherein the document collection can be divided into a plurality of text units, wherein each text unit can be a sentence, a paragraph, a document, or a group of documents, wherein the first score can further be divided by the total number of the text units.

3. The computer-assisted method of claim 1, further comprising:

obtaining a data set comprising a plurality of ETP values each associated with one of the terms, and

producing a third score value for each of the terms using its associated ITP and ETP value, wherein the third score value can be referred to as the collection-specific term prominence (CSTP) value, wherein the one or more terms are selected from the terms using the CSTP values of the respective terms as representing one or more topics associated with the document collection.

4. The computer-assisted method of claim 3, wherein the one or more terms selected from the plurality of tokens have their respective CSTP values above a predetermined threshold.

5. The computer-assisted method of claim 3, wherein the CSTP value is equal to the ITP value subtracted by the ETP value.

6. The computer-assisted method of claim 3, wherein the CSTP value is the square of ITP value divided by a sum of the ITP value and the ETP value.

7. The computer-assisted method of claim 1, wherein some of the subjects in the sentences comprise multi-word phrases, the computer-assisted method further comprising:

dividing each of the multi-word phrases in the subjects in the sentences into the grammatical components of a head and a modifier;

assigning a third weighting coefficient to heads in the multi-word phrases in the subjects in the sentences;

assigning a fourth weighting coefficient to modifiers in the multi-word phrases in the subjects in the sentences;

for each one of the terms,

calculating a third token count in the plurality of tokens in which the one of the terms matches a head in a subject in the sentences;

multiplying the third weighting coefficient with the third token count to produce a third weighted token count;

calculating a fourth token count in the plurality of tokens in which the one of the terms matches a modifier in a subject in the sentences;

multiplying the fourth weighting coefficient with the fourth token count to produce a fourth weighted token count; and

producing the first score based on the first weighted token count, the second weighted token count, the third weighted token count, and the fourth weighted token count.

8. The computer-assisted method of claim 1, wherein some of the predicates in the sentences comprise multi-word phrases, the computer-assisted method further comprising:

dividing each of the multi-word phrases in the predicates in the sentences into the grammatical components of a head and a modifier;

assigning a fifth weighting coefficient to heads in the multi-word phrases in the predicates in the sentences;

assigning a sixth weighting coefficient to modifiers in the multi-word phrases in the predicates in the sentences;

for each one of the terms,

calculating a fifth token count in the plurality of tokens in which the one of the terms matches a head in a predicate in the sentences;

multiplying the fifth weighting coefficient with the fifth token count to produce a fifth weighted token count;

calculating a sixth token count in the plurality of tokens in which the one of the terms matches a modifier in a predicate in the sentences;

multiplying the sixth weighting coefficient with the sixth token count to produce a sixth weighted token count; and

producing the first score based on the first weighted token count, the second weighted token count, the fifth weighted token count, and the sixth weighted token count.

9. The computer-assisted method of claim 1, wherein at least some of the sentences are obtained from one or more files in a file folder in a computer file system, or one or more emails in an email folder or an email mailbox, or a collection of other types of text contents including web contents.

10. The computer-assisted method of claim 1, further comprising:

displaying, at a user interface, the one or more selected terms, or at least a portion of the one or more documents in the document collection in connection with the display of the one or more selected terms.

11. The computer-assisted method of claim 1, further comprising:

grouping documents in the document collection into one or more categories or groups based on the one or more selected terms contained in the respective documents, wherein the selected terms can each represent a document category or group; and

displaying, at a user interface, at least a portion of the documents associated with each category or each group in connection with the display of the one or more selected terms.

12. A system for automatically discovering topics in a document collection comprising one or more documents that comprise sentences and terms, comprising:

a computer processor configured to automatically divide each of the sentences into the grammatical components of a subject and a predicate by a computer system, wherein the predicate is defined as the portion of a sentence other than the subject;

assign a first weighting coefficient to subjects in the sentences;

assign a second weighting coefficient to predicates in the sentences;

tokenize the sentences to produce a plurality of tokens, wherein each of the terms is associated with one or more of the plurality of tokens;

for each one of the terms,

calculate a first token count in the plurality of tokens in which the one of the terms matches a subject in the sentences;

multiply the first weighting coefficient with the first token count to produce a first weighted token count;

calculate a second token count in the plurality of tokens in which the one of the terms matches a predicate in the sentences;

multiply the second weighting coefficient with the second token count to produce a second weighted token count;

produce a first value for the one of the terms based on the first weighted token count and the second weighted token count, wherein the first value can be referred to as an internal term prominence (ITP) value in comparison to a second value, wherein the second value can be referred to as an external term prominence (ETP) value, wherein the ETP value represents the prominence of the term outside the document collection;

select one or more terms from the terms at least in part based on the score values of the terms; and

output the one or more terms to represent the topics associated with the document collection.

13. A computer-assisted method for determining a term score and displaying terms extracted from a document collection, comprising:

obtaining, by a computer system, a document collection comprising one or more documents each comprising sentences and terms, wherein each term comprises a word or a multi-word phrase;

wherein a multi-word phrase comprises two or more sub-phrases, wherein each sub-phrase comprises a word or another multi-word phrase;

assigning a sub-phrase weighting co-efficient to the sub-phrase of a multi-word phrase;

tokenizing the sentences to produce a plurality of words or multi-word phrases as tokens, wherein a term matches with one or more of the plurality of tokens;

for one or more terms in the document collection,

calculating a sub-phrase token count of the term for each token of the term that is a sub-phrase of a multi-word phrase;

multiplying the sub-phrase weighting co-efficient with the sub-phrase token count to produce a weighted sub-phrase token count for the term;

calculating a non-sub-phrase token count of the term for each token of the term that is not a sub-phrase of a multi-word phrase;

producing a score value for the term based on the weighted sub-phrase token count and the non-sub-phrase token count, wherein the score value can be referred to as an internal term prominence value in comparison to a second value, wherein the second value can be referred to as an external term prominence value, wherein the second value represents the prominence of the term outside the document collection;

selecting one or more terms at least in part based on the score values of the terms; and

outputting, for storage or display, the selected terms.

14. The computer-assisted method of claim 13, wherein the sub-phrases in a multi-word phrase are further identified as a first sub-phrase and a second sub-phrase, the method further comprising:

assigning a first weighting co-efficient to the first sub-phrase;

assigning a second weighting co-efficient to the second sub-phrase;

calculating a first token count of the term for each token of the term that is a first sub-phrase of a multi-word phrase;

multiplying the first weighting co-efficient with the first token count to produce a first weighted token count for the term;

calculating a second token count of the term for each token of the term that is a second sub-phrase of a multi-word phrase;

multiplying the second weighting co-efficient with the second token count to produce a second weighted token count of the term; and

producing a score value for the term based on the first weighted token count or the second weighted token count.

15. The computer-assisted method of claim 14, wherein the first sub-phrase can be associated with the grammatical role of a modifier in the multi-word phrase, and the second sub-phrase can be associated with the grammatical role of a head in the multi-word phrase.

16. The computer-assisted method of claim 13, wherein the document collection can be divided into a plurality of text units, wherein each text unit can be a sentence, a paragraph, a document, or a group of documents, wherein the score value can further be divided by the total number of the text units.

17. The computer-assisted method of claim 13, wherein the documents in the document collection are one or more files in a file folder in a computer file system, or one or more emails in an email folder or an email mailbox, or obtained from a collection of other types of text contents including web contents.

18. The computer-assisted method of claim 13, further comprising:

displaying, at a user interface, the selected terms, or at least a portion of the one or more documents associated with the selected terms in connection with the display of the selected terms.

19. The computer-assisted method of claim 13, further comprising:

grouping documents in the document collection into one or more categories or groups based on the selected terms contained in the respective documents, wherein the selected terms can each represent a document category or group.

20. The computer-assisted method of claim 19, further comprising:

displaying, at a user interface, at least a portion of the documents associated with each category or each group in connection with the display of the selected terms.