Automated topic discovery in documents and content categorization

• US 9,047,283 B1
• Filed: 12/07/2012
• Issued: 06/02/2015
• Est. Priority Date: 01/29/2010
• Status: Expired due to Fees

• Current Assignee: Linfo IP LLC (Pueblo Nuevo LLC)
• Original Assignee: Guangsheng Zhang
• Inventors: Zhang, Guangsheng, Zhang, Chizhong
• Primary Examiner(s): Ly, Anh
• Application Number:
  US13/707,940
• Time in Patent Office: 907 Days
• Field of Search: 707/749, 707/772, 707/748, 707/730, 707/731, 707/706, 707/722, 707/752, 707/736, 707/728, 707/723, 707/750, 707/708, 704/4, 704/9, 706/12, 706/14
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24553 of query operations
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/3334 Selection or weighting of terms from queries, including natural language queries
• CPC Class Codes: G06F 16/3346 using probabilistic model
• CPC Class Codes: G06F 16/355 Class or cluster creation or modification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/10 Text processing natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40
• CPC Class Codes: G06F 40/205 Parsing
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A computer-assisted method for discovering topics and categorizing contents in a document includes the steps of calculating an importance score for a term based on grammatical roles, parts of speech, and semantic attributes, selecting terms based on the importance score values of the respective terms, and outputting terms comprising the selected term to represent topics in the document, and building a category structure based on the selected terms.

Claims

1. A method implemented on a computer comprising one or more processors and memory, comprising:

receiving a text content;

tokenizing the text content into a plurality of terms, each term comprising one or more words or phrases;

identifying a first semantic attribute, wherein the first semantic attribute is selected from the group of attributes consisting of at least an action, a thing, a person, an agent of an action, a recipient of an action or a thing, a state of an object, a mental state of a person, a physical state of a person, a positive or negative opinion, a name of a product, a name of a service, a name of an organization;

identifying a first term in the text content, wherein the first term is associated with the first semantic attribute;

identifying a second term in the text content, wherein the second term is not associated with the first semantic attribute;

assigning an importance value to the first term as bearing more importance than the second term based on the first semantic attribute, wherein the importance value is a measurement for the role of the first term in representing a topic or an information focus in the text content; and

outputting the first term or the second term to represent the content of the document,

when the first term is output, the function of the first term includes being a tag or a label to represent a topic or a summary of the text content, or a category node,

when the first term and the second term are output and displayed, the display format includes selecting the font type, size, color, shape, position, or orientation of or distance between the first term and the second term based on the importance value,

when the text content containing the first term is made searchable using a query or is associated with a search index to produce a search result, the search result is ranked based at least on the importance value.

2. The method of claim 1, further comprising:

determining a first term importance score for the first term based on the first semantic attribute, or

identifying a second semantic attribute associated with the second term, and determining a second term importance score for the second term based on the second semantic attribute.

3. The method of claim 2, further comprising:

assigning a first weighting co-efficient to the first semantic attribute, wherein the first term importance score is determined based on the first weighting co-efficient;

or

assigning a second weighting co-efficient to the second semantic attribute, wherein the second term importance score is determined based on the second weighting co-efficient.

4. The method of claim 2, further comprising:

counting a first frequency of the first term in the text content, wherein the first term importance score is determined based on the first frequency;

or

counting a second frequency of the second term in the text content, wherein the second term importance score is determined based on the second frequency.

5. The method of claim 2, further comprising:

outputting the first term importance score or a first text unit, wherein the first text unit includes the first term, wherein the first text unit is associated with an attribute, wherein the attribute includes linking to at least a portion of the text content associated with the first term;

or

outputting the second term importance score or a second text unit, wherein the second text unit includes the second term, wherein the second text unit is associated with an attribute, wherein the attribute includes linking to at least a portion of the text content associated with the second term.

6. The method of claim 5, wherein the text content is displayed in a user interface, the method further comprising:

receiving, via a user interface object, a user indication; and

highlighting, by the computer system, the first term or the second term in the text content according to the user indication.

7. The method of claim 2, wherein the text content is made searchable using a query, or is associated with a search index, the method further comprising:

receiving a search query comprising a keyword;

matching the keyword with the first term or the second term;

returning the text content as a search result or part of a search result; and

ranking the search result based at least on the first term importance score or the second term importance score.

8. The computer-assisted method of claim 1, wherein the text content is a sub-segment text unit in a collection of sub-segment text units, wherein the sub-segment text unit is a sentence or a paragraph and the collection is a document comprising a plurality of sentences or paragraphs, or the sub-segment is an individual document and the collection is a document collection containing a plurality of documents.

9. A method implemented on a computer comprising one or more processors and memory, comprising:

receiving a text content;

tokenizing the text content into a plurality of terms, each term comprising one or more words or phrases;

identifying a first part of speech, wherein the first part of speech includes at least a noun or a pronoun, a transitive or intransitive verb, a modal verb, a link verb, an adjective, an adverb, a preposition, an article, a conjunction;

identifying a first term in the text content, wherein the first term is associated with the first part of speech;

identifying a second term in the text content, wherein the second term is not associated with the first part of speech;

assigning an importance value to the first term as bearing more importance than the second term based on the first part of speech, wherein the importance is a measurement for the role of the first term in representing a topic or an information focus in the text content; and

outputting the first term or the second term to represent the content of the document,

when the first term is output, the function of the first term includes being a tag or a label to represent a topic or a summary of the text content, or a category node,

when the first term and the second term are output and displayed, the display format includes selecting the font type, size, color, shape, position, or orientation of or distance between the first term and the second term based on the importance value,

when the text content containing the first term is made searchable using a query or is associated with a search index to produce a search result, the search result is ranked based at least on the importance value.

10. The method of claim 9, further comprising:

determining a first term importance score for the first term based on the first part of speech, or

identifying a second part of speech associated with the second term, and determining a second term importance score for the second term based on the second part of speech.

11. The method of claim 10, further comprising:

assigning a first weighting co-efficient to the first part of speech, wherein the first term importance score is determined based on the first weighting co-efficient;

or

assigning a second weighting co-efficient to the second part of speech, wherein the second term importance score is determined based on the second weighting co-efficient.

12. The method of claim 10, further comprising:

counting a first frequency of the first term in the text content, wherein the first term importance score is determined based on the first frequency;

or

counting a second frequency of the second term in the text content, wherein the second term importance score is determined based on the second frequency.

13. The method of claim 10, further comprising:

outputting the first term importance score or a first text unit, wherein the first text unit includes the first term, wherein the first text unit is associated with an attribute, wherein the attribute includes linking to at least a portion of the text content associated with the first term;

or

outputting the second term importance score or a second text unit, wherein the second text unit includes the second term, wherein the second text unit is associated with an attribute, wherein the attribute includes linking to at least a portion of the text content associated with the second term.

14. The method of claim 13, wherein the text content is displayed in a user interface, the method further comprising:

receiving, via a user interface object, a user indication; and

highlighting, by the computer system, the first term or the second term in the text content according to the user indication.

15. The method of claim 10, wherein the text content is made searchable using a query, or is associated with a search index, the method further comprising:

receiving a search query comprising a keyword;

matching the keyword with the first term or the second term;

returning the text content as a search result or part of a search result; and

ranking the search result based at least on the first term importance score or the second term importance score.

16. The method of claim 9, wherein the text content is a sub-segment text unit in a collection of sub-segment text units, wherein the sub-segment text unit is a sentence or a paragraph and the collection is a document comprising a plurality of sentences or paragraphs, or the sub-segment is an individual document and the collection is a document collection containing a plurality of documents.

17. A system for processing or presenting information, comprising:

one or more processors and memory configured to

receive a first term extracted from a text content;

receive a second term extracted from at least a portion of the text content, wherein the portion of the text content contains or is associated with the first term;

display the first term as a first-level entity in a hierarchical format;

display the second term as a second-level entity in the hierarchical format, wherein the second-level entity is displayed as an element under or subordinate to the first level entity, and

wherein the first term is extracted from the text content with the steps of;

(a) tokenizing the text content into a plurality of terms comprising the first term and the second term, each term comprising an element selected from the group consisting of a word, a phrase, a sentence, a paragraph,

(b) defining a first grammatical attribute or a first semantic attribute,

(c) pre-determining a first importance measure for the first grammatical attribute or the first semantic attribute,

(d) identifying a term associated with the first grammatical attribute or the first semantic attribute, and

(e) selecting the term as the first term based at least on the first grammatical attribute or the first semantic attribute or the first importance measure;

when the first term represents a first-level category node, and the second term represents a second-level category node or the content of the first-level category, the embodiment of at least one category node includes a folder or a directory, or a link name associated with the linked contents on a device selected from the group consisting at least of a computer file system, an email system, a web-based or cloud-based system, a mobile or handheld computing or communication device;

when the first term and the second term are displayed, the display format includes representing the first term as a topic or information focus in the text content, and the second term as a comment or attribute associated with the topic or the information focus;

when the text content or the first term is made searchable using a query or is associated with a search index to produce a search result, the display format of the search result includes the first term with one or more of its corresponding second terms if the first term matches a keyword in the search query.

18. The system of claim 17, wherein the second term is a term that is extracted from the text content with steps including the following:

(a) obtaining a portion of the text content, wherein the portion of the text content contains or is associated with the first term;

(b) tokenizing the portion of the text content into a plurality of terms comprising at least the second term, each term comprising an element selected from the group consisting of a word, a phrase, a sentence, a paragraph;

(c) defining a second grammatical attribute or a second semantic attribute, and pre-determining a second importance measure for the second grammatical attribute or the second semantic attribute;

(d) identifying a term associated with the second grammatical attribute or the second semantic attribute, or associated with the first grammatical attribute or the first semantic attribute; and

(e) selecting the term as the second term based at least on the second importance measure, or the first importance measure.

19. The system of claim 18, wherein the second term is grammatically related to the first term;

wherein the grammatical relationship between the first term and the second term includes a relationship between a subject and a non-subject of a sentence, or a modifier and a head of a complex phrases.

20. The system of claim 18, wherein the steps to extract the second term further include:

selecting the term as the second term if the term is different from the first term.

21. The system of claim 17, wherein the second term is a term that is extracted from the text content with steps including the following:

(a) obtaining a portion of the text content, wherein the portion of the text content contains or is associated with the first term;

(b) tokenizing the portion of the text content into a plurality of terms comprising at least the second term, each term comprising an element selected from the group consisting of a word, a phrase, a sentence, a paragraph;

(c) for one or more terms in the plurality of terms, counting the occurrence of each term in the portion of the text content; and

(d) selecting at least one of the terms as the second term if the occurrence of the term in the portion of the text content is above a threshold.

22. The system of claim 17, wherein the steps to extract the first term further include:

for one or more terms in the plurality of terms, counting the occurrence of each term in the text content; and

selecting at least one of the terms as the first term if the occurrence of the term in the text content is above a threshold.

23. The system of claim 17, wherein the first importance measure is represented by a first importance score, wherein the steps to extract the first term further include:

assigning a first weighting co-efficient to the first grammatical attribute or the first semantic attribute, wherein the first importance score is determined based on the first weighting co-efficient.

24. The system of claim 17, wherein

the first term represents a topic or information focus in the text content, and the second term represents a comment or attribute associated with the topic or the information focus, the system is further configured to link the first term or the second term to a portion of the text content associated with the first term or the second term.

25. The system of claim 17, wherein the text content includes user reviews of a product or a service, or comments on social or political or financial or other topics, wherein the first term represents a major topic in the text content, and the second term represents a minor topic or a description about the major topic;

wherein the first semantic attribute or the second semantic attribute includes at least a positive and a negative opinion, the system is further configured to;

in response to a user indication, selectively display, hide, or highlight the first term or the second term based on whether the first term or the second term indicates a positive or a negative opinion;

or to selectively highlight one or more terms in the text content associated with the first term or the second term, according to whether the one or more terms indicate a positive or negative opinion.