Systems and methods for analyzing electronic text

• US 8,606,815 B2
• Filed: 12/09/2008
• Issued: 12/10/2013
• Est. Priority Date: 12/09/2008
• Status: Active Grant

• Current Assignee: DoorDash, Inc.
• Original Assignee: International Business Machines Corporation
• Inventors: Chen, Ying, Proctor, Larry, Spangler, William Scott
• Primary Examiner(s): PEACH, POLINA G
• Application Number:
  US12/331,271
• Publication Number:
  US 20100145940A1
• Time in Patent Office: 1,827 Days
• Field of Search: None
• US Class Current: 707/777
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Systems and methods for systematically analyzing an electronic text are described. In one embodiment, the method includes receiving the electronic text from a plurality of sources. The method also includes determining an at least one term of interest to be identified in the electronic text. The method further includes identifying a plurality of locations within the electronic text including the at least one term of interest. The method also includes for each location within a plurality of locations, creating a snippet from a text segment around the at least one term of interest at the location within the electronic text. The method further includes creating multiple taxonomies for the at least one term of interest from the snippets, wherein the taxonomies include an at least one category. The method also includes determining co-occurrences between the multiple taxonomies to determine associations between categories of a different taxonomies of the multiple taxonomies.

Claims

1. A computer-implemented method for systematically analyzing an electronic text, comprising:

receiving by a computer the electronic text from a plurality of sources;

determining an at least one term of interest to be identified in the electronic text;

determining an at least one term of interest to be identified in the electronic text;

identifying by the computer a plurality of locations within the electronic text including the at least one term of interest;

for each location within a plurality of locations, creating by the computer a snippet from a text segment around the at least one term of interest at the location within the electronic text;

creating by the computer multiple taxonomies for the at least one term of interest from the snippets, wherein the taxonomies include an at least one category, the at least one category including a sentiment based taxonomy; and

determining by the computer associations between categories of a different taxonomies of the multiple taxonomies by determining;

co-occurrences between the multiple taxonomies; and

significance of co-occurrences between the multiple taxonomies,

wherein the determining the co-occurrences further comprises;

determining co-occurrences between a category of a single taxonomy and the at least one term of interest to determine significance of the at least one term of interest; and

sorting the at least one term of interest by significance; and

wherein at least one of the taxonomies is a time based taxonomy that is based on the creation date of the electronic text, the time based taxonomy generated by;

crawling sources of electronic text to extract the creation dates;

attaching an extracted creation date to a respective snippet to generate a dated snippet; and

organizing the dated snippets into chronologically contiguous categories,

wherein the sentiment based taxonomy is determined by;

creating a list of positive, negative and neutral terms indicative of different sentiments, respectively;

determining the level of sentiment corresponding to the at least one term generated from a respective snippet based on an assigned value;

normalizing the values to generate at least one term having a sentiment score corresponding thereto, the sentiment score including at least one of a positive sentiment score and a negative sentiment score; and

sorting snippets of the electronic text based on a calculated sentiment score differential between the at least one positive sentiment score and the at least one negative sentiment score.

2. The computer-implemented method of claim 1, further comprising sending the sorted at least one term of interest for user review.

3. The computer-implemented method of claim 1, wherein each taxonomy of the multiple taxonomies is one of the group consisting of:

a text clustering based taxonomy,

a taxonomy created from the occurrence of terms of interest.

4. The computer-implemented method of claim 3, further comprising:

creating from the categories of the electronic text a plurality of category/term of interest statistics of importance; and

determining from the electronic text within each category and the category/term of interest statistics the importance of each co-occurrence.

5. The computer-implemented method of claim 4, wherein the text clustering is configured to use a method based on selecting a cohesive terms of the electronic text to seed category selection, and wherein the at least one term having a sentiment score is categorized by:

identifying all terms having a positive sentiment score and summing all positive sentiment scores;

identifying all terms having a negative sentiment score and summing all negative sentiment scores; and

calculating a difference between the sum of the positive sentiment scores and the sum of the negative sentiment scores.

6. The computer-implemented method of claim 1, wherein the electronic text is web based.

7. A system for systematically analyzing an electronic text, comprising:

a receiver to receive the electronic text from a plurality of sources;

a processor coupled to the receiver to;

determine an at least one term of interest to be identified in the electronic text;

identify a plurality of locations within the electronic text including the at least one term of interest;

create for each location within a plurality of locations a snippet from a text segment around the at least one term of interest at the location within the electronic text;

create multiple taxonomies for the at least one term of interest from the snippets, wherein the taxonomies include an at least one category, the at least one category including a sentiment based taxonomy; and

determine between categories of a different taxonomies of the multiple taxonomies by determining;

co-occurrence between the multiple taxonomies;

wherein at least one of the taxonomies is a time based taxonomy that is based on the creation date of the electronic text, the time based taxonomy generated by;

crawling sources of electronic text to extract the creation dates;

attaching an extracted creation date to a respective snippet to generate a dated snippet; and

organizing the dated snippets into chronologically contiguous categories; and

a module in electrical communication with the processor, the module configured to determine co-occurrences for a single taxonomy against a term feature space to determine significance of the at least one term of interest; and

a module to sort the at least one term of interest by significance,

wherein the sentiment based taxonomy is determined by;

creating a list of positive, negative and neutral terms indicative of different sentiments, respectively;

determining the level of sentiment corresponding to the at least one term generated from a respective snippet based on an assigned value;

normalizing the values to generate at least one term having a sentiment score corresponding thereto, the sentiment score including at least one of a positive sentiment score and a negative sentiment score; and

sorting snippets of the electronic text based on a calculated sentiment score differential between the at least one positive sentiment score and the at least one negative sentiment score.

8. The system of claim 7, further comprising a module to send the sorted at least one term of interest for review.

9. The system of claim 7, wherein each taxonomy of the multiple taxonomies is one of the group consisting of:

a text clustering based taxonomy,

a taxonomy created from the occurrence of terms of interest.

10. The system of claim 7, further comprising:

determining for the at least one term of interest categories of the electronic text in the taxonomies;

creating from the categories of the electronic text a plurality of category/term of interest statistics of importance; and

determining from the electronic text within each category and the category/term of interest statistics the importance of each category.

11. The system of claim 10, wherein the text clustering is configured to use a method based on selecting a cohesive terms of the electronic text to seed category selection, and wherein the at least one term having a sentiment score is categorized by:

identifying all terms having a positive sentiment score and summing all positive sentiment scores;

identifying all terms having a negative sentiment score and summing all negative sentiment scores; and

calculating a difference between the sum of the positive sentiment scores and the sum of the negative sentiment scores.

12. The system of claim 7, wherein the electronic text is web based.

13. A computer program product comprising a non-transitory computer useable storage medium to store a computer readable program, wherein the non-transitory computer readable program, when executed on a computer, causes the computer to perform operations comprising:

receiving the electronic text from a plurality of sources;

determining an at least one term of interest to be identified in the electronic text;

determining an at least one term of interest to be identified in the electronic text;

identifying a plurality of locations within the electronic text including the at least one term of interest;

for each location within a plurality of locations, creating by a snippet from a text segment around the at least one term of interest at the location within the electronic text;

creating multiple taxonomies for the at least one term of interest from the snippets, wherein the taxonomies include an at least one category, the at least one category including a sentiment based taxonomy; and

determining associations between categories of a different taxonomies of the multiple taxonomies by determining;

co-occurrences between the multiple taxonomies; and

significance of co-occurrences between the multiple taxonomies;

determining co-occurrences between a category of a single taxonomy and the at least one term of interest to determine significance of the at least one term of interest;

sorting the at least one term of interest by a respective significance; and

outputting the sorted at least one term of interest,

wherein at least one of the taxonomies is a time based taxonomy that is based on the creation date of the electronic text, the time based taxonomy generated by;

crawling sources of electronic text to extract the creation dates;

attaching an extracted creation date to a respective snippet to generate a dated snippet; and

organizing the dated snippets into chronologically contiguous categories,

wherein the sentiment based taxonomy is determined by;

creating a list of positive, negative and neutral terms indicative of different sentiments, respectively;

determining the level of sentiment corresponding to the at least one term generated from a respective snippet based on an assigned value;

normalizing the values to generate at least one term having a sentiment score corresponding thereto, the sentiment score including at least one of a positive sentiment score and a negative sentiment score; and

sorting snippets of the electronic text based on a calculated sentiment score differential between the at least one positive sentiment score and the at least one negative sentiment score.

14. The computer program product of claim 13, wherein each taxonomy of the multiple taxonomies is one of the group consisting of:

a text clustering based taxonomy,

a taxonomy created from the occurrence of terms of interest.

15. The computer program product of claim 14, wherein the text clustering is configured to use a method based on selecting a cohesive terms of the electronic text to seed category selection, and wherein the at least one term having a sentiment score is categorized by:

identifying all terms having a positive sentiment score and summing all positive sentiment scores;

identifying all terms having a negative sentiment score and summing all negative sentiment scores; and

calculating a difference between the sum of the positive sentiment scores and the sum of the negative sentiment scores.

16. The computer program product of claim 13, wherein the electronic text is web based.