System and method for online information analysis

• US 20070198459A1
• Filed: 02/14/2006
• Published: 08/23/2007
• Est. Priority Date: 02/14/2006
• Status: Active Grant

• Current Assignee: Accenture Global Services Limited (Accenture PLC)
• Original Assignee: Accenture Global Services Limited (Accenture PLC)
• Inventors: Richter, Owen, Sheth, Neil, Boone, Gary
• Granted Patent: US 7,685,091 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/34 Browsing; Visualisation therefor
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data

Abstract

The present disclosure includes systems and techniques relating to online information analysis. In general, in one implementation, a system includes a collection engine configured to accumulate document information retrieved from publicly accessible network resources according to predefined subjects, and an analysis engine configured to analyze the accumulated document information to identify change over a time period in general discussion of a topic within a selected subject of the predefined subjects, the analysis engine further configured to normalize the identified change over the time period based on change in a total number of documents found for the selected subject during the time period.

Claims

1. A system comprising:

a collection engine configured to accumulate document information retrieved from publicly accessible network resources according to predefined subjects; and

an analysis engine configured to analyze the accumulated document information to identify change over a time period in general discussion of a topic within a selected subject of the predefined subjects, the analysis engine further configured to normalize the identified change over the time period based on change in a total number of documents found for the selected subject during the time period.

2. The system of claim 1, wherein the collection engine is configured to employ a two-stage document scraping process, a first stage including identifying links to follow for further document accumulation, and a second stage including extracting document text for a corpus.

3. The system of claim 1, wherein the collection engine is configured to select, from a found document, a proper subset of links to follow, for further document information accumulation, based on proximity of the links to one or more keywords associated with a predefined subject.

4. The system of claim 1, wherein the collection engine is configured to vectorize found documents, and generate a reverse index of the document information.

5. The system of claim 1, further comprising a user interface available as a Web service through a public network, the user interface configured to receive a user selection of the selected subject from the predefined subjects and to receive a user-defined query indicating the topic, wherein the analysis engine is configured to analyze the accumulated document information in response to the user-defined query, the analysis engine enabling user-defined interactive term searching within the user-selected subject.

6. The system of claim 1, wherein the analysis engine is configured to analyze the accumulated document information to identify top issues, and to display indicators for the top issues, the indicators denoting a temporal change in prevalence for the top issues.

7. The system of claim 1, wherein the analysis engine is configured to sort documents into categories of analytical interest using machine learning techniques and a document training set predefined for the categories of analytical interest, and to generate an analysis of category break down for the documents.

8. The system of claim 1, wherein the analysis engine is configured to identify novel words in the accumulated document information based on exponential type growth in word prevalence.

9. A method comprising:

collecting documents from network resources according to predefined subjects to form corpora from the documents according to the predefined subjects;

weighting token values for tokens identified in respective ones of the corpora to increase values of topic-indicating tokens, from the identified tokens, within respective ones of the predefined subjects; and

generating an analysis of prevalence over a time period of one or more words in at least one of the corpora based on the weighted token values.

10. The method of claim 9, wherein the weighting comprises adjusting a token count of a given token within a given document based on weighting factors incorporating a total number of tokens in the given document, a total number of documents in the at least one of the corpora, an average document length in the at least one of the corpora, and a number of documents in the at least one of the corpora that contain the given token.

11. The method of claim 9, further comprising:

receiving, after the weighting, the one or more words in a user-defined query for a user-selected one of the predefined subjects;

wherein the generating comprises normalizing the analysis based on a sum of weighted token values for the one or more words of the user-defined query, and creating a chart plotting change in prevalence of the one or more words of the user-defined query over the time period to indicate subject matter buzz trajectories over time.

12. The method of claim 9, wherein the generating comprises:

analyzing prevalence over the time period of multiple issues in the at least one of the corpora, each issue comprising one or more words;

identifying top issues from the multiple issues based on results of the analyzing; and

displaying an indicator for each of the top issues, the indicator denoting a temporal change in prevalence of a corresponding top issue.

13. The method of claim 9, further comprising:

sorting documents in the at least one of the corpora into categories of analytical interest using machine learning techniques, the weighted token values, and a document training set predefined for the categories of analytical interest; and

wherein the generating comprises generating an analysis of category break down for the documents in the at least one of the corpora based on results of the sorting.

14. The method of claim 9, wherein the generating comprises:

determining multiple time series of prevalence over the time period for tokens in the at least one of the corpora based on the weighted token values;

checking the multiple time series for exponential type growth in prevalence; and

identifying the one or more words as newly of interest based on exponential type growth in prevalence found in the multiple time series.

15. The method of claim 9, wherein the collecting comprises:

searching the network resources using keywords associated with the predefined subjects;

identifying links to follow in documents found during the searching using multiple scrapers configured for different source document formats; and

extracting text to retain from the documents based on the keywords and based on amounts of plain text found in partitions of the documents.

16. The method of claim 9, wherein the collecting comprises:

searching the network resources using keywords associated with one of the predefined subjects;

checking for the keywords in partitions of a document found during the searching; and

selecting links from the partitions that contain the keywords for further data collection for a corpus corresponding to the one of the predefined subjects.

17. The method of claim 16, wherein the collecting further comprises:

retaining selected links for later data collection; and

selectively following the retained links such that requests for different links to a single domain are separated by a predefined time interval.

18. The method of claim 16, wherein selecting the links from the partitions comprises selecting the links based on proximity to one or more of the keywords in the partition.

19. The method of claim 16, wherein the document found during the searching is a markup language document, and the partitions are table cells in the markup language document.

20. A computer program product, encoded on a computer-readable medium, operable to cause data processing apparatus to perform operations comprising:

collecting documents from network resources according to predefined subjects to form corpora from the documents according to the predefined subjects;

weighting token values for tokens identified in respective ones of the corpora to increase values of topic-indicating tokens, from the identified tokens, within respective ones of the predefined subjects; and

generating an analysis of prevalence over a time period of one or more words in at least one of the corpora based on the weighted token values.