System and method of ad-hoc analysis of data

• US 20080077582A1
• Filed: 08/31/2007
• Published: 03/27/2008
• Est. Priority Date: 09/27/2006
• Status: Active Grant

• Current Assignee: Nielsen Company (US) LLC (RR Donnelley & Sons Company)
• Original Assignee: Nielsen Company (US) LLC (RR Donnelley & Sons Company)
• Inventors: Reed, Mark
• Granted Patent: US 7,660,783 B2
• Time in Patent Office: Days
• US Class Current: 707/5
• CPC Class Codes: G06F 16/31 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

It is a first aspect of the present invention to provide a computer implemented method of performing ad-hoc analysis including the steps of: generating a text index of the textual information items, generating a metadata lookup structure based, at least in part, on the text index, searching the text index using a search query, compiling results of the text index search into aggregate information related to characteristics of the search results from the metadata items associated with the textual information items in the search results from the metadata lookup structure, and reporting the aggregate information.

Claims

1. A computer implemented method for performing ad-hoc analysis, the method comprising the steps of:

generating a text index of a plurality of textual information items;

generating a metadata lookup structure based, at least in part, on the text index, the metadata lookup structure including one or more metadata items associated with each of the textual information items;

searching the text index using one or more search queries, the searching step producing search results including one or more textual information items matching the search query;

compiling results of the text index search into aggregate information related to characteristics of the search results from the metadata items associated with each of the one or more textual information items in the search results from the metadata lookup structure; and

reporting the aggregate information.

2. The method of claim 1, further comprising the step of:

prior to generating the text index, accessing a plurality of information sources for a plurality of textual information items.

3. The method of claim 1, wherein the step of generating a metadata lookup structure based, at least in part, on the text index includes the steps of:

generating a plurality of metadata IDs, each metadata ID associated with at least a type of metadata;

analyzing each textual information item to determine which metadata ID(s) are associated with the respective textual information item; and

mapping each textual information item with the respective metadata ID(s) determined for it in the analyzing step.

4. The method of claim 1, wherein the step of generating a metadata lookup structure based, at least in part, on the text index includes the steps of:

generating one or more metadata items associated with the textual information items;

determining a quantity of the one or more metadata items; and

dynamically allocating a portion of a computer memory component based, at least in part, on the determined quantity.

5. The method of claim 1, wherein the step of generating a metadata lookup structure based, at least in part, on the text index includes the steps of:

generating one or more metadata items associated with the textual information items, wherein the number of metadata items generated is not the same for all of the textual information items;

determining a quantity of the one or more metadata items; and

dynamically allocating a portion of a computer memory component based, at least in part, on the determined quantity.

6. The method of claim 1, wherein the one or more metadata items includes at least one of date information, link information, author information, keyword information, sentiment information, demographic information, entity information, and language information.

7. The method of claim 6, wherein the link information includes a Uniform Resource Locator.

8. The method of claim 6, wherein the demographic information is generated and includes at least one of age and gender of a text item author.

9. The method of claim 6, wherein the language information includes one or more language specific annotation.

10. The method of claim 9, wherein the language specific annotation is provided by the textual information items.

11. The method of claim 9, wherein the language specific annotation is determined by analyzing the textual information items.

12. The method of claim 1, wherein the textual information items include electronic data from one or more of Internet message boards, blogs and news groups.

13. The method of claim 1, wherein the aggregate information includes at least one of date information, link information, author information, keyword information, sentiment information, demographic information, entity information, and language information pertaining to the search results.

14. The method of claim 1, wherein the text index is updated after a predetermined time period.

15. The method of claim 14, wherein the predetermined time period is between five and fifteen minutes.

16. A system for performing ad-hoc analysis, comprising:

a computer server having access to one or more information sources, the one or more information sources including a plurality of textual information items; and

a user computer device linked via one or more data links to the computer server, the user computer device including software configured to perform the steps of;

generating a text index of the textual information items;

generating a metadata lookup structure based, at least in part, on the text index, the metadata lookup structure including one or more metadata items associated with each of the textual information items;

searching the text index using one or more search queries, the searching step producing search results including one or more textual information items matching the search query;

compiling results of the text index search into aggregate information related to characteristics of the search results from the metadata items associated with each of the one or more textual information items in the search results from the metadata lookup structure; and

reporting the aggregate information.

17. The system of claim 16, wherein the step of generating a metadata lookup structure based, at least in part, on the text index includes the steps of:

generating a plurality of metadata IDs, each metadata ID associated with at least a type of metadata;

analyzing each textual information item to determine which metadata ID(s) are associated with the respective textual information item; and

mapping each textual information item with the respective metadata ID(s) determined for it in the analyzing step.

18. The system of claim 16, wherein the step of generating a metadata lookup structure based, at least in part, on the text index includes the steps of:

generating one or more metadata items associated with the textual information items;

determining a quantity of the one or more metadata items; and

dynamically allocating a portion of a computer memory component based, at least in part, on the determined quantity.

19. The system of claim 16, wherein the step of generating a metadata lookup structure based, at least in part, on the text index includes the steps of:

generating one or more metadata items associated with the textual information items, wherein the number of metadata items generated is not the same for all of the textual information items;

determining a quantity of the one or more metadata items; and

dynamically allocating a portion of a computer memory component based, at least in part, on the determined quantity.

20. The system of claim 16, wherein the one or more metadata items includes at least one of date information, link information, author information, keyword information, sentiment information, demographic information, entity information, and language information.

21. The system of claim 16, wherein the aggregate information includes at least one of date information, link information, author information, keyword information, sentiment information, demographic information, entity information, and language information pertaining to the search results.

22. The system of claim 16, wherein the text index is updated after a predetermined time period.

23. The system of claim 22, wherein the predetermined time period is between five and fifteen minutes.