Methods and apparatus for identification and analysis of temporally differing corpora

• US 9,135,243 B1
• Filed: 03/15/2013
• Issued: 09/15/2015
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Netbase Solutions, Inc.
• Original Assignee: Netbase Solutions, Inc.
• Inventors: Tellefsen, Jens Erik, Bhatia, Ranjeet Singh
• Primary Examiner(s): Godbold, Douglas
• Application Number:
  US13/836,416
• Time in Patent Office: 914 Days
• Field of Search: 704/9
• US Class Current: 1/1
• CPC Class Codes: G06F 16/26 Visual data mining; Browsing structured data
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G06Q 30/0201 Market modelling; Market analysis; Collecting market data
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules

Abstract

Differences are identified, at the lexical unit and/or phrase level, between time-varying corpora. A corpus for a time period of interest is compared with a reference corpus. N-grams are generated for both the corpus of interest and reference corpus. Numbers of occurrences are counted. An average number of occurrences, for each n-gram of the reference corpus, is determined. A difference value, between number of occurrences in corpus of interest and average number of occurrences, is determined. Each difference value is normalized. N-grams can be selected for display, or for further processing, on the basis of the normalized difference value. Further processing can include selecting a sample period. A plurality of reference corpora are produced, where a begin time, for each sub-corpus of the plurality of reference corpora, differs, from a begin time for the corpus of interest, by an integer multiple of the sample period. Word Cloud visualization is shown.

Claims

1. A method for identifying n-grams about an object, comprising:

identifying, as a result of computing hardware and programmable memory, an object-specific corpus, that is a subset of a first corpus, where approximately all statements of the object-specific corpus are about a same first object;

identifying, as a result of computing hardware and programmable memory, statements of the object-specific corpus, for inclusion in a corpus of interest, upon a basis of a statement relating to a time period of interest;

identifying, as a result of computing hardware and programmable memory, statements of the object-specific corpus, for inclusion in a reference corpus, upon a basis of a statement relating to a reference time period that is different from the time period of interest;

identifying, as a result of computing hardware and programmable memory, n-grams of the corpus of interest, for inclusion in a corpus-of-interest list of n-grams;

identifying, as a result of computing hardware and programmable memory, n-grams of the reference corpus, for inclusion in a reference-corpus list of n-grams;

identifying, as a result of computing hardware and programmable memory, for each n-gram of the corpus-of-interest list of n-grams, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the corpus of interest;

identifying, as a result of computing hardware and programmable memory, for each n-gram of the reference-corpus list of n-grams, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the reference corpus;

determining, as a result of computing hardware and programmable memory, a selected list of n-grams from the corpus-of-interest list of n-grams or the reference-corpus list of n-grams;

determining, as a result of computing hardware and programmable memory, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, an average number of occurrences of the n-gram, in the reference-corpus;

determining, as a result of computing hardware and programmable memory, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, a difference value, between a number of occurrences of the n-gram in the corpus of interest and an average number of occurrences of the n-gram in the reference-corpus;

normalizing, as a result of computing hardware and programmable memory, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, the difference value to produce a normalized difference value; and

determining, as a result of computing hardware and programmable memory, from the selected list of n-grams, a second selected list of n-grams, on a basis of the normalized difference value.

2. The method of claim 1, wherein a duration of the reference time period is equal to an integer times a duration of the time period of interest.

3. The method of claim 1, wherein the step of determining a selected list of n-grams further comprises:

determining an n-gram, for inclusion in the selected list of n-grams, on a basis of a number of occurrences of the n-gram in the corpus of interest or a number of occurrences of the n-gram in the reference corpus.

4. The method of claim 1, wherein the step of determining an average number of occurrences of an n-gram further comprises:

determining a first multiple that, when a time duration of the corpus of interest is multiplied by, produces a time duration represented by the reference corpus; and

using the first multiple to determine an average number of occurrences of an n-gram in the reference-corpus.

5. The method of claim 4, wherein the step of using the first multiple further comprises:

dividing, for each n-gram of the selected list of n-grams, a number of occurrences of the n-gram in the reference corpus by the multiple.

6. The method of claim 1, wherein the step of normalizing, to produce a normalized difference value, further comprises:

dividing, for each n-gram of the selected list of n-grams, a difference value by an average number of occurrences of the n-gram in the reference-corpus.

7. The method of claim 1, wherein the step of normalizing, to produce a normalized difference value, further comprises:

dividing, for each n-gram of the selected list of n-grams, a difference value by a number of occurrences of the n-gram in the corpus of interest.

8. The method of claim 1, wherein the step of normalizing, to produce a normalized difference value, further comprises:

dividing, for each n-gram of the selected list of n-grams, a difference value by a standard deviation of occurrences of the n-gram in the reference-corpus.

9. The method of claim 1, wherein the step of determining a second selected list of n-grams further comprises:

ordering the selected list of n-grams on a basis of decreasing normalized difference value; and

selecting a first predetermined number of n-grams from the ordered selected list of n-grams.

10. The method of claim 1, further comprising:

varying a first graphical dimension or characteristic, for a display of each n-gram of the second selected list of n-grams, on a basis of the normalized difference value of the n-gram.

11. The method of claim 10, wherein the first graphical dimension is a font size, for a display of each n-gram of the second selected list of n-grams.

12. The method of claim 11, wherein the second selected list of n-grams is displayed as a word cloud.

13. The method of claim 10, further comprising:

varying a second graphical dimension or characteristic, for a display of each n-gram of the second selected list of n-grams, on a basis of a number of occurrences of the n-gram.

14. The method of claim 13, wherein the first graphical characteristic is a font transparency and the second graphical dimension is a font size, for a display of each n-gram of the second selected list of n-grams.

15. The method of claim 14, wherein the second selected list of n-grams is displayed as a word cloud.

16. The method of claim 1, further comprising:

selecting a sample period;

producing a plurality of reference corpora, wherein a begin time, for each sub-corpus of the plurality of reference corpora, differs, from a begin time for the corpus of interest, by an integer multiple of the sample period;

identifying, for each n-gram of the second selected list of n-grams and for each sub-corpus of the plurality of reference corpora, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the sub-corpus;

determining, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, a second average number of occurrences of the n-gram, from a number of occurrences of the n-gram in each sub-corpus;

determining, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, a second difference value, between a number of occurrences of the n-gram in the corpus of interest and a second average number of occurrences of the n-gram;

normalizing, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, the second difference value to produce a second normalized difference value; and

determining, from the second selected list of n-grams, a third selected list of n-grams, on a basis of the second normalized difference value.

17. The method of claim 16, wherein the step of normalizing, to produce a second normalized difference value, further comprises:

determining, for each n-gram of the second selected list of n-grams, a standard deviation from a number of occurrences of the n-gram in each sub-corpus;

dividing, for each n-gram of the second selected list of n-grams, a second difference value by a standard deviation.

18. The method of claim 16, further comprising:

varying a first graphical dimension or characteristic, for a display of each n-gram of the third selected list of n-grams, on a basis of the second normalized difference value of the n-gram.

19. The method of claim 18, wherein the first graphical dimension is a font size, for a display of each n-gram of the second selected list of n-grams.

20. The method of claim 18, further comprising:

varying a second graphical dimension or characteristic, for a display of each n-gram of the second selected list of n-grams, on a basis of a number of occurrences of the n-gram.

21. The method of claim 20, wherein the first graphical characteristic is a font transparency and the second graphical dimension is a font size, for a display of each n-gram of the second selected list of n-grams.

22. The method of claim 1, wherein the first object is a brand.

23. A system for identifying n-grams about an object, comprising:

one or more processors and programmable memory, wherein the system is configured;

to accomplish identifying an object-specific corpus, that is a subset of a first corpus, where approximately all statements of the object-specific corpus are about a same first object;

to accomplish identifying statements of the object-specific corpus, for inclusion in a corpus of interest, upon a basis of a statement relating to a time period of interest;

to accomplish identifying statements of the object-specific corpus, for inclusion in a reference corpus, upon a basis of a statement relating to a reference time period that is different from the time period of interest;

to accomplish identifying n-grams of the corpus of interest, for inclusion in a corpus-of-interest list of n-grams;

to accomplish identifying n-grams of the reference corpus, for inclusion in a reference-corpus list of n-grams;

to accomplish identifying, for each n-gram of the corpus-of-interest list of n-grams, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the corpus of interest;

to accomplish identifying, for each n-gram of the reference-corpus list of n-grams, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the reference corpus;

to accomplish determining a selected list of n-grams from the corpus-of-interest list of n-grams or the reference-corpus list of n-grams;

to accomplish determining, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, an average number of occurrences of the n-gram, in the reference-corpus;

to accomplish determining, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, a difference value, between a number of occurrences of the n-gram in the corpus of interest and an average number of occurrences of the n-gram in the reference-corpus;

to accomplish normalizing, for each n-gram of the selected list of n-grams, for subsequent access in conjunction with an n-gram, the difference value to produce a normalized difference value; and

to accomplish determining, from the selected list of n-grams, a second selected list of n-grams, on a basis of the normalized difference value.

24. The system of claim 23, wherein the configuration to accomplish determining, an average number of occurrences of an n-gram, is further configured:

to accomplish determining a first multiple that, when a time duration of the corpus of interest is multiplied by, produces a time duration represented by the reference corpus; and

to accomplish using the first multiple to determine an average number of occurrences of an n-gram in the reference-corpus.

25. The system of claim 23, wherein the configuration to accomplish normalizing, to produce a normalized difference value, is further configured:

to accomplish dividing, for each n-gram of the selected list of n-grams, a difference value by a standard deviation of occurrences of the n-gram in the reference-corpus.

26. The system of claim 23, further configured:

to accomplish varying a first graphical dimension or characteristic, for a display of each n-gram of the second selected list of n-grams, on a basis of the normalized difference value of the n-gram.

27. The system of claim 26, further configured:

to accomplish varying a second graphical dimension or characteristic, for a display of each n-gram of the second selected list of n-grams, on a basis of a number of occurrences of the n-gram.

28. The system of claim 23, further configured:

to accomplish selecting a sample period;

to accomplish producing a plurality of reference corpora, wherein a begin time, for each sub-corpus of the plurality of reference corpora, differs, from a begin time for the corpus of interest, by an integer multiple of the sample period;

to accomplish identifying, for each n-gram of the second selected list of n-grams and for each sub-corpus of the plurality of reference corpora, for subsequent access in conjunction with an n-gram, a number of occurrences of the n-gram in the sub-corpus;

to accomplish determining, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, a second average number of occurrences of the n-gram, from a number of occurrences of the n-gram in each sub-corpus;

to accomplish determining, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, a second difference value, between a number of occurrences of the n-gram in the corpus of interest and a second average number of occurrences of the n-gram;

to accomplish normalizing, for each n-gram of the second selected list of n-grams, for subsequent access in conjunction with an n-gram, the second difference value to produce a second normalized difference value; and

to accomplish determining, from the second selected list of n-grams, a third selected list of n-grams, on a basis of the second normalized difference value.