Keyword Frequency Analysis System

• US 20160154797A1
• Filed: 12/01/2014
• Published: 06/02/2016
• Est. Priority Date: 12/01/2014
• Status: Active Grant

• Current Assignee: Bank of America Corp.
• Original Assignee: Bank of America Corp.
• Inventors: Kern, Daniel C., Maher, Pasha M., Sun, Adam Z.
• Granted Patent: US 9,529,860 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/248 Presentation of query results

Abstract

According to embodiments of the present disclosure, a keyword frequency analysis system stores a plurality of sets of records. Each set of records may be associated with a dimension and may comprise a first keyword and a second keyword. The system may also receive the plurality of sets of records, determine a frequency of the first keyword in each set of records and determine a frequency of the second keyword in each set of records. The system may further determine an expected frequency of the first keyword in a first set of records associated with a first dimension, based on the frequency of the first keyword and the frequency of the second keyword. The system also compares the frequency of the first keyword and the expected frequency and, based on the comparison, determines whether the first keyword is either overrepresented or underrepresented in the first set of records.

Claims

1. A keyword frequency analysis system, comprising:

a memory operable to store a plurality of sets of records, wherein each set of records is associated with a dimension and comprises a first keyword and a second keyword;

an interface operable to;

receive the plurality of sets of records;

receive a request to determine whether the first keyword is a selected one of overrepresented or underrepresented in a first set of records, the request comprising a selection of a method to calculate an expected frequency of the first keyword;

one or more hardware processors communicatively coupled to the interface and the memory and operable to;

determine a frequency of the first keyword in each set of records;

determine a frequency of the second keyword in each set of records;

determine the method to calculate the expected frequency of the first keyword based on the selection of the method in the request to determine whether the first keyword is a selected one of overrepresented or underrepresented in the first set of records;

calculate the expected frequency of the first keyword in the first set of records associated with a first dimension using the method, the expected frequency of the first keyword being a number of times the first keyword should appear in the first set of records, the expected frequency of the first keyword based on the frequency of the first keyword and the frequency of the second keyword;

determine a difference between the frequency of the first keyword and the expected frequency;

compare the difference to a threshold, the threshold indicating whether the difference is large enough to determine one of a selected group of overrepresentation or underrepresentation;

in response to determining that the difference is not greater than the first threshold, communicate a message indicating that the first keyword is not overrepresented and not underrepresented;

in response to determining that the difference is greater than the first threshold;

determine whether the frequency of the first keyword is less than the expected frequency;

in response to determining that the frequency of the first keyword is less than the expected frequency;

determine that the first keyword is underrepresented in the first set of records;

determine a degree of underrepresentation by comparing the threshold and the difference between the frequency of the first keyword and the expected frequency;

translate the frequency of the first keyword, the frequency of the second keyword, the degree of underrepresentation, and the expected frequency into the keyword report, the keyword report comprising the expected frequency, the degree of underrepresentation, and the determination that the first keyword is underrepresented in the first set of records; and

communicate the keyword report for display.

2. The system of claim 1, wherein determining the expected frequency of the first keyword in the first set of records comprises:

determining a total first keyword frequency by summing the frequency of the first keyword in each set of records;

determining a first set of records keyword frequency by summing the frequency of the first keyword in the first set of records associated with the first dimension and the frequency of the second keyword in the first set of records associated with the first dimension;

determining a total dimension keyword frequency based on the total first keyword frequency and the first set of records keyword frequency;

based on the total first keyword frequency, first set of records keyword frequency, and the total dimension keyword frequency, determining an expected proportion;

based on the expected proportion, determining the expected frequency of the first keyword in the first set of records associated with the first dimension.

3. The system of claim 1, wherein determining the expected frequency of the first keyword in the first set of records comprises:

determining the frequency of the first keyword in each set of records for a plurality of past time periods;

calculating an average of the frequency of the first keyword for the plurality of past time periods; and

determining the expected frequency of the first keyword in the first set of records based on the average.

4. The system of claim 1, wherein the expected frequency of the first keyword in the first set of records is a range of numbers, the range of numbers corresponding to a confidence interval.

5. The system of claim 1, wherein the hardware processor is further operable to:

determine the frequency of the first keyword in the first set of records for a first time period;

determine the frequency of the second keyword in the first set of records for the first time period;

determine a correlation between the frequency of the first keyword and the frequency of the second keyword;

determine the frequency of the second keyword in the first set of records for a second time period; and

based on the correlation and on the frequency of the second keyword in the first set of records for the second time period, determine the expected frequency for the first keyword in the first set of records for the second time period.

6. The system of claim 1, wherein the hardware processor is further operable to:

generate a visualization based on the first frequency of the first keyword and the degree of underrepresentation of the first keyword; and

communicate the visualization for display.

7. (canceled)

8. A non-transitory computer-readable medium encoded with logic, the logic operable when executed to:

store a plurality of sets of records, wherein each set of records is associated with a dimension and comprises a first keyword and a second keyword;

receive the plurality of sets of records;

receive a request to determine whether the first keyword is a selected one of overrepresented or underrepresented in a first set of records, the request comprising a selection of a method to calculate an expected frequency of the first keyword;

determine a frequency of the first keyword in each set of records;

determine a frequency of the second keyword in each set of records;

determine the method to calculate the expected frequency of the first keyword based on the selection of the method in the request to determine whether the first keyword is a selected one of overrepresented or underrepresented in the first set of records;

determine the expected frequency of the first keyword in the first set of records associated with a first dimension using the method, the expected frequency of the first keyword being a number of times the first keyword should appear in the first set of records, the expected frequency of the first keyword based on the frequency of the first keyword and the frequency of the second keyword;

determine a difference between the frequency of the first keyword and the expected frequency;

compare the difference to a threshold, the threshold indicating whether the difference is large enough to determine one of a selected group of overrepresentation or underrepresentation;

in response to determining that the difference is not greater than the first threshold, communicate a message indicating that the first keyword is not overrepresented and not underrepresented;

in response to determining that the difference is greater than the first threshold;

determine whether the frequency of the first keyword is less than the expected frequency;

in response to determining that the frequency of the first keyword is less than the expected frequency;

determine that the first keyword is underrepresented in the first set of records;

determine a degree of underrepresentation by comparing the threshold and the difference between the frequency of the first keyword and the expected frequency;

translate the frequency of the first keyword, the frequency of the second keyword, the degree of underrepresentation, and the expected frequency into the keyword report, the keyword report comprising the expected frequency, the degree of underrepresentation, and the determination that the first keyword is underrepresented in the first set of records; and

communicate the keyword report for display.

9. The computer-readable medium of claim 8, wherein the logic is further operable to:

determine a total first keyword frequency by summing the frequency of the first keyword in each set of records;

determine a first set of records keyword frequency by summing the frequency of the first keyword in the first set of records associated with the first dimension and the frequency of the second keyword in the first set of records associated with the first dimension;

determine a total dimension keyword frequency based on the total first keyword frequency and the first set of records keyword frequency;

based on the total first keyword frequency, first set of records keyword frequency, and the total dimension keyword frequency, determine an expected proportion;

based on the expected proportion, determine the expected frequency of the first keyword in the first set of records associated with the first dimension.

10. The computer-readable medium of claim 8, wherein the logic is further operable to:

determine the frequency of the first keyword in each set of records for a plurality of past time periods;

calculate an average of the frequency of the first keyword for the plurality of past time periods; and

determine the expected frequency of the first keyword in the first set of records based on the average.

11. The computer-readable medium of claim 8, wherein the expected frequency of the first keyword in the first set of records is a range of numbers, the range of numbers corresponding to a confidence interval.

12. The computer-readable medium of claim 8, wherein the logic is further operable to:

determine the frequency of the first keyword in the first set of records for a first time period;

determine the frequency of the second keyword in the first set of records for the first time period;

determine a correlation between the frequency of the first keyword and the frequency of the second keyword;

determine the frequency of the second keyword in the first set of records for a second time period; and

based on the correlation and on the frequency of the second keyword in the first set of records for the second time period, determine the expected frequency for the first keyword in the first set of records for the second time period.

13. The computer-readable medium of claim 8, wherein the logic is further operable to:

generate a visualization based on the first frequency of the first keyword and the degree of underrepresentation of the first keyword; and

communicate the visualization for display.

14. A keyword frequency analysis method, comprising:

storing a plurality of sets of records in a memory, wherein each set of records is associated with a dimension and comprises a first keyword and a second keyword;

receiving, at an interface, the plurality of sets of records;

receiving, at the interface, a request to determine whether the first keyword is a selected one of overrepresented or underrepresented in a first set of records, the request comprising a selection of a method to calculate an expected frequency of the first keyword;

determining, by a processor, a frequency of the first keyword in each set of records;

determining, by the processor, a frequency of the second keyword in each set of records;

determining, by the processor, the method to calculate the expected frequency of the first keyword based on the selection of the method in the request to determine whether the first keyword is a selected one of overrepresented or underrepresented in the first set of records;

calculating, by the processor, the expected frequency of the first keyword in the first set of records associated with a first dimension using the method, the expected frequency of the first keyword being a number of times the first keyword should appear in the first set of records, the expected frequency of the first keyword based on the frequency of the first keyword and the frequency of the second keyword;

determining a difference between, by the processor, the frequency of the first keyword and the expected frequency;

comparing the difference to a threshold, the threshold indicating whether the difference is large enough to determine one of a selected group of overrepresentation or underrepresentation;

in response to determining that the difference is not greater than the first threshold, communicating a message indicating that the first keyword is not overrepresented and not underrepresented;

in response to determining that the difference is greater than the first threshold;

determining whether the frequency of the first keyword is less than the expected frequency;

in response to determining that the frequency of the first keyword is less than the expected frequency;

determining that the first keyword is underrepresented in the first set of records;

determining a degree of underrepresentation by comparing the threshold and the difference between the frequency of the first keyword and the expected frequency;

translating the frequency of the first keyword, the frequency of the second keyword, the degree of underrepresentation, and the expected frequency into the keyword report, the keyword report comprising the expected frequency, the degree of underrepresentation, and the determination that the first keyword is underrepresented in the first set of records; and

communicating the keyword report for display by the interface.

15. The method of claim 14, wherein determining the expected frequency of the first keyword in the first set of records comprises:

determining, by the processor, a total first keyword frequency by summing the frequency of the first keyword in each set of records;

determining, by the processor, a first set of records keyword frequency by summing the frequency of the first keyword in the first set of records associated with the first dimension and the frequency of the second keyword in the first set of records associated with the first dimension;

determining, by the processor, a total dimension keyword frequency based on the total first keyword frequency and the first set of records keyword frequency;

based on the total first keyword frequency, first set of records keyword frequency, and the total dimension keyword frequency, determining, by the processor, an expected proportion;

based on the expected proportion, determining, by the processor, the expected frequency of the first keyword in the first set of records associated with the first dimension.

16. The method of claim 14, wherein determining the expected frequency of the first keyword in the first set of records comprises:

determining, by the processor, the frequency of the first keyword in each set of records for a plurality of past time periods;

calculating an average of the frequency of the first keyword for the plurality of past time periods; and

determining, by the processor, the expected frequency of the first keyword in the first set of records based on the average.

17. The method of claim 14, wherein the expected frequency of the first keyword in the first set of records is a range of numbers, the range of numbers corresponding to a confidence interval.

18. The method of claim 14, further comprising:

determining, by the processor, the frequency of the first keyword in the first set of records for a first time period;

determining, by the processor, the frequency of the second keyword in the first set of records for the first time period;

determining, by the processor, a correlation between the frequency of the first keyword and the frequency of the second keyword;

determining, by the processor, the frequency of the second keyword in the first set of records for a second time period; and

based on the correlation and on the frequency of the second keyword in the first set of records for the second time period, determining, by the processor, the expected frequency for the first keyword in the first set of records for the second time period.

19. The method of claim 14, further comprising:

generating a visualization based on the first frequency of the first keyword and the degree of underrepresentation of the first keyword; and

communicating the visualization for display by the interface.

20. (canceled)