Analyzing large data sets to find deviation patterns

• US 9,390,121 B2
• Filed: 07/11/2014
• Issued: 07/12/2016
• Est. Priority Date: 03/18/2005
• Status: Active Grant

• Current Assignee: Salesforce.com, Inc.
• Original Assignee: BeyondCore, Inc. (Salesforce.com, Inc.)
• Inventors: Sengupta, Arijit, Stronger, Brad A., Kane, Daniel
• Primary Examiner(s): Woo, Isaac M
• Application Number:
  US14/328,705
• Publication Number:
  US 20140324787A1
• Time in Patent Office: 732 Days
• Field of Search: 707600-899
• US Class Current: 1/1
• CPC Class Codes: G06F 16/00 Information retrieval; Database structures therefor; File system structures therefor
• CPC Class Codes: G06F 16/217 Database tuning G06F16/2282 takes precedence; database performance monitoring G06F11/3409
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06Q 10/06 Resources, workflows, human or project management; Enterprise or organisation planning; Enterprise or organisation modelling
• CPC Class Codes: G06Q 10/06393 Score-carding, benchmarking or key performance indicator [KPI] analysis
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records

Abstract

Operations, such as data processing operations, can be improved by applying clustering and statistical techniques to observed behaviors in the data processing operations.

Claims

1. A computer-implemented method for identifying abnormal behavior of sets of operators, the method comprising:

observing behaviors of the operators;

separating operators into groupings, each grouping comprising a plurality of operators;

subsequent to the separating, calculating a test statistic for each grouping based on the observed behaviors for the operators in the grouping, the test statistic testing whether all operators behave in a same manner; and

identifying abnormal behavior for one grouping relative to other groupings, based on statistically significant deviations of the test statistic for the one grouping relative to a normative baseline of similarity amongst the other groupings.

2. The computer-implemented method of claim 1 wherein an output produced by the operators is a function of a set of inputs to the operators, and the test statistic is calculated based on less than all of the inputs.

3. The computer-implemented method of claim 1 wherein an output produced by the operators is a function of a set of inputs to the operators, and the test statistic is based on whether operators produce similar outputs given similar inputs.

4. The computer-implemented method of claim 1 wherein separating operators into groupings comprises:

iteratively assigning operators to randomly defined groupings; and

computing the test statistic for the randomly defined groups for each iteration until abnormal behavior is identified.

5. The computer-implemented method of claim 1 wherein separating operators into groupings comprises repeatedly assigning operators into different groupings under different grouping plans.

6. The computer-implemented method of claim 1 further comprising:

for a grouping that has a statistically significant deviation of the test statistic relative to other groupings, observing behaviors of the operators in that grouping using a test data set.

7. The computer-implemented method of claim 1 wherein an output produced by the operators is a binary output.

8. The computer-implemented method of claim 1 wherein the observed behaviors are divided into a training set and a testing set;

the training set is used to identify candidate abnormal behaviors and the testing set is used to test whether the candidate abnormal behaviors are actually abnormal.

9. The computer-implemented method of claim 1 wherein the test statistic comprises an estimate of a variance in the means of the operators output.

10. The computer-implemented method of claim 1 further comprising using a search method to refine the groupings.

11. The computer-implemented method of claim 1, wherein:

each of the groupings includes employees of a distinct organization; and

the test statistic for each grouping is an error metric representing data entry inaccuracies detected for employees included in the grouping.

12. A computer program product for identifying abnormal behavior of sets of operators, the computer program product stored on a non-transitory computer-readable medium and including instructions that, when loaded into memory, cause a processor to carry out a method comprising:

observing behaviors of the operators;

separating operators into groupings, each grouping comprising a plurality of operators;

subsequent to the separating, calculating a test statistic for each grouping based on the observed behaviors for the operators in the grouping, the test statistic testing whether all operators behave in a same manner; and

identifying abnormal behavior for one grouping relative to other groupings, based on statistically significant deviations of the test statistic for the one grouping relative to a normative baseline of similarity amongst the other groupings.

13. The computer program product of claim 12 wherein an output produced by the operators is a function of a set of inputs to the operators, and the test statistic is calculated based on less than all of the inputs.

14. The computer program product of claim 12 wherein an output produced by the operators is a function of a set of inputs to the operators, and the test statistic is based on whether operators produce similar outputs given similar inputs.

15. The computer program product of claim 12 wherein separating operators into groupings comprises:

iteratively assigning operators to randomly defined groupings; and

computing the test statistic for the randomly defined groups for each iteration until abnormal behavior is identified.

16. The computer program product of claim 12 wherein separating operators into groupings comprises repeatedly assigning operators into different groupings under different grouping plans.

17. The computer program product of claim 12 wherein the method further comprises:

for a grouping that has a statistically significant deviation of the test statistic relative to other groupings, observing behaviors of the operators in that grouping using a test data set.

18. The computer program product of claim 12 wherein an output produced by the operators is a binary output.

19. The computer program product of claim 12 wherein the observed behaviors are divided into a training set and a testing set;

the training set is used to identify candidate abnormal behaviors and the testing set is used to test whether the candidate abnormal behaviors are actually abnormal.

20. The computer program product of claim 12 wherein the test statistic comprises an estimate of a variance in the means of the operators output.

21. The computer program product of claim 12 the method further comprises using a search method to refine the groupings.