Methods and apparatus for data analysis

• US 8,000,928 B2
• Filed: 04/29/2008
• Issued: 08/16/2011
• Est. Priority Date: 05/24/2001
• Status: Expired due to Fees

• Current Assignee: In-Depth Test LLC (Acacia Research Corporation)
• Original Assignee: Test Advantage Incorporated (Boston Semi Equipment, LLC)
• Inventors: Scott, Michael J., Buxton, Paul, Tabor, Eric Paul, Gorin, Jacky
• Primary Examiner(s): KUNDU, SUJOY K
• Application Number:
  US12/111,773
• Publication Number:
  US 20080249742A1
• Time in Patent Office: 1,204 Days
• Field of Search: 702/84, 702117-124, 702/182, 702/183, 700/55, 714/721, 438/10, 438/14
• US Class Current: 702/179
• CPC Class Codes: G01R 31/31707 Test strategies methods for generation of test sequences G01R31/318371
• CPC Class Codes: G01R 31/318314 Tools, e.g. program interfaces, test suite, test bench, simulation hardware, test compiler, test program languages simulation software G01R31/318357; emulators G06F11/261
• CPC Class Codes: G05B 2219/1112 Bit addressing, handling
• CPC Class Codes: G05B 2223/02 Indirect monitoring, e.g. monitoring production to detect faults of a system
• CPC Class Codes: G05B 23/0229 knowledge based, e.g. expert systems; genetic algorithms
• CPC Class Codes: G05B 23/0294 Optimizing process, e.g. process efficiency, product quality
• CPC Class Codes: G06F 18/2433 Single-class perspective, e.g. one-against-all classification; Novelty detection; Outlier detection

Abstract

A method and apparatus for data analysis according to various aspects of the present invention is configured to automatically select one or more outlier identification algorithms for identifying statistical outliers in test data for components.

Claims

1. A test data analysis system for analyzing test data for a set of components, comprising:

a memory for storing the test data; and

an outlier identification system having access to the memory and configured to identify outliers in the test data according to a selected outlier identification algorithm, wherein the outlier identification system is configured to automatically select the selected outlier identification algorithm from a plurality of outlier identification algorithms.

2. A test data analysis system according to claim 1, wherein the outlier identification system is configured to select the selected outlier identification algorithm according to at least one descriptive statistic derived from the test data.

3. A test data analysis system according to claim 1, wherein the outlier identification system is configured to forego automatically identifying the outliers in the test data if the test data is a selected data population type.

4. A test data analysis system according to claim 1, wherein the outlier identification system comprises a pre-processing engine configured to identify the outliers in the test data using more of the plurality of outlier identification algorithms than the selected outlier identification algorithm.

5. A test data analysis system according to claim 1, wherein the outlier identification system is configured to select the selected outlier identification algorithm according to user-defined rules.

6. A test data analysis system according to claim 1, wherein the outlier identification system is configured to select the selected outlier identification algorithm according to a classification hierarchy based on characteristics of the test data.

7. A test data analysis system according to claim 1, wherein the outlier identification system is configured to automatically identify outliers in the test data according to multiple outlier identification algorithms and discard results generated by outlier identification algorithms other than the selected outlier identification algorithm.

8. A test data analysis system according to claim 1, wherein the outlier identification system comprises:

a pre-processing engine configured to analyze the test data according to the plurality of outlier identification algorithms to generate pre-processing data; and

a classification engine configured to select the selected outlier identification algorithm based on the pre-processing data.

9. A test data analysis system according to claim 1, wherein the test data for the set of components includes test data from multiple tests for the components, and wherein the outlier identification system is configured to automatically identify outliers in the test data for each test according to a test-specific selected outlier identification algorithm selected for the test data for each test, wherein the test-specific selected outlier identification algorithm for each test is automatically selected for each test from the plurality of outlier identification algorithms.

10. A test data analysis system according to claim 1, wherein the plurality of outlier identification algorithms is configured to he modified by a user.

11. A test data analysis system according to claim 1, wherein the test data includes raw test data and filtered test data, wherein the filtered test data comprises the raw data without test data corresponding to failures.

12. A test data analysis system according to claim 1, wherein the outlier identification system is configured to:

forego automatically identifying the outliers in the test data according to the selected outlier identification algorithm if the test data is generated by a user-specified test; and

identify outliers in the test data for the selected test using a user-specified outlier identification algorithm.

13. A test data analysis system according to claim 1, wherein:

the selected outlier identification compares the test data to an upper threshold and a lower threshold;

the upper threshold and the lower threshold are defined according to different distances from a mean between an upper test limit and a lower test limit.

14. A test data analysis system according to claim 13, wherein the outlier identification system is configured to forego dynamically adjusting the upper threshold and the lower threshold and assign selected values to the upper threshold and the lower threshold if a selected portion of the tests data is at least one of (a) within a selected range of the upper test limit or the lower test limit, and (b) beyond a selected range of the upper test limit or the lower test limit.

15. A test data analysis system according to claim 1, wherein the outlier identification system is configured to:

dynamically adjust a threshold according to the test data; and

compare the test data to the threshold to identify the outliers.

16. A computer-implemented method for testing components, comprising:

obtaining lest data for the components;

automatically selecting an outlier identification algorithm from a plurality of outlier identification algorithms by the computer; and

automatically identifying outliers in the test data according to the selected outlier identification algorithm by the computer.

17. A computer-implemented method according to claim 16 wherein selecting the outlier identification algorithm includes selecting the selected outlier identification algorithm according to at least one descriptive statistic derived from the test data.

18. A computer-implemented method according to claim 16, further comprising foregoing automatically identifying the outliers in the test data if the test data is a selected data population type.

19. A computer-implemented method according to claim 16, wherein selecting the outlier identification algorithm includes identifying the outliers in the test data using more of the plurality of outlier identification algorithms than the selected outlier identification algorithm.

21. A computer-implemented method according to claim 16, wherein selecting the outlier identification algorithm includes selecting the selected outlier identification algorithm according to a classification hierarchy based on characteristics of the test data.

22. A computer-implemented method according to claim 16, wherein selecting the outlier identification algorithm includes automatically identifying outliers in the test data according to multiple outlier identification algorithms and discarding results generated by outlier identification algorithms other than the selected outlier identification algorithm.

23. A computer-implemented method according to claim 16, wherein selecting the outlier identification algorithm includes:

analyzing the test data according to the plurality of outlier identification algorithms to generate pre-processing data; and

selecting the selected outlier identification algorithm based on the pre-processing data.

24. A computer-implemented method according to claim 16, wherein the test data for the set of components includes tests data from multiple tests for the components, and wherein selecting the outlier identification algorithm includes selecting a test-specific selected outlier identification algorithm for each test from the plurality of outlier identification algorithms.

25. A computer-implemented method according to claim 16, wherein the plurality of outlier identification algorithms is configured to be modified by a user.

26. A computer-implemented method according to claim 16, wherein the test data includes raw test data and filtered test data, wherein the filtered test data comprises the raw data without test data corresponding to failures.

27. A computer-implemented method according to claim 16, wherein automatically identifying outliers comprises:

foregoing automatically identifying the outliers in the test data according to the selected outlier identification algorithm if the test data is generated by a user-specified test; and

identifying outliers in the test data for the selected test using a user-specified outlier identification algorithm.

28. A computer-implemented method. according to claim 16, wherein automatically identifying outliers comprises:

comparing the test data to an upper threshold and a lower threshold., wherein the upper threshold and the lower threshold are defined according to different distances from a mean between an upper test limit and a lower test limit.

29. A computer-implemented method according to claim 28, wherein automatically identifying outliers further comprises foregoing dynamically adjusting the upper threshold and the lower threshold and assigning selected values to the upper threshold and the lower threshold if a selected portion of the tests data is at least one of (a) within a selected range of the upper test limit or the lower test limit and (b) beyond a selected range of the upper test limit or the lower test limit.

30. A computer-implemented method according to claim 16, wherein automatically identifying outliers comprises:

dynamically adjusting a threshold according to the test data; and

comparing the test data to the threshold to identify the outliers.

20. A computer-implemented method according to claim selecting the 16, wherein selecting the outlier identification algorithm includes selected outlier identification algorithm according to user-defined rules.

31. A non-transitory computer readable medium containing computer instructions stored therein for causing a computer processor to perform a method for analyzing test data comprising:

obtaining test data for the components;

automatically selecting an outlier identification algorithm from a plurality of outlier identification algorithms; and

automatically identifying an outlier in the test data using the selected outlier identification algorithm.

32. A non-transitory computer readable medium according to claim 31, wherein selecting the outlier identification algorithm includes selecting the selected outlier identification algorithm according to at least one descriptive statistic derived from the test data.

33. A non-transitory computer readable medium according to claim 31, further comprising foregoing automatically identifying the outliers in the test data if the test data is a selected data population type.

34. A non-transitory computer readable medium according to claim 31, wherein selecting the outlier identification algorithm includes identifying the outliers in the test data using more of the plurality of outlier identification algorithms than the selected outlier identification algorithm.

35. A non-transitory computer readable medium according to claim 31, wherein selecting the outlier identification algorithm includes selecting the selected outlier identification algorithm according to user-defined rules.

36. A non-transitory computer readable medium according to claim 31, wherein selecting the outlier identification algorithm includes selecting the selected outlier identification algorithm according to a classification hierarchy based on characteristics of the test data.

37. A non-transitory computer readable medium according to claim 31, wherein selecting the outlier identification algorithm includes automatically identifying outliers in the test data according to multiple outlier identification algorithms and discarding results generated by outlier identification algorithms other than the selected outlier identification algorithm.

38. A non-transitory computer readable medium according to claim 31, wherein selecting the outlier identification algorithm includes:

analyzing the test data according to the plurality of outlier identification algorithms to generate pre-processing data; and

selecting the selected outlier identification algorithm based on the pre-processing data.

39. A non-transitory computer readable medium according to claim 31, wherein the test data for the set of components includes tests data from multiple tests for the components, and wherein selecting the outlier identification algorithm includes selecting a test-specific selected outlier identification algorithm for each test from the plurality of outlier identification algorithms.

40. A non-transitory computer readable medium according to claim 31, wherein the plurality of outlier identification algorithms is configured to be modified by a user.

41. A non-transitory computer readable medium according to claim 31, wherein the test data includes raw test data and filtered test data, wherein the filtered test data comprises the raw data without test data corresponding to failures.

42. A non-transitory computer readable medium according to claim 31, wherein automatically identifying outliers comprises:

foregoing automatically identifying the outliers in the test data according to the selected outlier identification algorithm if the test data is generated by a user-specified test; and

identifying outliers in the test data for the selected test using a user-specified outlier identification algorithm.

43. A non-transitory computer readable medium according to claim 31, wherein automatically identifying outliers comprises:

comparing the test data to an upper threshold and a lower threshold, wherein the upper threshold and the lower threshold are defined according to different distances from a mean between an upper test limit and a lower test limit.

44. A non-transitory computer readable medium according to claim 43, wherein automatically identifying outliers further comprises foregoing dynamically adjusting the upper threshold and the lower threshold and assigning selected values to the upper threshold and the lower threshold if a selected portion of the tests data is at least one of (a) within a selected range of the upper test limit or the lower test limit and (b) beyond a selected range of the upper test limit or the lower test limit.

45. A non-transitory computer readable medium according to claim 35, wherein automatically identifying outliers comprises:

dynamically adjusting a threshold. according to the test data; and

comparing the test data to the threshold to identity the outliers.