EVALUATING SYSTEM PERFORMANCE WITH SPARSE PRINCIPAL COMPONENT ANALYSIS AND A TEST STATISTIC

• US 20170115238A1
• Filed: 10/22/2015
• Published: 04/27/2017
• Est. Priority Date: 10/22/2015
• Status: Active Grant

• Current Assignee: Battelle Memorial Institute
• Original Assignee: Battelle Memorial Institute
• Inventors: Regensburger, Joe, Mooney, Douglas, Aume, Laura, Rahimi, Amir, Cochran, Joe
• Granted Patent: US 10,656,102 B2
• Time in Patent Office: Days
• CPC Class Codes: G01N 2223/302 comparative arrangements
• CPC Class Codes: G01N 2223/419 computed tomograph
• CPC Class Codes: G01N 23/046 using tomography, e.g. computed tomography [CT]
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 18/2135 based on approximation criteria, e.g. principal component analysis
• CPC Class Codes: G06F 18/2136 based on sparsity criteria, e.g. with an overcomplete basis
• CPC Class Codes: G06F 18/2193 based on specific statistical tests
• CPC Class Codes: G06V 2201/05 Recognition of patterns representing particular kinds of hidden objects, e.g. weapons, explosives, drugs

Abstract

A method for evaluating system performance can include comparing a test average of instances of variables of test system variables to a baseline average of a baseline variables. Each of the instances of the variable of the test system variables may be shifted by a shift amount for a subset of the variables. A modified test data set may be generated from the shifted test data set. The modified test data set may be transformed with a sparse principal component analysis into test components. The test components may be compared to baseline components using a Hotelling T² as a test statistic. Performance of the system may be quantified based upon the test statistic.

Claims

1. A method for evaluating system performance, the method comprising:

collecting test measurements of a calibration standard with a sensor of a system;

transforming, automatically with one or more processors, the test measurements into a test data set, wherein the test data set comprises instances of test system variables, and wherein each of the instances of the test system variables corresponds to the test measurements;

comparing, automatically with the one or more processors, a test average of the instances of a variable of the test system variables to a baseline average of a baseline variable;

determining a shift amount based upon the test average and the baseline average;

shifting each of the instances of the variable of the test system variables by the shift amount to generate a modified test data set from the test data set;

transforming, automatically with the one or more processors, the modified test data set with a sparse principal component analysis into test components;

comparing, automatically with the one or more processors, the test components to baseline components using a Hotelling T² test to generate a test statistic; and

quantifying performance of the system based upon the test statistic.

2. The method of claim 1, comprising selecting the variable of the test system variables, when the variable is indicative of improved quality.

3. The method of claim 2, comprising:

comparing the variable of the test system variables to an improved quality metric; and

determining that the variable is indicative of the improved quality based upon the improved quality metric.

4. The method of claim 1, comprising selecting the variable of the test system variables, when the test average differs from the baseline average by less than a practically significant difference.

5. The method of claim 4, wherein the practically significant difference is a pre-defined percentage of the baseline average.

6. The method of claim 1, wherein the shift amount is a delta of the test average and the baseline average.

7. The method of claim 1, wherein a modified test average of the variable of the modified test system variables is the same as the baseline average.

8. The method of claim 1, wherein the performance of the system is quantified by comparing the test statistic to a statistical distribution.

9. The method of claim 8, wherein the statistical distribution is an empirical distribution derived from baseline measurements of baseline systems having a known performance level.

10. The method of claim 9, wherein:

the test statistic is compared to a threshold of the empirical distribution, and

the threshold is computed from a quantile of the empirical distribution.

11. A method for evaluating system performance, the method comprising:

providing a test data set, wherein;

the test data set comprises instances of test system variables, and

each of the instances of the test system variables corresponds to test measurements;

selecting, automatically with one or more processors, a variable of the test system variables, when the variable of the test system variables is indicative of improved quality, or when a test average of the instances of the variable of the test system variables differs from a baseline average by less than a practically significant difference;

determining a shift amount based upon the test average and the baseline average;

shifting, automatically with the one or more processors, each of the instances of the variable of the test system variables by the shift amount to generate a modified test data set from the test data set;

transforming, automatically with the one or more processors, the modified test data set with a sparse principal component analysis into test components;

comparing, automatically with the one or more processors, the test components to baseline components using a Hotelling T² test to generate a test statistic; and

quantifying performance of the system based upon the test statistic.

12. The method of claim 11, wherein the practically significant difference is a pre-defined percentage of the baseline average.

13. The method of claim 11, wherein the shift amount is a delta of the test average and the baseline average.

14. The method of claim 11, wherein a modified test average of the variable of the modified test system variables is the same as the baseline average.

15. The method of claim 11, wherein:

the performance of the system is quantified by comparing the test statistic to a statistical distribution, and

the statistical distribution is an empirical distribution derived from baseline measurements of baseline systems having a known performance level.

16. The method of claim 15, wherein:

the test statistic is compared to a threshold of the empirical distribution, and

the threshold is computed from a quantile of the empirical distribution.

17. A system capable of evaluating system performance, the system comprising:

a sensor communicatively coupled to one or more processors;

memory communicatively coupled to the one or more processors, wherein the memory comprises machine readable instructions that are executed by the one or more processors to;

collect test measurements of a calibration standard with the sensor;

transform the test measurements into a test data set, wherein;

the test data set comprises instances of test system variables, and

each of the instances of the test system variables correspond the test measurements;

compare a test average of the instances of a variable of the test system variables to a baseline average of a baseline variable;

determine a shift amount based upon the test average and the baseline average;

shift each of the instances of the variable of the test system variables by the shift amount to generate a modified test data set from the test data set;

transform the modified test data set with a sparse principal component analysis into test components;

compare the test components to baseline components using a Hotelling T-squared test to generate a test statistic; and

quantify performance of the system based upon the test statistic.

18. The system of claim 17, comprising an X-ray source communicatively coupled to one or more processors and configured to emit photons, wherein the sensor is an X-ray detector configured to detect the photons.

19. The system of claim 18, comprising an actuation assembly communicatively coupled to the one or more processors, and configured to manipulate the calibration standard, the sensor, the X-ray source, or a combination thereof.

20. The system of claim 17, wherein the machine readable instructions that are executed by the one or more processors to select the variable of the test system variables, when the variable of the test system variables is indicative of improved quality, or when the test average differs from the baseline average by less than a practically significant difference.