Method and system for pre-processing data using the Mahalanobis Distance (MD)

• US 20100004898A1
• Filed: 07/03/2008
• Published: 01/07/2010
• Est. Priority Date: 07/03/2008
• Status: Active Grant

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: Felty, Timothy J., Mason, James R., Grichnik, Anthony J.
• Granted Patent: US 8,073,652 B2
• Time in Patent Office: Days
• US Class Current: 702/179
• CPC Class Codes: G06F 18/10 Pre-processing; Data cleansing

Abstract

A computer-implemented method for pre-processing data. The method may include detecting one or more erroneous vectors in a plurality of vectors, detecting one or more erroneous elements in the one or more erroneous vectors, and deleting the detected one or more erroneous elements. The method may also include detecting one or more missing elements in the plurality of vectors. Further, the method may include populating one or more offending vectors that include one or more missing elements and/or deleted erroneous elements with one or more elements that are based on a distance metric.

Claims

1. A computer-implemented method for pre-processing data, comprising:

detecting one or more erroneous vectors in a plurality of vectors;

detecting one or more erroneous elements in the one or more erroneous vectors;

deleting the one or more detected erroneous elements;

detecting one or more missing elements in the plurality of vectors; and

populating one or more offending vectors that include one or more missing elements and/or deleted erroneous elements with one or more elements that are based on a distance metric.

2. The computer-implemented method according to claim 1, wherein the distance metric is a Mahalanobis Distance.

3. The computer-implemented method according to claim 2, wherein detecting one or more erroneous vectors in a plurality of vectors includes calculating a Mahalanobis Distance for each vector to detect vectors with a Mahalanobis Distance greater by a predetermined threshold than that of surrounding vectors in time.

4. The computer-implemented method according to claim 2, wherein detecting one or more erroneous elements in the one or more erroneous vectors includes calculating the Mahalanobis Distance for each subset of the erroneous vector, and comparing the subset Mahalanobis Distance values of the erroneous vector with Mahalanobis Distance values calculated for a plurality of the correspondingly-located subsets of surrounding vectors in time.

5. The computer-implemented method according to claim 4, further including determining that the Mahalanobis Distance value of one of the subsets of the erroneous vector is comparable to the Mahalanobis Distance value of one of the subsets of the surrounding vectors, such determination indicating the one or more elements not used in the subset of the erroneous vector are erroneous elements.

6. The computer-implemented method according to claim 2, wherein populating one or more offending vectors includes calculating values for the missing and/or erroneous elements of the offending vector by using a covariance matrix for the offending vector.

7. The computer-implemented method according to claim 2, wherein populating one or more offending vectors includes calculating the Mahalanobis Distance of the offending vector using a populated element subset of the offending vector and comparing the Mahalanobis Distance of the populated element subset of the offending vector to other vectors with elements that have a comparable Mahalanobis Distance value.

8. The computer-implemented method according to claim 7, wherein populating one or more offending vectors further includes populating offending vectors by calculating the Mahalanobis Distance of other vectors which had elements with a comparable Mahalanobis Distance value to the populated element subset in the offending vector, and using that calculated Mahalanobis Distance for the other vectors to determine the missing and/or deleted elements for the offending vector by calculating values for the missing and/or deleted elements that would give a Mahalanobis Distance value for the offending vector comparable to the Mahalanobis Distance of the surrounding vectors.

9. A computer-readable medium for use on a computer system configured to perform pre-processing of data, the computer-readable medium having computer-executable instructions for performing a method comprising:

detecting one or more erroneous vectors in a plurality of vectors;

detecting one or more erroneous elements in the one or more erroneous vectors;

deleting the detected one or more erroneous elements;

detecting one or more missing elements in the plurality of vectors; and

populating one or more offending vectors that include one or more missing elements and/or deleted erroneous elements with one or more elements that are based on a distance metric.

10. The computer-readable medium according to claim 9, wherein the distance metric is a Mahalanobis Distance.

11. The computer-readable medium according to claim 10, wherein detecting one or more erroneous vectors in a plurality of vectors includes calculating a Mahalanobis Distance for each vector to detect vectors with a Mahalanobis Distance greater by a predetermined threshold than that of surrounding vectors in time.

12. The computer-readable medium according to claim 10, wherein detecting one or more erroneous elements in the one or more erroneous vectors includes calculating the Mahalanobis Distance for each subset of the erroneous vector, and comparing the subset Mahalanobis Distance values of the erroneous vector with Mahalanobis Distance values calculated for a plurality of the correspondingly-located subsets of surrounding vectors in time.

13. The computer-readable medium according to claim 12, further including determining that the Mahalanobis Distance value of one of the subsets of the erroneous vector is comparable to the Mahalanobis Distance value of one of the subsets of the surrounding vectors, such determination indicating the one or more elements not used in the subset of the erroneous vector are erroneous elements.

14. The computer-readable medium according to claim 10, wherein populating one or more offending vectors includes calculating values for the missing and/or erroneous elements of the offending vector by using a covariance matrix for the offending vector.

15. The computer-readable medium according to claim 10 wherein populating one or more offending vectors includes calculating the Mahalanobis Distance of the offending vector using a populated element subset of the offending vector and comparing the Mahalanobis Distance of the populated element subset of the offending vector to other vectors with elements that have a comparable Mahalanobis Distance value.

16. The computer-readable medium according to claim 15, wherein populating one or more offending vectors further includes populating offending vectors by calculating the Mahalanobis Distance of other vectors which had elements with a comparable Mahalanobis Distance value to the populated element subset in the offending vector, and using that calculated Mahalanobis Distance for the other vectors to determine the missing and/or deleted elements for the offending vector by calculating values for the missing and/or deleted elements that would give a Mahalanobis Distance value for the offending vector comparable to the Mahalanobis Distance of the surrounding vectors.

18. The computer system according to claim 15, wherein, to detect one or more erroneous vectors in a plurality of vectors, the processor is configured to calculate a Mahalanobis Distance for each vector to detect vectors with a Mahalanobis Distance greater by a predetermined threshold than that of surrounding vectors in time.

19. The computer system according to claim 15, wherein, to detect one or more erroneous elements in the one or more erroneous vectors, the processor is configured to calculate the Mahalanobis Distance for each subset of the erroneous vector, and compare the subset Mahalanobis Distance values of the erroneous vector with Mahalanobis Distance values calculated for a plurality of the correspondingly-located subsets of surrounding vectors in time and is further configured to determine that the Mahalanobis Distance value of one of the subsets of the erroneous vector is comparable to the Mahalanobis Distance value of one of the subsets of the surrounding vectors, such determination indicating the one or more elements not used in the subset of the erroneous vector are erroneous elements.

20. The computer system according to claim 15, wherein, to populate one or more offending vectors, the processor is configured to calculate values for the missing and/or erroneous elements of the offending vector by using a covariance matrix for the offending vector.

21. The computer system according to claim 15, wherein, to populate one or more offending vectors, the processor is configured to calculate the Mahalanobis Distance of offending vectors using a populated element subset of the offending vector and comparing the Mahalanobis Distance of the populated element subset of the offending vector to other vectors with elements which have a comparable Mahalanobis Distance value.

22. The computer system according to claim 19, wherein, to populate one or more offending vectors, the processor is further configured to populate offending vectors by calculating the Mahalanobis Distance of other vectors which had elements with a comparable Mahalanobis Distance value to the populated element subset in the offending vector, and using that calculated Mahalanobis Distance for the other vectors to determine the missing and/or deleted elements for the offending vector by calculating values for the missing and/or deleted elements that would give a Mahalanobis Distance value for the offending vector comparable to the Mahalanobis Distance of the surrounding vectors.

17. A computer system, comprising:

a console;

at least one input device;

a database configured to store information relevant to a virtual sensor process model; and

a processor configured to;

obtain a set of data records corresponding to a plurality of vectors;

detect one or more erroneous vectors in the plurality of vectors;

detect one or more erroneous elements in the one or more erroneous vectors;

delete the detected one or more erroneous elements;

detect one or more missing elements in the plurality of vectors; and

populate one or more offending vectors that include one or more missing elements and/or deleted erroneous elements with one or more elements that are based on a Mahalanobis Distance.