Method and system for pre-processing data using the mahalanobis distance (MD)

• US 8,073,652 B2
• Filed: 07/03/2008
• Issued: 12/06/2011
• Est. Priority Date: 07/03/2008
• Status: Active Grant

• Current Assignee: Caterpillar Incorporated
• Original Assignee: Caterpillar Incorporated
• Inventors: Grichnik, Anthony J., Felty, Timothy J., Mason, James R.
• Primary Examiner(s): NGHIEM, MICHAEL P
• Application Number:
  US12/216,406
• Publication Number:
  US 20100004898A1
• Time in Patent Office: 1,251 Days
• Field of Search: 702/179, 702/185, 702/189
• US Class Current: 702/179
• CPC Class Codes: G06F 18/10 Pre-processing; Data cleansing

Abstract

A computer-implemented method for pre-processing data. The method may include detecting one or more erroneous vectors in a plurality of vectors, detecting one or more erroneous elements in the one or more erroneous vectors, and deleting the detected one or more erroneous elements. The method may also include detecting one or more missing elements in the plurality of vectors. Further, the method may include populating one or more offending vectors that include one or more missing elements and/or deleted erroneous elements with one or more elements that are based on a distance metric.

Claims

1. A computer-implemented method for pre-processing data, comprising:

detecting one or more erroneous vectors in a plurality of vectors;

detecting one or more erroneous elements in the one or more erroneous vectors;

deleting values of the one or more detected erroneous elements from the one or more erroneous vectors;

identifying one or more surrounding vectors surrounding the one or more erroneous vectors with respect to time;

determining a surrounding vector distance metric based on one or more elements of the one or more surrounding vectors corresponding to the one or more erroneous elements of the one or more erroneous vectors;

calculating one or more new values for the one or more erroneous elements such that an erroneous vector distance metric, determined based on at least one or more remaining values of the one or more erroneous vectors, corresponds to the surrounding vector distance metric; and

replacing the deleted values of the one or more erroneous elements with the calculated one or more new values for the one or more erroneous elements of the one or more erroneous vectors.

2. The computer-implemented method according to claim 1, wherein the distance metric is a Mahalanobis Distance.

3. The computer-implemented method according to claim 2, wherein detecting one or more erroneous vectors in the plurality of vectors includes calculating a Mahalanobis Distance for each of the plurality of vectors to detect vectors with a Mahalanobis Distance greater by a predetermined threshold than that of the one or more surrounding vectors in time.

4. The computer-implemented method according to claim 2, wherein detecting one or more erroneous elements in the one or more erroneous vectors includes calculating a Mahalanobis Distance value for each subset of elements of the one or more erroneous vectors, and comparing the calculated Mahalanobis Distance value for each subset of elements of the one or more erroneous vectors with Mahalanobis Distance values calculated for a plurality of correspondingly-located subsets of elements of the one or more surrounding vectors in time.

5. The computer-implemented method according to claim 4, wherein detecting one or more erroneous elements in the one or more erroneous vectors further includes determining whether a Mahalanobis Distance value of each subset of elements of the one or more erroneous vectors is comparable to a Mahalanobis Distance value of the correspondingly-located subsets of elements of the one or more surrounding vectors.

6. The computer-implemented method according to claim 2, wherein calculating one or more new values for the one or more erroneous elements of the one or more erroneous vectors includes using a covariance matrix for the one or more erroneous vectors.

7. The computer-implemented method according to claim 2, wherein calculating one or more new values for the one or more erroneous elements includes calculating a Mahalanobis Distance of the one or more erroneous vectors using a populated element subset of the one or more erroneous vectors and comparing the calculated Mahalanobis Distance of the populated element subset of the one or more erroneous vectors to one or more of the surrounding vectors with elements that have a comparable Mahalanobis Distance value.

8. The computer-implemented method according to claim 7, wherein calculating one or more new values for the one or more erroneous elements further includes:

calculating the Mahalanobis Distance of the one or more surrounding vectors which have elements with a comparable Mahalanobis Distance value to the populated element subset of the one or more erroneous vectors; and

using the calculated Mahalanobis Distance for the one or more surrounding vectors to calculate the new values for the one or more erroneous elements of the one or more erroneous vectors that would give a Mahalanobis Distance value for the one or more erroneous vectors comparable to the Mahalanobis Distance of the one or more surrounding vectors.

9. A non-transitory computer-readable medium for use on a computer system configured to perform pre-processing of data, the non-transitory computer-readable medium having computer-executable instructions for performing a method comprising:

detecting one or more erroneous vectors in a plurality of vectors;

detecting one or more erroneous elements in the one or more erroneous vectors;

deleting values of the one or more detected erroneous elements from the one or more erroneous vectors;

identifying one or more surrounding vectors surrounding the one or more erroneous vectors with respect to time;

determining a surrounding vector distance metric based on one or more elements of the one or more surrounding vectors corresponding to the one or more erroneous elements of the one or more erroneous vectors;

calculating one or more new values for the one or more erroneous elements such that an erroneous vector distance metric, determined based on at least one or more remaining values of the one or more erroneous vectors, corresponds to the surrounding vector distance metric; and

replacing the deleted values of the one or more erroneous elements with the calculated one or more new values for the one or more erroneous elements of the one or more erroneous vectors.

10. The non-transitory computer-readable medium according to claim 9, wherein the distance metric is a Mahalanobis Distance.

11. The non-transitory computer-readable medium according to claim 10, wherein detecting one or more erroneous vectors in the plurality of vectors includes calculating a Mahalanobis Distance for each of the plurality of vectors to detect vectors with a Mahalanobis Distance greater by a predetermined threshold than that of the one or more surrounding vectors in time.

12. The non-transitory computer-readable medium according to claim 10, wherein detecting one or more erroneous elements in the one or more erroneous vectors includes calculating a Mahalanobis Distance value for each subset of elements of the one or more erroneous vectors, and comparing the calculated Mahalanobis Distance value for each subset of elements of the one or more erroneous vectors with Mahalanobis Distance values calculated for a plurality of correspondingly-located subsets of elements of the one or more surrounding vectors in time.

13. The non-transitory computer-readable medium according to claim 12, wherein detecting one or more erroneous elements in the one or more erroneous vectors further includes determining whether a Mahalanobis Distance value of each subset of elements of the one or more erroneous vectors is comparable to a Mahalanobis Distance value of the correspondingly-located subsets of elements of the one or more surrounding vectors.

14. The non-transitory computer-readable medium according to claim 10, wherein calculating one or more new values for the one or more erroneous elements of the one or more erroneous vectors includes using a covariance matrix for the one or more erroneous vectors.

15. The non-transitory computer-readable medium according to claim 10, wherein calculating one or more new values for the one or more erroneous elements includes calculating a Mahalanobis Distance of the one or more erroneous vectors using a populated element subset of the one or more erroneous vectors and comparing the calculated Mahalanobis Distance of the populated element subset of the one or more erroneous vectors to one or more of the surrounding vectors with elements that have a comparable Mahalanobis Distance value.

16. The non-transitory computer-readable medium according to claim 15, wherein calculating one or more new values for the one or more erroneous elements further includes:

calculating the Mahalanobis Distance of the one or more surrounding vectors which have elements with a comparable Mahalanobis Distance value to the populated element subset of the one or more erroneous vectors; and

using the calculated Mahalanobis Distance for the one or more surrounding vectors to calculate the new values for the one or more erroneous elements of the one or more erroneous vectors that would give a Mahalanobis Distance value for the one or more erroneous vectors comparable to the Mahalanobis Distance of the one or more surrounding vectors.

17. A computer system, comprising:

at least one input/output interface;

a database configured to store information relevant to a virtual sensor process model; and

a processor configured to;

obtain a set of data records corresponding to a plurality of vectors;

detect one or more erroneous vectors in the plurality of vectors;

detect one or more erroneous elements in the one or more erroneous vectors;

delete values of the detected one or more erroneous elements from the one or more erroneous vectors;

identify one or more surrounding vectors surrounding the one or more erroneous vectors with respect to time;

determine a surrounding vector distance metric based on one or more elements of the one or more surrounding vectors corresponding to the one or more erroneous elements of the one or more erroneous vectors;

calculate one or more new values for the one or more erroneous elements such that an erroneous vector distance metric, determined based on at least one or more remaining values of the one or more erroneous vectors, corresponds to the surrounding vector distance metric;

replace the deleted values of the one or more erroneous elements with the calculated one or more new values for the one or more erroneous elements of the one or more erroneous vectors.

18. The computer system according to claim 17, wherein, to detect one or more erroneous vectors in a plurality of vectors, the processor is configured to calculate a Mahalanobis Distance for each of the plurality of vectors to detect vectors with a Mahalanobis Distance greater by a predetermined threshold than that of surrounding vectors in time.

19. The computer system according to claim 17, wherein, to calculate one or more new values for the one or more erroneous elements, the processor is configured to use a covariance matrix for the one or more erroneous vectors.

20. The computer system according to claim 17, wherein, to calculate one or more new values for the one or more erroneous elements, the processor is configured to calculate a Mahalanobis Distance of the one or more erroneous vectors using a populated element subset of the one or more erroneous vectors and compare the calculated Mahalanobis Distance of the populated element subset of the one or more erroneous vectors to one or more of the surrounding vectors with elements which have a comparable Mahalanobis Distance value.

21. The computer system according to claim 17, wherein, to calculate one or more new values for the one or more erroneous vectors, the processor is further configured to:

calculate the Mahalanobis Distance of the one or more surrounding vectors which have elements with a comparable Mahalanobis Distance value to the populated element subset of the one or more erroneous vectors; and

use the calculated Mahalanobis Distance for the one or more surrounding vectors to calculate the one or more new values for the elements of the one or more erroneous vectors that would give a Mahalanobis Distance value for the one or more erroneous vectors comparable to the calculated Mahalanobis Distance of the one or more surrounding vectors.