DATA VALIDATION USING SIGNATURES AND SAMPLING

• US 20090132461A1
• Filed: 01/23/2009
• Published: 05/21/2009
• Est. Priority Date: 05/28/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Rajasekaran, Kulothungan, Garg, Neeraj, Carter, Cale D., Martin, Deuane J., Singh, Pankaj K.
• Granted Patent: US 7,966,279 B2
• Time in Patent Office: Days
• US Class Current: 706/54
• CPC Class Codes: H04L 2209/30 Compression, e.g. Merkle-Damgard construction
• CPC Class Codes: H04L 2209/56 Financial cryptography, e.g. electronic payment or e-cash
• CPC Class Codes: H04L 2209/80 Wireless
• CPC Class Codes: H04L 9/3247 involving digital signatures

Abstract

Architecture that facilitates validation of a data mapping of data from a data source to a data target. There is included a signature generation component that generates a source signature of all or a portion of the data source and a target signature of all or a corresponding portion of the data target, and a sampling component that obtains a sample of the source data a corresponding sample of the target data. The data signatures and data samples are compared respectively and processed with a processing component to determine the status of the validation process.

Claims

1. A system that facilitates validation of a mapping of data between a raw data source and a raw data target, comprising:

a classifier component that determines at least one of a type of data or amount of data of the raw data source to utilize for validation;

a signature generation component that generates respective data signatures of the raw data source and the raw data target;

a sampling component that obtains respective data samples of the raw data source and the raw data target; and

a processing component that compares at least a portion of the sample of raw source data against corresponding portion of raw target data and logically combines a result of the comparison with another comparison result.

2. The system of claim 1, the signature generation component utilizes all of the data of the data source to generate the respective data signature.

3. The system of claim 1, the classifier comprises at least one of a support vector machine, a Bayesian network, a decision tree, statistical regression, or a probabilistic classification model.

4. The system of claim 1, the classifier determines a weighting of at least one of all of the data source or a portion of the data source to determine which data source of a plurality of data sources should be used for validation or which portions of the selected data source will be used for validation.

5. The system of claim 1, the classifier determines one of sampling rate or amount of compression that the signature component uses to generate the data signature of the data source.

6. The system of claim 1, the classifier is used to determine one of the type of records that the sampling component uses to obtain the data sample of the data source.

7. The system of claim 1, the another comparison result is a result of a comparison between the data source signature and the data target signature.

8. The system of claim 7, the processing component processes the raw data samples and data signatures to determine status of a validation process.

9. A system that facilitates validation of a mapping of data between a data source and a data target, comprising:

means for obtaining at least one data sample of the data source and at least one data sample of the data target; and

means for processing the at least one data sample against at least one data signature to determine status of a validation process through comparison of the data samples against a comparison of related data signatures.

10. The system of claim 9, further comprising means for determining at least one of a type of the data or amount of the data of the data source to utilize for validation.

11. The system of claim 10, the means for determining comprises at least one of a support vector machine, a Bayesian network, a decision tree, statistical regression, or a probabilistic classification model.

12. The system of claim 10, the means for determining determines a weighting of at least a portion of the data source to determine which data source of a plurality of data sources should be used for validation or which portions of the selected data source will be used for validation.

13. The system of claim 10, further comprising a means for generating the respective data signatures of the data source and the data target, the means for determining determines one of the sampling rate or amount of compression that the signature component uses to generate the data signature of the data source.

14. The system of claim 10, the means for determining determines one of a type of record that the means for obtaining uses to obtain the data sample of the data source.

15. The system of claim 10, the means for processing compares at least one source data sample against at least one target data sample that corresponds to the source data sample and logically combines the compared samples result of the comparison.

16. The system of claim 15, the source data sample is a raw source data sample and the target data sample is a raw target data sample.

17. The system of claim 10, further comprising a means for generating at least one data signature of the data source and the data target, at least a portion of the generated data signatures are used to determine status of a validation process.

18. The system of claim 17, the means for generating utilizes all of the data of the data source to generate the respective data signature.

19. A method that is at least partially executed upon a processor for facilitating validation of a mapping of data between a raw data source and a raw data target, comprising:

determining at least one of a type of data or amount of the data of the raw data source to utilize for validation;

generating respective data signatures of the raw data source and the raw data target;

obtaining respective data samples of the raw data source and the raw data target;

processing the data samples and data signatures to determine status of a validation process through comparison of the sample the raw source data against corresponding raw target data as well as comparison of the source data signature against the target data signature; and

transforming the signature compare result and the sample compare result into a construct in a logical manner.

20. The method of claim 19, further comprising determining one of sampling rate or amount of compression that is used to generate the data signature of the data source.