SYSTEM AND METHOD FOR DATA VALIDATION

• US 20150261824A1
• Filed: 04/23/2014
• Published: 09/17/2015
• Est. Priority Date: 03/11/2014
• Status: Active Grant

• Current Assignee: Workday Incorporated
• Original Assignee: Wipro Limited
• Inventors: Jha, Dharmendra
• Granted Patent: US 9,424,290 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/219 Managing data history or versioning querying versioned data G06F16/2474; querying temporal data G06F16/2477
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24564 Applying rules; Deductive queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Systems and methods for validating data in a communication network environment include receiving a data transformation specification from a user. The data transformation specification is analyzed to determine data transformation rules which are indicative of a relationship between corresponding fields of the source repository and the target repository. Test cases and test scripts are generated based on the data transformation rules and the test cases and the test scripts are executed on the source repository and the target repository to validate the relationship between the corresponding fields of the source repository and the target repository. Thereafter, a log file, indicative of the outcome of the execution of the test cases and the test scripts, is generated.

Claims

1. A data validation system for data validation over a communication network of data transformed from a source repository to a target repository, the data validation system comprising:

a processor;

a memory communicatively coupled to the processor, wherein the memory stores processor-executable instructions, which, on execution, cause the processor to;

receive a data transformation specification from a user;

analyze the data transformation specification to determine data transformation rules, wherein the data transformation rules are indicative of a relationship between corresponding fields of the source repository and the target repository;

generate test cases and test scripts based on the data transformation rules;

execute the test cases and the test scripts on the source repository and the target repository to validate the relationship between the corresponding fields of the source repository and the target repository; and

generate a log file indicative of the outcome of the execution of the test cases and the test scripts.

2. The data validation system as claimed in claim 1, wherein the instructions, on execution, further cause the processor to:

determine the number of records on which the test cases and the test scripts are to be executed;

initiate parallel execution of the test cases and the test scripts on determining the number of records to have exceeded a critical threshold; and

initiate serial execution of the test cases and the test scripts on determining the number of records to be less than the critical threshold.

3. The data validation system as claimed in claim 1, wherein the instructions, on execution, further cause the processor to generate, by the data validation system, a report, based on the log file, wherein the report is indicative of whether values stored in the corresponding fields of the source repository and the target repository are in accordance with the data transformation rules.

4. The data validation system as claimed in claim 1, wherein the instructions, on execution, further cause the processor to:

determine a schema of the source repository and the target repository;

identify attributes of the corresponding fields of the source repository and the target repository; and

determine the data transformation rules based on the identified attributes.

5. The data validation system as claimed in claim 1, wherein the instructions, on execution, further cause the processor to ascertain at least one of inconsistent standards between corresponding fields of the source repository and the target repository, discrepancies in data format between the corresponding fields of the source repository and the target repository, difference in structure of data between the corresponding fields of the source repository and the target repository, missing values between the corresponding fields of the source repository and the target repository, data fields filled with default values or nulls in the corresponding fields of the source repository and the target repository, or data anomalies.

6. The data validation system as claimed in claim 1, wherein the instructions, on execution, further cause the processor to:

receive an identifier from a user, wherein the identifier is indicative of an application whose associated data repositories are to be validated;

identify the source repository and the target repository based on the identifier;

receive configuration parameters of the source repository and the target repository, wherein the configuration parameters are indicative of at least one of data formats, structure, and schema of the source repository and the target repository;

receive connection parameters of the source repository and the target repository, wherein the connection parameters are indicative of the authentication details associated with accessing source repository and the target repository;

establish connection with the source repository and the target repository, based on the connection parameters;

identify the corresponding tables and fields of the source repository and the target repository; and

prompt the user to confirm the identified corresponding tables and fields of the source repository and the target repository.

7. A method for validating data transformed from a source repository to a target repository, the method comprising:

receiving, by a data validation apparatus, a data transformation specification from a user;

analyzing, by the data validation apparatus, the data transformation specification to determine data transformation rules, wherein the data transformation rules are indicative of a relationship between corresponding fields of the source repository and the target repository;

generating, by the data validation apparatus, test cases and test scripts based on the data transformation rules;

executing, by the data validation apparatus, the test cases and the test scripts on the source repository and the target repository to validate the relationship between the corresponding fields of the source repository and the target repository; and

generating, by the data validation apparatus, a log file indicative of the outcome of the execution of the test cases and the test scripts.

8. The method as claimed in claim 7, wherein the method further comprises:

determining, by the data validation apparatus, the number of records on which the test cases and the test scripts are to be executed;

initiating, by the data validation apparatus, parallel execution of the test cases and the test scripts on determining the number of records to have exceeded a critical threshold; and

initiating, by the data validation apparatus, serial execution of the test cases and the test scripts on determining the number of records to be less than the critical threshold.

9. The method as claimed in claim 7, wherein the method further comprises generating, by the data validation apparatus, a report, based on the log file, wherein the report is indicative of whether values stored in the corresponding fields of the source repository and the target repository are in accordance with the data transformation rules.

10. The method as claimed in claim 7, wherein the analyzing the data transformation specification further comprises:

determining, by the data validation apparatus, a schema of the source repository and the target repository;

identifying, by the data validation apparatus, attributes of the corresponding fields of the source repository and the target repository; and

determining, by the data validation apparatus, the data transformation rules based on the identified attributes.

11. The method as claimed in claim 7, wherein the execution of the test cases and the test scripts further comprise ascertaining, by the data validation apparatus, at least one of inconsistent standards between corresponding fields of the source repository and the target repository, discrepancies in data format between the corresponding fields of the source repository and the target repository, difference in structure of data between the corresponding fields of the source repository and the target repository, missing values between the corresponding fields of the source repository and the target repository, data fields filled with default values or nulls in the corresponding fields of the source repository and the target repository, or data anomalies.

12. The method as claimed in claim 7, wherein the method further comprises:

receiving, by the data validation apparatus, an identifier from a user, wherein the identifier is indicative of an application whose associated data repositories are to be validated;

identifying, by the data validation apparatus, the source repository and the target repository based on the identifier;

receiving from the user, by the data validation apparatus, configuration parameters of the source repository and the target repository, wherein the configuration parameters are indicative of at least one of data formats, structure, and schema of the source repository and the target repository;

receiving from the user, by the data validation apparatus, connection parameters of the source repository and the target repository, wherein the connection parameters are indicative of the authentication details associated with accessing source repository and the target repository;

establishing, by the data validation apparatus, connection with the source repository and the target repository, based on the connection parameters;

identifying, by the data validation apparatus, the corresponding tables and fields of the source repository and the target repository; and

prompting, by the data validation apparatus, the user to confirm the identified corresponding tables and fields of the source repository and the target repository.

13. A non-transitory computer readable medium including instructions stored thereon that when processed by at least one processor cause the processor to perform acts comprising:

receiving a data transformation specification from a user;

analyzing the data transformation specification to determine data transformation rules, wherein the data transformation rules are indicative of a relationship between corresponding fields of the source repository and the target repository;

generating test cases and test scripts based on the data transformation rules;

executing the test cases and the test scripts on the source repository and the target repository to validate the relationship between the corresponding fields of the source repository and the target repository; and

generating a log file indicative of the outcome of the execution of the test cases and the test scripts.

14. The non-transitory computer readable medium as claimed in claim 13, which include further instructions when processed by at least one processor cause the processor to perform acts comprising:

determining the number of records on which the test cases and the test scripts are to be executed;

initiating parallel execution of the test cases and the test scripts on determining the number of records to have exceeded a critical threshold; and

initiating serial execution of the test cases and the test scripts on determining the number of records to be less than the critical threshold.