System and method for data validation

• US 9,424,290 B2
• Filed: 04/23/2014
• Issued: 08/23/2016
• Est. Priority Date: 03/11/2014
• Status: Active Grant

• Current Assignee: Workday Incorporated
• Original Assignee: Wipro Limited
• Inventors: Jha, Dharmendra
• Primary Examiner(s): Pham, Hung Q
• Application Number:
  US14/260,033
• Publication Number:
  US 20150261824A1
• Time in Patent Office: 853 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 16/1734 Details of monitoring file system events, e.g. by the use of hooks, filter drivers, logs
• CPC Class Codes: G06F 16/219 Managing data history or versioning querying versioned data G06F16/2474; querying temporal data G06F16/2477
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/24564 Applying rules; Deductive queries
• CPC Class Codes: G06F 16/254 Extract, transform and load [ETL] procedures, e.g. ETL data flows in data warehouses
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques

Abstract

Systems and methods for validating data in a communication network environment include receiving a data transformation specification from a user. The data transformation specification is analyzed to determine data transformation rules which are indicative of a relationship between corresponding fields of the source repository and the target repository. Test cases and test scripts are generated based on the data transformation rules and the test cases and the test scripts are executed on the source repository and the target repository to validate the relationship between the corresponding fields of the source repository and the target repository. Thereafter, a log file, indicative of the outcome of the execution of the test cases and the test scripts, is generated.

Claims

1. A data validation system for data validation over a communication network of data transformed from a source repository to a target repository, the data validation system comprising:

a processor;

a memory communicatively coupled to the processor, wherein the memory stores processor-executable instructions, which, on execution, cause the processor to;

receive an identifier from a user, wherein the identifier is indicative of an application whose associated data repositories are to be validated;

identify the source repository and the target repository based on the identifier;

receive configuration parameters of the source repository and the target repository, wherein the configuration parameters are indicative of at least one of data formats, structure, and schema of the source repository and the target repository;

receive connection parameters of the source repository and the target repository, wherein the connection parameters are indicative of authentication details associated with accessing the source repository and the target repository;

establish connection with the source repository and the target repository, based on the connection parameters;

identify corresponding tables and fields of the source repository and the target repository;

prompt the user to confirm the identified corresponding tables and fields of the source repository and the target repository;

receive a data transformation specification from the user;

analyze the data transformation specification to determine data transformation rules, wherein the data transformation rules are indicative of a relationship between the corresponding fields of the source repository and the target repository;

generate test cases and test scripts based on the data transformation rules;

execute the test cases and the test scripts on the source repository and the target repository to validate the relationship between the corresponding fields of the source repository and the target repository; and

generate a log file indicative of the outcome of the execution of the test cases and the test scripts.

2. The data validation system as claimed in claim 1, wherein the instructions, on execution, further cause the processor to:

determine a number of records on which the test cases and the test scripts are to be executed;

initiate parallel execution of the test cases and the test scripts on determining the number of records to have exceeded a critical threshold; and

initiate serial execution of the test cases and the test scripts on determining the number of records to be less than the critical threshold.

3. The data validation system as claimed in claim 1, wherein the instructions, on execution, further cause the processor to generate, by the data validation system, a report, based on the log file, wherein the report is indicative of whether values stored in the corresponding fields of the source repository and the target repository are in accordance with the data transformation rules.

4. The data validation system as claimed in claim 1, wherein the instructions, on execution, further cause the processor to:

determine a schema of the source repository and the target repository;

identify attributes of the corresponding fields of the source repository and the target repository; and

determine the data transformation rules based on the identified attributes.

5. The data validation system as claimed in claim 1, wherein the instructions, on execution, further cause the processor to ascertain at least one of inconsistent standards between corresponding fields of the source repository and the target repository, discrepancies in data format between the corresponding fields of the source repository and the target repository, difference in structure of data between the corresponding fields of the source repository and the target repository, missing values between the corresponding fields of the source repository and the target repository, data fields filled with default values or nulls in the corresponding fields of the source repository and the target repository, or data anomalies.

6. A method for data validation over a communication network of data transformed from a source repository to a target repository, the method comprising:

receiving, by a data validation computing device, an identifier from a user, wherein the identifier is indicative of an application whose associated data repositories are to be validated;

identifying, by the data validation computing device, the source repository and the target repository based on the identifier;

receiving, by the data validation computing device, configuration parameters of the source repository and the target repository, wherein the configuration parameters are indicative of at least one of data formats, structure, and schema of the source repository and the target repository;

receiving, by the data validation computing device, connection parameters of the source repository and the target repository, wherein the connection parameters are indicative of authentication details associated with accessing the source repository and the target repository;

establishing, by the data validation computing device, connection with the source repository and the target repository, based on the connection parameters;

identifying, by the data validation computing device, corresponding tables and fields of the source repository and the target repository;

prompting, by the data validation computing device, the user to confirm the identified corresponding tables and fields of the source repository and the target repository;

receiving, by the data validation computing device, a data transformation specification from the user;

analyzing, by the data validation computing device, the data transformation specification to determine data transformation rules, wherein the data transformation rules are indicative of a relationship between the corresponding fields of the source repository and the target repository;

generating, by the data validation computing device, test cases and test scripts based on the data transformation rules;

executing, by the data validation computing device, the test cases and the test scripts on the source repository and the target repository to validate the relationship between the corresponding fields of the source repository and the target repository; and

generating, by the data validation computing device, a log file indicative of the outcome of the execution of the test cases and the test scripts.

7. The method as claimed in claim 6 further comprising:

determining, by the data validation computing device, a number of records on which the test cases and the test scripts are to be executed;

initiating, by the data validation computing device, parallel execution of the test cases and the test scripts on determining the number of records to have exceeded a critical threshold; and

initiating, by the data validation computing device, serial execution of the test cases and the test scripts on determining the number of records to be less than the critical threshold.

8. The method as claimed in claim 6 further comprising generating, by the data validation computing device, a report, based on the log file, wherein the report is indicative of whether values stored in the corresponding fields of the source repository and the target repository are in accordance with the data transformation rules.

9. The method as claimed in claim 6 further comprising:

determining, by the data validation computing device, a schema of the source repository and the target repository;

identifying, by the data validation computing device, attributes of the corresponding fields of the source repository and the target repository; and

determining, by the data validation computing device, the data transformation rules based on the identified attributes.

10. The method as claimed in claim 6 further comprising ascertaining, by the data validation computing device, at least one of inconsistent standards between corresponding fields of the source repository and the target repository, discrepancies in data format between the corresponding fields of the source repository and the target repository, difference in structure of data between the corresponding fields of the source repository and the target repository, missing values between the corresponding fields of the source repository and the target repository, data fields filled with default values or nulls in the corresponding fields of the source repository and the target repository, or data anomalies.

11. A non-transitory computer readable medium having stored thereon instructions for data validation over a communication network of data transformed from a source repository to a target repository comprising executable code which when executed by a processor causes the processor to perform steps comprising:

receiving an identifier from a user, wherein the identifier is indicative of an application whose associated data repositories are to be validated;

identifying the source repository and the target repository based on the identifier;

receiving configuration parameters of the source repository and the target repository, wherein the configuration parameters are indicative of at least one of data formats, structure, and schema of the source repository and the target repository;

receiving connection parameters of the source repository and the target repository, wherein the connection parameters are indicative of authentication details associated with accessing the source repository and the target repository;

establishing connection with the source repository and the target repository, based on the connection parameters;

identifying corresponding tables and fields of the source repository and the target repository;

prompting the user to confirm the identified corresponding tables and fields of the source repository and the target repository;

receiving a data transformation specification from the user;

analyzing the data transformation specification to determine data transformation rules, wherein the data transformation rules are indicative of a relationship between the corresponding fields of the source repository and the target repository;

generating test cases and test scripts based on the data transformation rules;

executing the test cases and the test scripts on the source repository and the target repository to validate the relationship between the corresponding fields of the source repository and the target repository; and

generating a log file indicative of the outcome of the execution of the test cases and the test scripts.

12. The medium as claimed in claim 11 further having stored thereon at least one additional instruction that when executed by the processor causes the processor to perform additional steps comprising:

determining a number of records on which the test cases and the test scripts are to be executed;

initiating parallel execution of the test cases and the test scripts on determining the number of records to have exceeded a critical threshold; and

initiating serial execution of the test cases and the test scripts on determining the number of records to be less than the critical threshold.

13. The medium as claimed in claim 11 further having stored thereon at least one additional instruction that when executed by the processor causes the processor to perform at least one additional step comprising generating a report, based on the log file, wherein the report is indicative of whether values stored in the corresponding fields of the source repository and the target repository are in accordance with the data transformation rules.

14. The medium as claimed in claim 11 further having stored thereon at least one additional instruction that when executed by the processor causes the processor to perform additional steps comprising:

determining a schema of the source repository and the target repository;

identifying attributes of the corresponding fields of the source repository and the target repository; and

determining the data transformation rules based on the identified attributes.

15. The medium as claimed in claim 11 further having stored thereon at least one additional instruction that when executed by the processor causes the processor to perform at least one additional step comprising ascertaining at least one of inconsistent standards between corresponding fields of the source repository and the target repository, discrepancies in data format between the corresponding fields of the source repository and the target repository, difference in structure of data between the corresponding fields of the source repository and the target repository, missing values between the corresponding fields of the source repository and the target repository, data fields filled with default values or nulls in the corresponding fields of the source repository and the target repository, or data anomalies.