Systems and methods for validating data

• US 10,503,574 B1
• Filed: 08/07/2017
• Issued: 12/10/2019
• Est. Priority Date: 04/10/2017
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Lisuk, David, Xu, Guodong, Voloch, Luis, Elkherj, Matthew
• Primary Examiner(s): Le, Dieu Minh T
• Application Number:
  US15/670,304
• Time in Patent Office: 855 Days
• Field of Search: 714 48, 714 49, 714 50, 714 381, 714 57
• CPC Class Codes: G06F 11/0709 in a distributed system consisting of a plurality of standalone computer nodes, e.g. clusters, client-server systems
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/23 Updating
• CPC Class Codes: G06F 16/43 Querying
• CPC Class Codes: G06F 17/18 for evaluating statistical data , e.g. average values, frequency distributions, probability functions, regression analysis forecasting specially adapted for a specific administrative, business or logistic context G06Q10/04
• CPC Class Codes: G06F 3/048 Interaction techniques based on graphical user interfaces [GUI]
• CPC Class Codes: G06Q 10/10 Office automation; Time management

Abstract

Systems and methods are validating data in a data set. A data set including data to validate and a validator to use in validating the data is selected based on user input generated based on interactions of a user with a graphical user interface. The validator is applied to the data to determine whether one or more statistics generated through application of the validator to the data is valid or invalid based on a validation routine associated with the validator. A data quality report indicating whether the data set is valid or invalid, based on a determination of whether the one or more statistics is valid or invalid, is generated and selectively presented to the user through the graphical user interface.

Claims

1. A system for validating data comprising:

one or more processors and a memory storing instructions that, when executed by the one or more processors, cause the system to;

select a data set including data to validate based on user input received from a user in interacting with a graphical user interface;

select a validator including at least one validation parameter to use in validating the data in the data set based on the user input;

apply the validator to the data in the data set to determine whether one or more statistics generated through application of the validator to the data in the data set using the at least one validation parameter is valid according to at least one validation routine associated with the validator;

flag the data set as invalid if at least one of the one or more statistics generated through application of the validator are determined to be invalid according to the at least one validation routine;

generate a data quality report for the data set indicating whether the data set is valid or invalid based on a determination of whether the one or more statistics are valid according to the at least one validation routine;

selectively present the data quality report to the user through the graphical user interface;

re-apply the validator to the data in the data set to determine whether the one or more statistics generated through application of the validator to the data in the data set using the at least one validation parameter are valid according to the at least one validation routine;

generate another data quality report for the data indicating whether the data set is valid or invalid based on a determination made through re-application of the validator to the data in the data set of whether the one or more statistics are valid according to the at least one validation routine; and

selectively present the another data quality report to the user through the graphical user interface.

2. The system of claim 1, wherein the system is further caused to select the at least one validation parameter based on the user input.

3. The system of claim 1, wherein the system is further caused to:

select the at least one validation routine based on the user input;

customize the validator by associating the validator with the at least one validation routine selected based on the user input for use in applying the validator to the data in the data set to determine whether the one or more statistics are valid according to the at least one validation routine.

4. The system of claim 1, wherein the system is further caused to:

select the at least one validation parameter based on the user input;

customize the validator by associating the validation parameters with the validator according to the user input.

5. The system of claim 1, wherein the system is further caused to:

receive additional user input;

modify the validator to create a modified validator by modifying the at least one validation parameter to create at least one modified validation parameter in the modified validator according to the additional user input;

apply the modified validator to the data in the data set to determine whether an additional one or more statistics generated through application of the modified validator to the data in the data set using the at least one modified validation parameter are valid according to the at least one validation routine;

generate another data quality report for the data set indicating whether the data set is valid or invalid based on a determination of whether the additional one or more statistics are valid according to the at least one validation routine; and

selectively present the another data quality report to the user through the graphical user interface.

6. The system of claim 1, wherein the system is further caused to:

receive additional user input;

customize the validator to create a customized validator by modifying the at least one validation routine to create at least one modified validation routine in the customized validator according to the additional user input;

apply the customized validator to the data in the data set to determine whether an additional one or more statistics generated through application of the customized validator to the data in the data set using the at least one validation parameter are valid according to the at least one modified validation routine;

generate another data quality report for the data set indicating whether the data set is valid or invalid based on a determination of whether the additional one or more statistics are valid according to the at least one modified validation routine; and

selectively present the another data quality report to the user through the graphical user interface.

7. The system of claim 1, wherein the system is further caused to re-apply the validator to the data in the data set based on one or a combination of re-application instructions received from a user, a passage of a specific amount of time, and a change made to the data in the data set.

8. The system of claim 1, wherein the system is further caused to:

select another data set including additional data to validate using the validator based on the user input;

concurrently with application of the validator to the data in the data set, apply the validator to the additional data in the another data set to determine whether one or more additional statistics generated through application of the validator to the additional data in the another data set using the at least one validation parameter is valid according to the at least one validation routine associated with the validator;

generate another data quality report for the data set and the another data set indicating whether either or both the data set and the another data set are valid or invalid based on a determination of whether either or both the one or more statistics and the one or more additional statistics are valid according to the at least one validation routine; and

selectively present the another data quality report to the user through the graphical user interface.

9. The system of claim 1, wherein the system is further caused to:

select another validator including at least one another validation parameter and at least one another validation routine to use in validating the data in the data set based on the user input;

concurrently with application of the validator to the data in the data set, apply the another validator to the data in the data set to determine whether one or more additional statistics generated through application of the another validator to the data in the data set using the at least one another validator parameter are valid according to the at least one another validation routine associated with the another validator;

generate another data quality report for the data set indicating whether the data set is valid or invalid based on either or both the determination of whether the one or more statistics are valid according to the at least one validation routine and a determination of whether the one or more additional statistics are valid according to the at least one another validation routine; and

selectively present the another data quality report to the user through the graphical user interface.

10. A method being implemented by a computing system including one or more physical processors and storage media storing machine-readable instructions, the method comprising:

selecting a data set including data to validate based on user input received from a user in interacting with a graphical user interface;

selecting a validator including at least one validation parameter to use in validating the data in the data set based on the user input;

applying the validator to the data in the data set to determine whether one or more statistics generated through application of the validator to the data in the data set using the at least one validation parameter is valid according to at least one validation routine associated with the validator;

flagging the data set as invalid if at least one of the one or more statistics generated through application of the validator are determined to be invalid according to the at least one validation routine;

generating a data quality report for the data set indicating whether the data set is valid or invalid based on a determination of whether the one or more statistics are valid according to the at least one validation routine;

selectively presenting the data quality report to the user through the graphical user interface;

re-applying the validator to the data in the data set to determine whether the one or more statistics generated through application of the validator to the data in the data set using the at least one validation parameter are valid according to the at least one validation routine;

generating another data quality report for the data indicating whether the data set is valid or invalid based on a determination made through re-application of the validator to the data in the data set of whether the one or more statistics are valid according to the at least one validation routine; and

selectively presenting the another data quality report to the user through the graphical user interface.

11. The method of claim 10, further comprising selecting the at least one validation parameter based on the user input.

12. The method of claim 10, further comprising:

selecting the at least one validation routine based on the user input;

customizing the validator by associating the validator with the at least one validation routine selected based on the user input for use in applying the validator to the data in the data set to determine whether the one or more statistics are valid according to the at least one validation routine.

13. The method of claim 10, further comprising:

selecting the at least one validation parameter based on the user input;

customizing the validator by associating the validation parameters with the validator according to the user input.

14. The method of claim 10, further comprising:

receiving additional user input;

modifying the validator to create a modified validator by modifying the at least one validation parameter to create at least one modified validation parameter in the modified validator according to the additional user input;

applying the modified validator to the data in the data set to determine whether an additional one or more statistics generated through application of the modified validator to the data in the data set using the at least one modified validation parameter are valid according to the at least one validation routine;

generating another data quality report for the data set indicating whether the data set is valid or invalid based on a determination of whether the additional one or more statistics are valid according to the at least one validation routine; and

selectively presenting the another data quality report to the user through the graphical user interface.

15. The method of claim 10, further comprising:

receiving additional user input;

customizing the validator to create a customized validator by modifying the at least one validation routine to create at least one modified validation routine in the customized validator according to the additional user input;

applying the customized validator to the data in the data set to determine whether an additional one or more statistics generated through application of the customized validator to the data in the data set using the at least one validation parameter are valid according to the at least one modified validation routine;

generating another data quality report for the data set indicating whether the data set is valid or invalid based on a determination of whether the additional one or more statistics are valid according to the at least one modified validation routine; and

selectively presenting the another data quality report to the user through the graphical user interface.

16. The method of claim 10, further comprising:

selecting another data set including additional data to validate using the validator based on the user input;

concurrently with application of the validator to the data in the data set, applying the validator to the additional data in the another data set to determine whether one or more additional statistics generated through application of the validator to the additional data in the another data set using the at least one validation parameter is valid according to the at least one validation routine associated with the validator;

generating another data quality report for the data set and the another data set indicating whether either or both the data set and the another data set are valid or invalid based on a determination of whether either or both the one or more statistics and the one or more additional statistics are valid according to the at least one validation routine; and

selectively presenting the another data quality report to the user through the graphical user interface.

17. The method of claim 10, further comprising:

selecting another validator including at least one another validation parameter and at least one another validation routine to use in validating the data in the data set based on the user input;

concurrently with application of the validator to the data in the data set, applying the another validator to the data in the data set to determine whether one or more additional statistics generated through application of the another validator to the data in the data set using the at least one another validator parameter are valid according to the at least one another validation routine associated with the another validator;

generating another data quality report for the data set indicating whether the data set is valid or invalid based on either or both the determination of whether the one or more statistics are valid according to the at least one validation routine and a determination of whether the one or more additional statistics are valid according to the at least one another validation routine; and

selectively presenting the another data quality report to the user through the graphical user interface.

18. A non-transitory computer readable medium comprising instructions that, when executed, cause one or more processors to perform:

selecting a data set including data to validate based on user input received from a user in interacting with a graphical user interface;

selecting a validator including at least one validation parameter to use in validating the data in the data set based on the user input;

applying the validator to the data in the data set to determine whether one or more statistics generated through application of the validator to the data in the data set using the at least one validation parameter is valid according to at least one validation routine associated with the validator;

flagging the data set as invalid if at least one of the one or more statistics generated through application of the validator are determined to be invalid according to the at least one validation routine;

generating a data quality report for the data set indicating whether the data set is valid or invalid based on a determination of whether the one or more statistics are valid according to the at least one validation routine;

selectively presenting the data quality report to the user through the graphical user interface;

re-applying the validator to the data in the data set to determine whether the one or more statistics generated through application of the validator to the data in the data set using the at least one validation parameter are valid according to the at least one validation routine;

generating another data quality report for the data indicating whether the data set is valid or invalid based on a determination made through re-application of the validator to the data in the data set of whether the one or more statistics are valid according to the at least one validation routine; and

selectively presenting the another data quality report to the user through the graphical user interface.