Data integrity checks

• US 9,454,564 B1
• Filed: 09/09/2015
• Issued: 09/27/2016
• Est. Priority Date: 09/09/2015
• Status: Active Grant

• Current Assignee: Palantir Technologies Incorporated
• Original Assignee: Palantir Technologies Incorporated
• Inventors: Miliauskas, Gintautas
• Primary Examiner(s): Peng, Huawen A
• Application Number:
  US14/849,454
• Time in Patent Office: 384 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 11/327 Alarm or error message display
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 16/955 using information identifiers, e.g. uniform resource locators [URL]
• CPC Class Codes: G06F 21/64 Protecting data integrity, e.g. using checksums, certificates or signatures

Abstract

Aspects of the present disclosure relate to performing agnostic data integrity checks on source data, and based on the data integrity checks, generating a human-readable report that may be useable to identify specific errors or anomalies within the source data. Example embodiments involve systems and methods for performing the data integrity checks and generating the human-readable reports. For example, the method may include operations to ingest data from a source database through a data pipeline and into a local database, access the data from the data pipeline, determine a data type of the data, determine subtypes of data elements which make up the data, determine a count of each subtype, and generate a human-readable report, to be displayed at a client device.

Claims

1. A method comprising:

ingesting data through a data pipeline, the data pipeline located between a source database and a local database, the data comprising a plurality of data elements;

identifying a first data type associated with one or more data elements of the plurality of data elements;

identifying a subtype of the first data type;

retrieving a threshold value of the first data type and the subtype, the retrieving the threshold value including;

accessing historical data based on the source database of the ingested data, the historical data comprising a plurality of historical data elements;

determining the threshold value based on the historical data elements of the historical data;

calculating a count of data elements of the identified subtype within the ingested data;

detecting a difference between the threshold value and the count of the data elements of the identified subtype within the ingested data;

generating a human-readable report, the human-readable report including the subtype of each data element and the count of each subtype; and

causing display of the human-readable report at a client device, the human-readable report including a presentation of the count of the data elements of the identified subtype within the ingested data, the threshold value, and the difference between the threshold value and the count of the data elements.

2. The method of claim 1, wherein the retrieving the threshold value further comprises:

identifying a historical data type among the historical data;

identifying a historical subtype of the historical data type;

determining a historical count based on the historical subtype within the historical data elements; and

generating a historical human-readable report, the historical human-readable report including at least the historical subtype and the historical count of the historical subtype.

3. The method of claim 2, wherein the causing display of the human-readable report comprises:

comparing the historical data type and historical count of the historical human-readable report with the first data type and the count of the human-readable report;

detecting a difference between the historical data type and historical count presented in the historical human-readable report and the first data type and the count presented in the human-readable report; and

highlighting the difference.

4. The method of claim 1, wherein the identifying the data type comprises:

performing one or more Boolean checks to the data.

5. The method of claim 1, wherein the threshold value includes a numerical value and a percentage and the method further comprising:

causing display of a notification indicating the transgression in response to detecting the transgression, the notification including a presentation of the transgression.

6. The method of claim 5, further comprising receiving the threshold value from the client device.

7. The method of claim 1, further comprising:

receiving a set of expected subtypes corresponding to the data;

receiving a set of expected counts corresponding to the set of expected subtypes;

identifying a set of subtypes among the plurality of data elements;

calculating a count of each data element of each subtype from among the set of subtypes;

comparing the set of expected subtypes with the identified set of subtypes;

comparing the set of expected counts with the count of each data element of each subtype; and

detecting a difference between the set of expected subtypes and the identified set of subtypes.

8. The method of claim 7, wherein the method further comprises:

causing the difference to be visually distinguishing from the data.

9. A non-transitory machine-readable storage medium comprising instructions that, when executed by one or more processors of a machine, cause the machine to perform operations comprising:

ingesting data through a data pipeline, the data pipeline located between a source database and a local database, the data comprising a plurality of data elements;

identifying a first data type associated with one or more data elements of the plurality of data elements;

identifying a subtype of the first data type;

retrieving a threshold value of the first data type and the subtype, the retrieving the threshold value including;

accessing historical data based on the source database of the ingested data, the historical data comprising a plurality of historical data elements;

determining the threshold value based on the historical data elements of the historical data;

calculating a count of data elements of the identified subtype within the ingested data;

detecting a difference between the threshold value and the count of the data elements of the identified subtype within the ingested data;

generating a human-readable report, the human-readable report including the subtype of each data element and the count of each subtype;

causing display of the human-readable report at a client device, the human-readable report including a presentation of the count of the data elements of the identified subtype within the ingested data, the threshold value, and the difference between the threshold value and the count of the data elements.

10. The non-transitory machine-readable storage medium of claim 9, wherein the operations to retrieve the threshold value further comprise:

identifying a historical data type among the historical data;

identifying a historical subtype of the historical data type;

determining a historical count based on the historical subtype within the historical data elements; and

generating a historical human-readable report, the historical human-readable report including at least the historical subtype and the historical count of the historical subtype.

11. The non-transitory machine-readable storage medium of claim 10, wherein the operations to cause display of the human-readable report further comprise:

comparing the historical data type and historical count of the historical human-readable report with the first data type and the count of the human-readable report;

detecting a difference between the historical data type and historical count presented in the historical human-readable report and the first data type and the count presented in the human-readable report; and

highlighting the difference.

12. The non-transitory machine-readable storage medium of claim 9, wherein the operations further comprise:

accessing one or more Boolean checks;

applying the one or more Boolean checks to the data; and

determining the data type based on the one or more Boolean checks.

13. The non-transitory machine-readable storage medium of claim 9, wherein the threshold value includes a numerical value and a percentage and the operations further comprise:

causing display of a notification indicating the transgression in response to detecting the transgression, the notification including a presentation of the transgression.

14. The non-transitory machine-readable storage medium of claim 9, wherein the operations further comprise receiving the threshold value from the client device.

15. The non-transitory machine-readable storage medium of claim 9, wherein the operations further comprise:

receiving a set of expected subtypes corresponding to the data;

receiving a set of expected counts corresponding to the set of expected subtypes;

identifying a set of subtypes among the plurality of data elements;

calculating a count of each data element of each subtype from among the set of subtypes;

comparing the set of expected subtypes with the identified set of subtypes; and

comparing the set of expected counts with the count of each data element of each subtype.

16. The non-transitory machine-readable storage medium of claim 15, wherein the operations further comprise visually distinguishing the difference from the data.

17. A system comprising:

processors; and

a memory storing instructions that, when executed by at least one processor among the processors, cause the system to perform operations comprising;

ingesting data through a data pipeline, the data pipeline located between a source database and a local database, the data comprising a plurality of data elements;

identifying a first data type associated with one or more data elements of the plurality of data elements;

identifying a subtype of the first data type;

retrieving a threshold value of the first data type and the subtype, the retrieving the threshold value including;

accessing historical data based on the source database of the ingested data, the historical data comprising a plurality of historical data elements;

determining the threshold value based on the historical data elements of the historical data;

calculating a count of data elements of the identified subtype within the ingested data;

detecting a difference between the threshold value and the count of the data elements of the identified subtype within the ingested data;

generating a human-readable report, the human-readable report including the subtype of each data element and the count of each subtype; and

causing display of the human-readable report at a client device, the human-readable report including a presentation of the count of the data elements of the identified subtype within the ingested data, the threshold value, and the difference between the threshold value and the count of the data elements.

18. The system of claim 17, wherein the historical data comprises a plurality of historical data elements, and the retrieving the threshold value further comprises:

identifying a historical data type among the historical data;

identifying a historical subtype of the historical data type;

determining a historical count based on the historical subtype within the historical data elements; and

generating a historical human-readable report, the historical human-readable report including at least the historical subtype of each data element and the historical count of each historical subtype.

19. The system of claim 18, wherein the causing display of the human-readable report includes:

comparing the historical data type and historical count of the historical human-readable report with the first data type and the count of the human-readable report;

detecting a difference between the historical data type and historical count presented in the historical human-readable report and the first data type and the count presented in the human-readable report; and

highlighting the difference.

20. The system of claim 17, wherein the instructions for identifying the data type and subtype includes:

accessing one or more Boolean checks;

applying the one or more Boolean checks to the data; and

determining the data type and subtype based on the one or more Boolean checks.