Computer-implemented system and method for analyzing data quality

• US 10,462,107 B2
• Filed: 08/08/2016
• Issued: 10/29/2019
• Est. Priority Date: 09/05/2014
• Status: Active Grant

• Current Assignee: Xerox Corporation (Xerox Holdings Corp.)
• Original Assignee: Palo Alto Research Center, Inc. (Xerox Holdings Corp.)
• Inventors: Freudiger, Julien, Rane, Shantanu, Brito, Alejandro E., Uzun, Ersin
• Primary Examiner(s): Hirl, Joseph P
• Assistant Examiner(s): Bell, Kalish K
• Application Number:
  US15/231,689
• Publication Number:
  US 20160352693A1
• Time in Patent Office: 1,177 Days
• CPC Class Codes: H04L 63/0407 wherein the identity of one or more communicating identities is hidden cryptographic mechanisms or cryptographic arrangements for anonymous credentials or for identity based cryptographic systems H04L9/00
• CPC Class Codes: H04L 63/0428 wherein the data content is protected, e.g. by encrypting or encapsulating the payload
• CPC Class Codes: H04L 63/0853 using an additional device, e.g. smartcard, SIM or a different communication terminal cryptographic mechanisms or cryptographic arrangements for entity authentication involving additional secure or trusted devices H04L9/3234
• CPC Class Codes: H04L 63/0876 based on the identity of the terminal or configuration, e.g. MAC address, hardware or software configuration or device fingerprint
• CPC Class Codes: H04L 67/01 Protocols

Abstract

A computer-implemented system and method for analyzing data quality is provided. Attributes each associated with one or more elements are maintained. A request from a user is received for determining data quality of at least one attribute based on an interest vector having a listing of the elements of that attribute and a selection of elements of interest. Each element is encrypted. A condensed vector having the same listing of elements as the interest vector is populated with occurrence frequencies for each of the listed elements. The elements of the condensed vector are encrypted by computing an encrypted product of each element in the condensed vector and the corresponding element of the interest vector. An aggregate is determined based on the encrypted products of each element of the interest vector and the corresponding element of the condensed vector. The aggregate is provided as results of the data quality.

Claims

1. A computer-implemented system for analyzing data quality, comprising:

memory storing a dataset comprising attributes each associated with one or more elements;

a client comprising an interest vector module to receive from a user a request for determining data quality of at least one attribute of the dataset based on an interest vector comprising a listing of the elements of that attribute and a selection of one or more of the elements as elements of interest, wherein the client generates the interest vector from one of a hashmap and a histogram and each element is encrypted; and

a server, comprising;

a condensed vector module to populate a condensed vector comprising the same listing of elements as the interest vector with occurrence frequencies for each of the listed elements;

an encryption module to encrypt the elements of the condensed vector by computing an encrypted product of each element in the condensed vector and the corresponding element of the interest vector;

an aggregate module to determine an aggregate based on the encrypted products of each element of the interest vector and the corresponding element of the condensed vector; and

providing the aggregate as results of the data quality request.

2. A system according to claim 1, wherein the client generates the interest vector by assigning a value of one to each of the elements of interest and a value of zero to each of the elements not of interest.

3. A system according to claim 1, wherein the client determines a length of the interest vector by one of setting the vector length as a number of the elements associated with that attribute and setting the vector length as a number of bins into which the elements are organized.

4. A system according to claim 1, wherein the server generates the condensed vector via one of a counting hashmap and a histogram.

5. A system according to claim 4, wherein the hashmap of the interest vector and the counting hashmap of the condensed vector use the same hashing functions.

6. A system according to claim 4, wherein the server obtains the frequencies of occurrence from one of the counting hashmap and histogram.

7. A system according to claim 1, wherein the client decrypts the aggregate for the attribute and calculates a quality of data value based on the decrypted aggregate.

8. A system according to claim 7, wherein a threshold is applied to the quality of data value to provide an indication of the data quality.

9. A system according to claim 1, wherein testing of the attributes comprises at least one of testing each attribute in the dataset for the determined data quality and testing the attributes in the dataset in order until the first x number of attributes test high for the determined data quality, then the testing of the remaining attributes is skipped.

10. A computer-implemented method for analyzing data quality, comprising:

maintaining a dataset comprising attributes each associated with one or more elements;

receiving from a user a request for determining data quality of at least one attribute of the dataset based on an interest vector comprising a listing of the elements of that attribute and a selection of one or more of the elements as elements of interest, wherein the interest vector is generated via one of a hashmap and a histogram and each element is encrypted;

populating a condensed vector comprising the same listing of elements as the interest vector with occurrence frequencies for each of the listed elements;

encrypting the elements of the condensed vector by computing an encrypted product of each element in the condensed vector and the corresponding element of the interest vector;

determining an aggregate based on the encrypted products of each element of the interest vector and the corresponding element of the condensed vector; and

providing the aggregate as results of the data quality request.

11. A method according to claim 10, further comprising:

generating the interest vector by assigning a value of one to each of the elements of interest and a value of zero to each of the elements not of interest.

12. A method according to claim 10, further comprising:

determining a length of the interest vector, comprising one of;

setting the vector length as the number of elements associated with that attribute; and

organizing the elements into bins and setting the vector length as the number of bins.

13. A method according to claim 10, further comprising:

generating the condensed vector via one of a counting hashmap and a histogram.

14. A method according to claim 13, wherein the hashmap of the interest vector and the counting hashmap of the condensed vector use the same hashing functions.

15. A method according to claim 13, further comprising:

obtaining the frequencies of occurrence from one of the counting hashmap and histogram.

16. A method according to claim 10, further comprising:

decrypting via a client the aggregate for the attribute; and

calculating a quality of data value based on the decrypted aggregate.

17. A method according to claim 16, further comprising:

providing an indication of the data quality by applying a threshold to the quality of data value.

18. A method according to claim 10, further comprising at least one of:

testing each attribute in the dataset for the determined data quality; and

ordering the attributes in the dataset for testing and when the first x number of attributes test high for the determined data quality, skipping testing of the remaining attributes.