Training Machine Learning Algorithms with Temporally Variant Personal Data, and Applications Thereof

• US 20190311299A1
• Filed: 04/09/2018
• Published: 10/10/2019
• Est. Priority Date: 04/09/2018
• Status: Active Grant

• Current Assignee: Veda Data Solutions Inc.
• Original Assignee: Veda Data Solutions Inc.
• Inventors: Lindner, Robert Raymond
• Granted Patent: US 11,568,302 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/2365 Ensuring data consistency and integrity
• CPC Class Codes: G06F 16/248 Presentation of query results
• CPC Class Codes: G06F 21/6209 to a single file or object, e.g. in a secure envelope, encrypted and accessed using a key, or with access control rules appended to the object itself
• CPC Class Codes: G06F 21/6245 Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks

Abstract

To train models, training data is needed. As personal data changes over time, the training data can get stale, obviating its usefulness in training the model. Embodiments deal with this by developing a database with a running log specifying how each person'"'"'s data changes at the time. When data is ingested, it may not he normalized. To deal with this, embodiments clean the data to ensure the ingested data fields are normalized. Finally, the various tasks needed to train the model and solve for accuracy of personal data can quickly become cumbersome to a computing device. They can conflict with one another and compete inefficiently for computing resources, such as processor power and memory capacity. To deal with these issues, a scheduler is employed to queue the various tasks involved.

Claims

1. A computer-implemented method for training a machine learning algorithm with temporally variant personal data, comprising:

(a) at a plurality of times, monitoring a data source to determine whether data relating to a person has updated;

(b) when data for the person has been updated, storing the updated data in a database such that the database includes a running log specifying how the person'"'"'s data has changed over time, wherein the person'"'"'s data includes values for a plurality of properties relating to the person;

(c) receiving an indication that a value for the particular property in the person'"'"'s data was verified as accurate or inaccurate at a particular time;

(d) retrieving, from the database based on the particular time, the person'"'"'s data, including values for the plurality of properties, that were up-to-date at the particular time; and

(e) training a model using the retrieved data and the indication such that the model can predict whether another person'"'"'s value for the particular property is accurate, whereby having the retrieved data be current to the particular time maintains the retrieved data'"'"'s significance in training the model.

2. The method of claim 1, further comprising:

(f) determining, based on the person'"'"'s data retrieved in (d), a plurality of features, each of the plurality of features describing a fact about the person'"'"'s data retrieved in (d),

wherein the training (e) comprises training the model using the determined features.

3. The method of claim 2, wherein the determining (f) comprises determining the features based on and which of the plurality of properties is the particular property.

4. The method of claim 1, wherein the training (e) comprises training a plurality of models, each module utilizing a different type of machine learning algorithm, further comprising:

(f) evaluating accuracy of the plurality of models using available training data; and

(g) selecting a model from the plurality of models determined based on the evaluated accuracy.

5. The method of claim 1, further comprising:

(f) applying the model to predict whether the other person'"'"'s value in the plurality of properties is accurate.

6. The method of claim 1, wherein the applying (f) comprises:

(i) for respective values in a plurality of values for the particular property of the other person, applying the model to the respective value to determine a score; and

(ii) selecting at least one value from the plurality of values based on the respective scores determined in (i).

7. The method of claim 6, wherein the monitoring (a) comprises monitoring a plurality of data sources to determine whether data relating to a person has updated, and wherein the applying (f) further comprises:

(iii) determining which of the plurality of data sources the at least one value selected in (ii) originated from;

(iv) determining whether a client has permission to the data source determined in (iii); and

(v) if the client lacks permission to the data source determined in (iii), filtering the at least one value from results before the results are presented to the client.

8. The method of claim 1, wherein the person and the other person are health care providers and the person'"'"'s and the other person'"'"'s data includes demographic information.

9. The method of claim 1, wherein the person and the other person are health care providers and the person'"'"'s data includes an indication of whether the person has engaged in fraud.

10. A non-transitory program storage device having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform a method for training a machine learning algorithm with temporally variant personal data, the method comprising:

(a) at a plurality of times, monitoring a data source to determine whether data relating to a person has updated;

(b) when data for the person has been updated, storing the updated data in a database such that the database includes a running log specifying how the person'"'"'s data has changed over time, wherein the person'"'"'s data includes values for a plurality of properties relating to the person;

(c) receiving an indication that a value for the particular property in the person'"'"'s data was verified as accurate or inaccurate at a particular time;

(d) retrieving, from the database based on the particular time, the person'"'"'s data, including values for the plurality of properties, that were up-to-date at the particular time; and

(e) training a model using the retrieved data and the indication such that the model can predict whether another person'"'"'s value for the particular property is accurate, whereby having the retrieved data be current to the particular time maintains the retrieved data'"'"'s significance in training the model.

11. The program storage device of claim 10, the method further comprising:

(f) determining, based on the person'"'"'s data retrieved in (d), a plurality of features, each of the plurality of features describing a fact about the person'"'"'s data retrieved in (d),

wherein the training (e) comprises training the model using the determined features.

12. The program storage device of claim 11, wherein the determining (f) comprises determining the features based on and which of the plurality of properties is the particular property.

13. The program storage device of claim 10, wherein the training (e) comprises training a plurality of models, each module utilizing a different type of machine learning algorithm, further comprising:

(f) evaluating accuracy of the plurality of models using available training data; and

(g) selecting a model from the plurality of models determined based on the evaluated accuracy.

14. The program storage device of claim 10, the method further comprising:

(f) applying the model to predict whether the other person'"'"'s value in the plurality of properties is accurate.

15. The program storage device of claim 10, wherein the applying (f) comprises:

(i) for respective values in a plurality of values for the particular property of the other person, applying the model to the respective value to determine a score; and

(ii) selecting at least one value from the plurality of values based on the respective scores determined in (i).

16. The program storage device of claim 15, wherein the monitoring (a) comprises monitoring a plurality of data sources to determine whether data relating to a person has updated, and wherein the applying (f) further comprises:

(iii) determining which of the plurality of data sources the at least one value selected in (ii) originated from;

(iv) determining whether a client has permission to the data source determined in (iii); and

(v) if the client lacks permission to the data source determined in (iii), filtering the at least one value from results before the results are presented to the client.

17. The program storage device of claim 10, wherein the person and the other person are health care providers and the person'"'"'s and the other person'"'"'s data includes demographic information.

18. The program storage device of claim 10, wherein the person and the other person are health care providers and the person'"'"'s data includes an indication of whether the person has engaged in fraud.

22. The system of claim 18, wherein the trainer trains a plurality of models, each module utilizing a different type of machine learning algorithm, further comprising:

a grid searcher that evaluates accuracy of the plurality of models using available training data and selects a model from the plurality of models determined based on the evaluated accuracy.

19. A system for training a machine learning algorithm with temporally variant personal data, comprising:

a computing device;

a database that includes a running log specifying how a person'"'"'s data has changed over time, wherein the person'"'"'s data includes values for a plurality of properties relating to the person;

a data ingestion process implemented on the computing device and configured to;

(i) at a plurality of times, monitor a data source to determine whether data relating to the person has updated; and

(ii) when data for the person has been updated, storing the updated data in the database;

an API monitor implemented on the computing device and configured to receive an indication that a value for the particular property in the person'"'"'s data was verified as accurate or inaccurate at a particular time;

a queries implemented on the computing device and configured to retrieve, from the database based on the particular time, the person'"'"'s data, including values for the plurality of properties, that were up-to-date at the particular time; and

a trainer implemented on the computing device and configured to train a model using the retrieved data and the indication such that the model can predict whether another person'"'"'s value for the particular property is accurate, whereby having the retrieved data be current to the particular time maintains the retrieved data'"'"'s significance in training the model.

20. The system of claim 19, further comprising:

a featurizer configured to determine, based on the person'"'"'s data retrieved in (d), a plurality of features, each of the plurality of features describing a fact about the person'"'"'s data retrieved in (d),

wherein the training (e) comprises training the model using the determined features.

21. The system of claim 19, wherein the model predicts whether the other person'"'"'s value in the plurality of properties is accurate.