Management and Evaluation of Machine-Learned Models Based on Locally Logged Data

• US 20180144265A1
• Filed: 11/21/2016
• Published: 05/24/2018
• Est. Priority Date: 11/21/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Bonawitz, Keith, Ramage, Daniel
• Granted Patent: US 10,769,549 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/098 Distributed learning, e.g. federated learning
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06N 7/06 Simulation on general purpose computers

Abstract

The present disclosure provides systems and methods for the management and/or evaluation of machine-learned models based on locally logged data. In one example, a user computing device can obtain a machine-learned model (e.g., from a server computing device) and can evaluate at least one performance metric for the machine-learned model. In particular, the at least one performance metric for the machine-learned model can be evaluated relative to data that is stored locally at the user computing device. The user computing device and/or the server computing device can determine whether to activate the machine-learned model on the user computing device based at least in part on the at least one performance metric. In another example, the user computing device can evaluate a plurality of machine-learned models against locally stored data. At least one of the models can be selected based on the evaluated performance metrics.

Claims

1. A computer-implemented method to manage machine-learned models, the method comprising:

obtaining, by a user computing device, a plurality of machine-learned models;

evaluating, by the user computing device, at least one performance metric for each of the plurality of machine-learned models, wherein the at least one performance metric for each machine-learned model is evaluated relative to data that is stored locally at the user computing device;

determining, by the user computing device, a selection of a first machine-learned model from the plurality of machine-learned models based at least in part on the performance metrics respectively evaluated for the plurality of machine-learned models; and

using, by the user computing device, the selected first machine-learned model to obtain one or more predictions.

2. The computer-implemented method of claim 1, wherein evaluating, by the user computing device, the at least one performance metric for each of the plurality of machine-learned models comprises using, by the user computing device, historical data that was previously logged at the user computing device to evaluate the at least one performance metric for each of the plurality of machine-learned models.

3. The computer implemented method of claim 2, wherein using, by the user computing device, the historical data that was previously logged at the user computing device to evaluate the at least one performance metric for each of the plurality of machine-learned models comprises:

inputting, by the user computing device, at least a first portion of the historical data into each of the plurality of machine-learned models;

receiving, by the user computing device, at least one prediction from each of the plurality of machine-learned models in response to input of the first portion of the historical data; and

comparing, by the user computing device, the at least one prediction from each machine-learned model to a second portion of the historical data to respectively determine the at least one performance metric for each of the plurality of machine-learned models.

4. The computer-implemented method of claim 1, wherein evaluating, by the user computing device, the at least one performance metric for each of the plurality of machine-learned models comprises, after obtaining, by the user computing device, the plurality of machine-learned models, using, by the user computing device, new data that is newly logged at the user computing device to evaluate the at least one performance metric for each of the plurality of machine-learned models.

5. The computer-implemented method of claim 4, wherein using, by the user computing device, new data that is newly logged at the user computing device to evaluate the at least one performance metric for each of the plurality of machine-learned models comprises:

inputting, by the user computing device, at least a first portion of the new data into each of the plurality of machine-learned models;

receiving, by the user computing device, at least one prediction from each of the plurality of machine-learned models in response to input of the first portion of the new data; and

comparing, by the user computing device, the at least one prediction from each machine-learned model to a second portion of the new data to respectively determine the at least one performance metric for each of the plurality of machine-learned models.

6. The computer-implemented method of claim 1, wherein determining, by the user computing device, the selection of the first machine-learned model from the plurality of machine-learned models comprises:

selecting, by the user computing device, the first machine-learned model from the plurality of machine-learned models based at least in part on the performance metrics respectively evaluated for the plurality of machine-learned models.

7. The computer-implemented method of claim 1, wherein determining, by the user computing device, the selection of the first machine-learned model from the plurality of machine-learned models comprises:

providing, by the user computing device, the performance metrics respectively evaluated for the plurality of machine-learned models to a server computing device; and

receiving, by the user computing device, the selection of the first machine-learned model from the server computing device.

8. The computer-implemented method of claim 1, wherein evaluating, by the user computing device, at least one performance metric for each of the plurality of machine-learned models comprises evaluating, by the user computing device for each of the plurality of machine-learned models, an average probability of correct prediction.

9. The computer-implemented method of claim 1, wherein obtaining, by the user computing device, the plurality of machine-learned models comprises obtaining, by the user computing device, the plurality of machine-learned models that have been respectively trained on different types of training data.

10. The computer-implemented method of claim 1, further comprising:

reporting, by the user computing device, one or more of the performance metrics evaluated for the plurality of machine-learned models to a server computing device.

11. The computer-implemented method of claim 10, further comprising:

aggregating, by the server computing device, the one or more performance metrics received from the user computing device with other performance metrics reported by other computing devices to assess a global performance for each of the plurality of machine-learned models.

16. The computing device of claim 10, wherein to determine whether to activate the machine-learned model based at least in part on the at least one performance metric the computing device:

provides the at least one performance metric evaluated for the machine-learned model to a server computing device; and

receives a determination of whether to activate the machine-learned model from the server computing device.

12. A computing device, the computing device comprising:

one or more processors; and

one or more non-transitory computer-readable media that collectively store instructions that, when executed by the one or more processors, cause the computing device to;

obtain a machine-learned model;

evaluate at least one performance metric for the machine-learned model, wherein the at least one performance metric for the machine-learned model is evaluated relative to data that is stored locally at the computing device;

determine whether to activate the machine-learned model based at least in part on the at least one performance metric evaluated for the machine-learned model; and

when it is determined that the machine-learned model should be activated, use the machine-learned model to obtain one or more predictions.

13. The computing device of claim 12, wherein the computing device uses historical data that was previously logged at the user computing device prior to receipt of the machine-learned model to evaluate the at least one performance metric the machine-learned model.

14. The computing device of claim 12, wherein the computing device uses new data that is newly logged at the user computing device after receipt of the machine-learned model to evaluate the at least one performance metric for each of the plurality of machine-learned models.

15. The computing device of claim 12, wherein to determine whether to activate the machine-learned model based at least in part on the at least one performance metric the computing device:

compares the at least one performance metric to at least one threshold value; and

determines whether the at least one performance metric compares favorably to the at least one threshold value, wherein it is determined that the machine-learned model should be activated when the at least one performance metric compares favorably to the at least one threshold value.

17. The computing device of claim 12, wherein the at least one performance metric comprises an average probability of correct prediction.

18. One or more non-transitory computer-readable media that collectively store instructions that, when executed by one or more processors of a computing system, cause the computing system to:

obtain a plurality of performance metrics respectively associated with a plurality of machine-learned models, wherein the performance metric for each machine-learned model indicates a performance of such machine-learned model when evaluated against a set of data that is stored locally at a user computing device;

select at least one of the plurality of machine-learned models based at least in part on the plurality of performance metrics; and

cause use of the selected at least one machine-learned model at the user computing device.

19. The one or more non-transitory computer-readable media of claim 18, wherein the computing system consists of a server computing device, and wherein to obtain the plurality of performance metrics, the server computing device receives the plurality of performance metrics from the user computing device.

20. The one or more non-transitory computer-readable media of claim 19, wherein execution of the instructions further causes the server computing device to aggregate the plurality of performance metrics received from the user computing device with a plurality of additional performance metrics received from additional computing devices to determine a global performance for each of the plurality of machine-learned models.