Weight benefit evaluator for training data

• US 10,558,935 B2
• Filed: 08/05/2014
• Issued: 02/11/2020
• Est. Priority Date: 11/22/2013
• Status: Active Grant

• Current Assignee: California Institute of Technology
• Original Assignee: California Institute of Technology
• Inventors: Abu-Mostafa, Yaser Said, Gonzalez, Carlos Roberto
• Primary Examiner(s): Shmatov, Alexey
• Assistant Examiner(s): Alabi, Oluwatosin O
• Application Number:
  US14/451,859
• Publication Number:
  US 20150206065A1
• Time in Patent Office: 2,016 Days
• CPC Class Codes: G06N 20/00 Machine learning

Abstract

Technologies are generally described for methods and systems effective to determine a weight benefit associated with application of weights to training data in a machine learning environment. In an example, a device may determine a first function based on the training data, where the training data includes training inputs and training labels. The device may determine a second function based on weighted training data, which is based on application of weights to the training data. The device may determine a third function based on target data, where the target data is generated based on a target function. The target data may include target labels different from the training labels. The device may determine a fourth function based on weighted target data, which is a result of application of weights to the target data. The device may determine the weight benefit based on the first, second, third, and fourth functions.

Claims

1. A method to improve accuracy of a machine learning system based on a weight benefit value associated with training data, the method comprising:

determining, by a machine learning module of a device, a first function based on the training data, wherein the training data includes training inputs and training labels;

applying, by a processing module of the device, a set of weights to the training data to generate weighted training data;

determining, by the machine learning module of the device, a second function based on the weighted training data;

generating, by a target function generation module of the device, target data based on a target function, wherein the target data includes target labels different from the training labels;

determining, by the machine learning module of the device, a third function based on the target data;

applying, by the processing module of the device, the set of weights to the target data to generate weighted target data;

determining, by the machine learning module of the device, a fourth function based on the weighted target data;

determining, by an arithmetic module of the device, a first expected value between the first function and the second function;

determining, by the arithmetic module of the device, a second expected value between the third function and the target function;

determining, by the arithmetic module of the device, a third expected value between the fourth function and the target function;

determining, by the arithmetic module of the device, a fourth expected value between the third function and the fourth function;

determining, by an evaluation module of the device, an evaluation value with use of the second, third, and fourth expected values;

determining, by the evaluation module of the device, a count based on a comparison of the evaluation value with the first expected value; and

comparing, by the evaluation module of the device, the count with a threshold;

determining, by the evaluation module of the device, the weight benefit value based on the comparison of the count with the threshold, wherein the weight benefit value is associated with application of the set of weights to the training data;

receiving, at the machine learning system, an input;

in response to a determination that the count is greater than the threshold, applying, by the processing module of the device, the received input to the first function, wherein the first function is based on the training data without the set of weights applied thereto;

in response to a determination that the count is less than the threshold, applying, by the processing module of the device, the received input to the second function, wherein the second function is based on the weighted training data; and

generating, by the machine learning system, a first output, wherein the generation of the first output is based on the applying the received input to the first function, wherein the first output is different than a second output which is generated based on the applying the received input to the second function, wherein the generation of the first output or the second output based on the applying the received input to one of the first function and the second function is effective to benefit a performance of the machine learning system to provide improved accuracy by enabling the machine learning system to use or to refrain from the use of the set of weights depending on whether the use of the set of weights will result in the improved accuracy.

2. The method of claim 1, wherein the first, second, third, and fourth expected values are based on an application of test inputs to the first, second, third, and fourth functions, respectively.

3. The method of claim 1, wherein the generation of the weighted target data is based on the target function.

4. The method of claim 1, further comprising:

determining, by the target function generation module of the device, a set of parameters to generate an artificial function;

generating, by the target function generation module of the device, artificial data with use of the training inputs and the artificial function, wherein the artificial data includes the training inputs and artificial labels; and

generating, by the target function generation module of the device, the target function based on the training data and the artificial data.

5. The method of claim 4, wherein the generation of the target function comprises:

determining, by the target function generation module of the device, a first mean associated with the training data;

determining, by the target function generation module of the device, a first standard deviation associated with the training data;

determining, by the target function generation module of the device, a second mean associated with the artificial data; and

determining, by the target function generation module of the device, a second standard deviation associated with the artificial data, wherein the target function is based on the first mean, the first standard deviation, the second mean, and the second standard deviation.

6. The method of claim 5, wherein the target function is further based on a noise parameter.

7. The method of claim 5, wherein the target function is based on an application of one of:

a sign function to the first mean, the first standard deviation, the second mean, and the second standard deviation;

a round function to the first mean, the first standard deviation, the second mean, and the second standard deviation;

or a nearest number function to the first mean, the first standard deviation, the second mean, and the second standard deviation.

8. The method of claim 1, wherein determining the second function includes determining a relationship between weighted training inputs and weighted training labels, and wherein the weighted training inputs are associated with a user profile and the weighted training labels are associated with a movie rating.

9. A machine learning system effective to operate with improved accuracy based on a weight benefit value associated with training data, the machine learning system comprising:

a memory configured to;

store the training data, wherein the training data includes training inputs and training labels; and

store a set of weights;

a target function generation module configured to be in communication with the memory, wherein the target function generation module is configured to;

generate target data based on a target function, wherein the target data includes target labels different from the training labels; and

store the target data in the memory;

a processing module configured to be in communication with the target function generation module and the memory, wherein the processing module is configured to;

apply the set of weights to the training data to generate weighted training data;

apply the set of weights to the target data to generate weighted target data; and

store the weighted training data and the weighted target data in the memory;

a machine learning module configured to be in communication with the target function generation module, the processing module, and the memory, wherein the machine learning module is configured to;

determine a first function based on the training data;

determine a second function based on the weighted training data;

determine a third function based on the target data; and

determine a fourth function based on the weighted target data;

an arithmetic module configured to be in communication with the target function generation module, the processing module, the machine learning module, and the memory, wherein the arithmetic module is configured to;

determine a first expected value between the first function and the second function;

determine a second expected value between the third function and the target function;

determine a third expected value between the fourth function and the target function; and

determine a fourth expected value between the third function and the fourth function; and

an evaluation module configured to be in communication with the target function generation module, the processing module, the machine learning module, the arithmetic module, and the memory, wherein the evaluation module is configured to;

receive the first, second, third, and fourth expected values from the arithmetic module;

determine an evaluation value with use of the second, third, and fourth expected values;

determine a count based on a comparison of the evaluation value with the first expected value;

compare the count with a threshold; and

determine the weight benefit value based on the comparison of the count with the threshold, wherein the weight benefit value is associated with application of the set of weights to the training data,

wherein the processing module is further configured to;

in response to a determination that the count is greater than the threshold, apply an input, received by the machine learning system, to the first function, wherein the first function is based on the training data without the set of weights applied thereto; and

in response to a determination that the count is less than the threshold, apply the received input to the second function, wherein the second function is based on the weighted training data, and

wherein the machine learning module is further configured to;

generate a first output based on the processing module having applied the received input to the first function, wherein the first output is different than a second output which is generated based on the processing module having applied the received input to the second function, wherein the generation of the first output or the second output based on the processing module having applied the received input to one of the first function and the second function is effective to benefit a performance of the machine learning system to provide the improved accuracy.

10. The machine learning system of claim 9, wherein the arithmetic module is further configured to:

retrieve the target function from the memory, wherein the target function is based on the training data.

11. The machine learning system of claim 10, wherein the processing module is configured to generate the weighted target data with use of the target function.

12. The machine learning system of claim 9, wherein the target function generation module is further configured to:

determine a set of parameters to generate an artificial function;

generate artificial data with use of the training inputs and the artificial function, wherein the artificial data includes the training inputs and artificial labels; and

generate the target function based on the training data and the artificial data.

13. The machine learning system of claim 12, wherein the target function generation module is further configured to:

determine a first mean associated with the training data;

determine a first standard deviation associated with the training data;

determine a second mean associated with the artificial data; and

determine a second standard deviation associated with the artificial data,

wherein the target function is based on the first mean, the first standard deviation, the second mean, and the second standard deviation.

14. A method to improve accuracy of a machine learning system based on a weight benefit value associated with training data, the method comprising:

receiving, at a first device from a second device, a first function that is based on the training data, wherein the training data includes training inputs and training labels;

receiving, at the first device from the second device, a set of weights;

receiving, at the first device from the second device, a second function that is based on weighted training data, wherein the weighted training data is based on the set of weights;

generating, by a target function generation module of the first device, target data based on a target function, wherein the target data includes target labels different from the training labels;

determining, by a machine learning module of the first device, a third function based on the target data;

applying, by a processing module of the first device, the set of weights to the target data to generate weighted target data;

determining, by the machine learning module of the first device, a fourth function based on the weighted target data;

determining, by an arithmetic module of the first device, a first expected value between the first function and the second function;

determining, by the arithmetic module of the first device, a second expected value between the third function and the target function;

determining, by the arithmetic module of the first device, a third expected value between the fourth function and the target function;

determining, by the arithmetic module of the first device, a fourth expected value between the third function and the fourth function;

determining, by an evaluation module of the device, an evaluation value with use of the second, third, and fourth expected values;

determining, by the evaluation module of the device, a count based on a comparison of the evaluation value with the first expected value;

comparing, by the evaluation module of the device, the count with a threshold;

determining, by the evaluation module of the first device, the weight benefit value based on the comparison of the count with the threshold, wherein the weight benefit value is associated with application of the set of weights to the training data;

receiving, at the machine learning system, an input;

in response to a determination that the count is greater than the threshold, applying, by the processing module of the device, the received input to the first function, wherein the first function is based on the training data without the set of weights applied thereto;

in response to a determination that the count is less than the threshold, applying, by the processing module of the device, the received input to the second function, wherein the second function is based on the weighted training data; and

generating, by the machine learning system, a first output based on the applying the received input to the first function, wherein the first output is different than a second output which is generated by the applying the received input to the second function, wherein the generation of the first output or the second output based on applying the received input to one of the first function and the second function is effective to benefit a performance of the machine learning system to provide improved accuracy by enabling the machine learning system to use or to refrain from the use of the set of weights depending on whether the use of the set of weights will result in the improved accuracy.

15. The method of claim 14, wherein the first, second, third, and fourth expected values are based on an application of test inputs to the first, second, third, and fourth functions, respectively.

16. The method of claim 14, further comprising:

determining, by the target function generation module of the first device, a set of parameters to generate an artificial function;

generating, by the target function generation module of the first device, artificial data with use of the training inputs and the artificial function, wherein the artificial data includes the training inputs and artificial labels;

determining, by the target function generation module of the first device, a first mean associated with the training data;

determining, by the target function generation module of the first device, a first standard deviation associated with the training data;

determining, by the target function generation module of the first device, a second mean associated with the artificial data;

determining, by the target function generation module of the first device, a second standard deviation associated with the artificial data; and

generating, by the target function generation module of the first device, the target function with use of the first mean, the first standard deviation, the second mean, and the second standard deviation.