Systems and methods for training neural networks for regression without ground truth training samples

• US 10,565,686 B2
• Filed: 11/08/2017
• Issued: 02/18/2020
• Est. Priority Date: 06/12/2017
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Lehtinen, Jaakko T., Aila, Timo Oskari, Hasselgren, Jon Niklas Theodor, Munkberg, Carl Jacob
• Primary Examiner(s): Couso, Yon J
• Application Number:
  US15/807,401
• Publication Number:
  US 20180357753A1
• Time in Patent Office: 832 Days
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06T 5/70 Denoising; Smoothing

Abstract

A method, computer readable medium, and system are disclosed for training a neural network. The method includes the steps of selecting an input sample from a set of training data that includes input samples and noisy target samples, where the input samples and the noisy target samples each correspond to a latent, clean target sample. The input sample is processed by a neural network model to produce an output and a noisy target sample is selected from the set of training data, where the noisy target samples have a distribution relative to the latent, clean target sample. The method also includes adjusting parameter values of the neural network model to reduce differences between the output and the noisy target sample.

Claims

1. A computer-implemented method for training a neural network, comprising:

selecting an input sample from a set of training data that includes input samples and noisy target samples, wherein the input samples and the noisy target samples each correspond to a latent, clean target sample;

processing the input sample by a neural network model to produce an output;

selecting a noisy target sample from the set of training data, wherein the noisy target samples have a distribution relative to the latent, clean target sample; and

adjusting parameter values of the neural network model to reduce differences between the output and the noisy target sample.

2. The computer-implemented method of claim 1, wherein the clean target sample equals a mean of the distribution of the noisy target samples.

3. The computer-implemented method of claim 2, wherein the parameter values are adjusted using a least squares loss function based on the differences.

4. The computer-implemented method of claim 1, wherein the clean target sample equals a median of the distribution of the noisy target samples.

5. The computer-implemented method of claim 4, wherein the parameter values are adjusted using a least absolute deviations loss function based on the differences.

6. The computer-implemented method of claim 4, wherein the input sample is corrupted with noise that is not correlated with either the input sample or the noisy target sample.

7. The computer-implemented method of claim 1, further comprising:

repeating, for at least one additional output, selecting the input sample, processing the input sample, and selecting the noisy target sample; and

before adjusting the parameter values, combining the differences for the output and additional differences for the at least one additional output and then adjusting the parameter values of the neural network model to reduce the combined differences.

8. The computer-implemented method of claim 7, wherein the combining averages the differences for the at least two outputs.

9. The computer-implemented method of claim 1, wherein noise in the noisy target samples is statistically independent, zero-mean noise.

10. The computer-implemented method of claim 1, wherein the input samples are clean input samples.

11. The computer-implemented method of claim 1, wherein the input samples are noisy input samples and the noise associated with the noisy input samples is not correlated with the noise associated with the noisy target samples.

12. The computer-implemented method of claim 1, further comprising:

processing the input sample to generate additional outputs;

selecting additional noisy target samples from the set of training data; and

adjusting the parameter values of the neural network model to reduce additional differences between the additional outputs and the additional noisy target samples.

13. The computer-implemented method of claim 1, further comprising:

selecting additional input samples from the set of training data;

processing the additional input samples to generate additional outputs;

selecting additional noisy target samples from the set of training data; and

adjusting the parameter values of the neural network model to reduce additional differences between the additional outputs and the additional noisy target samples.

14. The computer-implemented method of claim 1, further comprising:

selecting an additional noisy target sample from the set of training data;

computing importance values based on the output, the noisy target sample, and the additional noisy target sample; and

scaling loss function values of the neural network model by the importance values.

15. The computer-implemented method of claim 1, wherein the clean target sample is not used to compute the differences.

16. The computer-implemented method of claim 1, wherein each noisy target sample is different than the other noisy target samples in the set of training data.

17. The computer-implemented method of claim 1, wherein the input samples have a second distribution relative to the latent, clean target sample that is different compared with the distribution.

18. The computer-implemented method of claim 1, wherein the input samples also have the distribution relative to the latent, clean target sample.

19. A system, comprising:

a memory storing a set of training data that includes input samples and noisy target samples, wherein the input samples and the noisy target samples each correspond to a latent, clean target sample;

a parallel processing unit that is coupled to the memory and configured to;

select an input sample from the set of training data;

process the input sample by a neural network model to produce an output;

select a noisy target sample from the set of training data, wherein the noisy target samples have a distribution relative to the latent, clean target sample; and

adjust parameter values of the neural network model to reduce differences between the output and the noisy target sample.

20. A non-transitory computer-readable media storing computer instructions for training a neural network that, when executed by a processor, cause the processor to perform the steps of:

selecting an input sample from a set of training data that includes input samples and noisy target samples, wherein the input samples and the noisy target samples each correspond to a latent, clean target sample;

processing the input sample by a neural network model to produce an output;

selecting a noisy target sample from the set of training data, wherein the noisy target samples have a distribution relative to the latent, clean target sample; and

adjusting parameter values of the neural network model to reduce differences between the output and the noisy target sample.