Image restoration cascade

• US 9,396,523 B2
• Filed: 07/24/2013
• Issued: 07/19/2016
• Est. Priority Date: 07/24/2013
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Jancsary, Jeremy, Nowozin, Reinhard Sebastian Bernhard, Rother, Carsten Curt Eckard
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Lee, Jonathan S
• Application Number:
  US13/949,940
• Publication Number:
  US 20150030237A1
• Time in Patent Office: 1,091 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks

Abstract

Image restoration cascades are described, for example, where digital photographs containing noise are restored using a cascade formed from a plurality of layers of trained machine learning predictors connected in series. For example, noise may be from sensor noise, motion blur, dust, optical low pass filtering, chromatic aberration, compression and quantization artifacts, down sampling or other sources. For example, given a noisy image, each trained machine learning predictor produces an output image which is a restored version of the noisy input image; each trained machine learning predictor in a given internal layer of the cascade also takes input from the previous layer in the cascade. In various examples, a loss function expressing dissimilarity between input and output images of each trained machine learning predictor is directly minimized during training. In various examples, data partitioning is used to partition a training data set to facilitate generalization.

Claims

1. A computer-implemented method of restoring an image comprising:

receiving, at a processor, a poor quality image;

applying the poor quality image to each of a plurality of trained machine learning predictors which are arranged in a cascade architecture, the cascade architecture comprising a plurality of layers connected in series, each layer comprising at least one of the trained machine learning predictors, each internal layer of the cascade architecture being trained using more training data examples than the first layer of the cascade architecture;

obtaining from each of the trained machine learning predictors, a restored version of the poor quality image.

2. A method as claimed in claim 1 wherein the cascade architecture comprises at least three layers and at least one internal layer receives at least some input from at least one earlier layer other than the immediately previous layer.

3. A method as claimed in claim 1 comprising displaying one or more of the restored versions of the poor quality image at an end user device and, in response to user input, stopping application of the poor quality image to at least some of the plurality of trained machine learning predictors.

4. A method as claimed in claim 1 comprising applying the poor quality image to the plurality of trained machine learning predictors by sending the poor quality image to an image restoration cascade over a communications network.

5. A method as claimed in claim 1 where trained machine learning predictors in internal layers of the cascade architecture receive input from one or more trained machine learning predictors of earlier layers.

6. A method as claimed in claim 1 where at least two of the layers of the cascade architecture comprise a plurality of trained machine learning predictors arranged in rows.

7. A method as claimed in claim 1 comprising obtaining from each of the trained machine learning predictors, certainty information associated with the restored versions of the poor quality image.

8. A method as claimed in claim 6 comprising:

obtaining from each of the trained machine learning predictors, certainty information associated with the restored versions of the poor quality image; and

using the certainty information to select or weight data input to layers of the cascade architecture.

9. A method as claimed in claim 1 comprising training the plurality of machine learning predictors using a loss function which expresses restoration quality between a poor quality image and a restored version of the poor quality image.

10. A method as claimed in claim 1 comprising training each of the machine learning predictors using a different training data set.

11. A method as claimed in claim 1 wherein the trained machine learning predictors are non-parametric.

12. A method as claimed in claim 1 wherein each trained machine learning predictor is any of:

a neural network, a linear regression model, a regression tree, a regression tree field.

13. A method of restoring an image comprising:

receiving, at a processor, a poor quality image;

applying the poor quality image to a plurality of trained machine learning predictors which are arranged in a cascade architecture, the cascade architecture comprising a plurality of layers connected in series, each layer comprising at least one of the trained machine learning predictors, each internal layer of the cascade architecture being trained using more training data examples than the first layer of the cascade architecture;

obtaining at least one of a series of restored versions of the poor quality image from the trained machine learning predictors;

displaying the obtained restored versions of the poor quality image and monitoring for user input;

in response to user input, stopping application of the poor quality image to at least some of the plurality of trained machine learning predictors.

14. An image restoration engine comprising:

a processor arranged to receive a poor quality image; and

a plurality of trained machine learning predictors arranged to use the poor quality image to calculate restored versions of the poor quality image, the plurality of trained machine learning predictors being arranged in a cascade architecture, the cascade architecture comprising a plurality of layers connected in series, each layer comprising at least one of the trained machine learning predictors, each internal layer of the cascade architecture being trained using more training data examples than the first layer of the cascade architecture.

15. An image restoration engine as claimed in claim 14 where at least one internal layer receives image features of at least one of the restored versions of the poor quality image as input.

16. An image restoration engine as claimed in claim 14 where trained machine learning predictors in internal layers of the cascade architecture receive input from one or more trained machine learning predictors of earlier layers.

17. An image restoration engine as claimed in claim 14 comprising obtaining from each of the trained machine learning predictors, certainty information associated with the restored versions of the poor quality image.

18. An image restoration engine as claimed in claim 14 where each of the machine learning predictors has been trained using a different training data set.

19. An image restoration engine as claimed in claim 14 wherein each trained machine learning predictor is any of:

a neural network, a linear regression model, a regression tree, a regression tree field.

20. An image restoration engine as claimed in claim 14 which is at least partially implemented using hardware logic selected from any one or more of:

a field-programmable gate array, a program-specific integrated circuit, a program-specific standard product, a system-on-a-chip, a complex programmable logic device, a graphics processing unit.