Discriminative indexing for patch-based image enhancement

• US 9,342,869 B2
• Filed: 04/29/2014
• Issued: 05/17/2016
• Est. Priority Date: 04/29/2014
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Cho, Sunghyun, Wang, Yan, Wang, Jue
• Primary Examiner(s): Kassa, Yosef
• Application Number:
  US14/265,012
• Publication Number:
  US 20150310295A1
• Time in Patent Office: 749 Days
• Field of Search: 382/155, 382/159, 382/260, 382/263, 382/264, 382/274, 382/275, 358/1.2, 358/3.26, 358/3.27
• US Class Current: 1/1
• CPC Class Codes: G06F 18/211 Selection of the most significant subset of features
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/60 using machine learning, e.g. neural networks
• CPC Class Codes: G06V 10/56 relating to colour

Abstract

Methods for enhancing images with increased efficiency include using a discriminative index tree to expedite image optimization processes. The discriminative index tree indexes patch-based image priors for modifying an image by using classifiers determined by exploiting a structure of the patch-based image priors. The discriminative index tree quickly and efficiently parses a space of patch-based image patches to determine approximate dominant patch-based image priors for the space of image patches. To further improve the efficiency of the discriminative index tree, one or more embodiments can limit a number of potential patch-based image priors from which a dominant patch-based image prior is selected.

Claims

1. A method of generating an index tree for identifying patch-based image priors for patch-based image enhancement, comprising:

determining a plurality of patch-based image priors from a plurality of natural image training patches;

calculating, by at least one processor, a plurality of informed candidate linear classifiers based on a structure of the plurality of patch-based image priors;

selecting a linear classifier from the plurality of informed candidate linear classifiers for each non-leaf node of a discriminative index tree; and

generating, by the at least one processor, an enhanced output digital image by applying the discriminative index tree to an input digital image.

2. The method as recited in claim 1, wherein determining the plurality of patch-based image priors from the plurality of natural image training patches comprises learning a Gaussian mixture model over pixels of the natural image training patches to generate a plurality of Gaussians.

3. The method as recited in claim 2, further comprising identifying a subset of prominent Gaussians from the plurality of Gaussians.

4. The method as recited in claim 3, wherein calculating the plurality of informed candidate linear classifiers based on the structure of the plurality of patch-based image priors comprises generating informed candidate linear classifiers based on relationships between eigenvectors associated with randomly sampled pairs of Gaussians of the subset of prominent Gaussians.

5. The method as recited in claim 4, wherein selecting a linear classifier from the plurality of informed candidate linear classifiers for each non-leaf node of the discriminative index tree comprises determining an informed candidate linear classifier that maximizes an information gain.

6. The method as recited in claim 5, wherein the information gain is based on ground truth labels identifying dominant Gaussians, the dominant Gaussians being determined using a Maximum A Posteriori estimation with a Gaussian likelihood function and the assumption of Gaussian noise.

7. The method as recited in claim 5, wherein determining the informed candidate linear classifier that maximizes the information gain further comprises determining which informed candidate linear classifier maximizes an objective function comprising the information gain and a balance factor.

8. The method as recited in claim 4, wherein generating informed candidate linear classifiers based on relationships between eigenvectors associated with randomly sampled pairs of Gaussians of the subset of prominent Gaussians comprises sampling one principal direction from the eigenvectors of a covariance matrix of each Gaussian of a randomly sampled pair of Gaussians.

9. The method as recited in claim 8, further comprising:

determining a vector extending from a peak of a first principal direction of a first Gaussian of a randomly sampled pair of Gaussians to a peak in a first principal direction of a second Gaussian of the randomly sampled pair of Gaussians; and

calculating a first informed candidate linear classifier from a normal of the vector through a shared center of the first Gaussian and the second Gaussian.

10. The method as recited in claim 9, further comprising:

determining a second informed candidate linear classifier based on a vector extending from the peak of the first principal direction of the first Gaussian to a peak in a second principal direction of the second Gaussian;

determining a third informed candidate linear classifier based on a vector extending from a peak of a second principal direction of the first Gaussian to the peak in the first principal direction of the second Gaussian; and

determining a fourth informed candidate linear classifier based on a vector extending from the peak of the second principal direction of the first Gaussian to the peak in the second principal direction of the second Gaussian.

11. A method of patch-based image enhancement, comprising:

identifying a plurality of image patches associated with an input digital image to be enhanced;

determining, by at least one processor, an approximate dominant patch-based image prior for an image patch from a plurality of image patches associated with the input digital image using an index tree indexing a plurality of potential dominant patch-based image priors;

modifying, by the at least one processor, the input digital image by adjusting the image patch from the plurality of image patches using the determined approximate dominant patch-based image prior; and

generating, by the at least one processor, an enhanced output digital image comprising the adjusted image patch.

12. The method as recited in claim 11, wherein determining an approximate dominant patch-based image prior for the particular image patch using the index tree comprises:

traversing, for the image patch from the plurality of image patches, a plurality of non-leaf nodes each including a linear classifier to arrive at a leaf node;

based on a path taken traversing the plurality of non-leaf nodes, determining an expected probability that one or more patch-based image priors is a dominant patch-based image prior for the image patch from the plurality of image patches.

13. The method as recited in claim 12, further comprising:

determining an approximate dominant patch-based image prior for each image patch associated with the input digital image using the index tree;

applying a smoothing function to select the approximate dominant patch-based image prior for the image patch from the plurality of image patches based on the expected probability and approximate dominant patch-based image priors for neighboring patches.

14. The method as recited in claim 13, wherein generating the enhanced output digital image comprises deblurring the input digital image by modifying one or more image patches from the plurality of image patches associated with the input digital image.

15. The method as recited in claim 12, wherein traversing, for the image patch, the plurality of non-leaf nodes comprises:

determining a sign value for the linear classifier at a particular non-leaf node using the image patch from the plurality of image patches; and

traversing to a child node of the particular non-leaf node based on the determined sign value for the linear classifier.

16. The method as recited in claim 11, wherein modifying the input digital image using the determined approximate dominant patch-based image prior comprises applying a Wiener filter based on the determined approximate dominant patch-based image prior to the image patch from the plurality of image patches.

17. The method as recited in claim 11, further comprising:

modifying the image in a plurality of iterations; and

using a new index tree indexing a plurality of potential dominant patch-based image priors for each of the plurality of iterations to determine an approximate dominant patch-based image prior for the image patch from the plurality of image patches.

18. A system for patch-based image enhancement, comprising:

at least one processor;

at least one non-transitory computer readable storage medium storing instructions thereon, that, when executed by the at least one processor, cause the system to;

identify a plurality of image patches associated with an input digital image to be enhanced;

determine an approximate dominant patch-based image prior for an image patch from a plurality of image patches associated with the input digital image using an index tree indexing a plurality of potential dominant patch-based image priors;

modify the input digital image by adjusting the image patch from the plurality of image patches using the determined approximate dominant patch-based image prior; and

generate an enhanced output digital image comprising the adjusted image patch.

19. The system as recited in claim 18, wherein determining an approximate dominant patch-based image prior for the image patch from the plurality of image patches using the index tree comprises:

traversing, for the particular image patch, a plurality of non-leaf nodes each including a linear classifier to arrive at a leaf node;

based on a path taken traversing the plurality of non-leaf nodes, determining an expected probability that one or more patch-based image priors is a dominant patch-based image prior for the image patch from the plurality of image patches.

20. The system as recited in claim 19, further comprising instructions that when that when executed by the at least one processor, cause the system to:

determine an approximate dominant patch-based image prior for each image patch associated with the input digital image using the index tree;

apply a smoothing function to select the approximate dominant patch-based image prior for the image patch from the plurality of image patches based on the expected probability and approximate dominant patch-based image priors for neighboring patches.