Multi-stage image super-resolution with reference merging using personalized dictionaries

• US 20170186135A1
• Filed: 12/26/2015
• Published: 06/29/2017
• Est. Priority Date: 12/26/2015
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Chinya, Gautham N., Lin, Chit Kwan, Comiter, Marcus Z., Gulati, Harnek
• Granted Patent: US 9,697,584 B1
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/21355 nonlinear criteria, e.g. embedding a manifold in a Euclidean space
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G06V 10/7715 Feature extraction, e.g. by transforming the feature space, e.g. multi-dimensional scaling [MDS]; Mappings, e.g. subspace methods
• CPC Class Codes: G06V 40/171 Local features and components; Facial parts eye characteristics G06V40/18; Occluding parts, e.g. glasses; Geometrical relationships

Abstract

An apparatus for multi-stage super-resolution is described herein. The apparatus includes a personalized dictionary, a plurality of super-resolution stages, and a reference merger. Each of the plurality of super-resolution stages correspond to at least one personalized dictionary, and the personalized dictionary is applied to an input image that is sparse-coded to generate a reconstructed image. The reference merger is to merge the reconstructed image and the input image to generate a super-resolved image for each stage.

Claims

1. An apparatus, comprising:

a plurality of personalized dictionaries;

a plurality of super-resolution stages, wherein each of the plurality of super-resolution stages correspond to at least one personalized dictionary and the personalized dictionary is applied to an input image that is sparse-coded to generate a reconstructed image; and

a reference merger to merge the reconstructed image and the input image to generate a super-resolved image for each stage.

2. The apparatus of claim 1, wherein each stage of the plurality of super-resolution stages adds information to the input image lost to a distinct degradation operator.

3. The apparatus of claim 1, wherein each stage of the plurality of super-resolution stages uses at least one personalized dictionary in the plurality of personalized dictionaries.

4. The apparatus of claim 1, wherein the input image is sparse-coded using a second, low-quality personalized dictionary.

5. The apparatus of claim 1, wherein the plurality of personalized dictionaries are trained on images that are personalized to a user.

6. A method for multi-stage super-resolution, comprising:

generating a plurality of super-resolution stages, wherein each stage corresponds to a degradation operator and at least one personalized dictionary;

reconstructing an input image at each super-resolution stage, wherein the at least one personalized dictionary is applied to a sparse code to form a reconstructed image; and

applying reference merging to the reconstructed image and the input image to reduce error in the reconstructed image.

7. The method of claim 6, wherein a second personalized dictionary provides data to calculate a sparse code and reconstruct an image from the sparse code using the at least one personalized dictionary.

8. The method of claim 7, wherein the second personalized dictionary for each stage is generated by applying the degradation operator to atoms in the at least one dictionary for each stage.

9. The method of claim 6, wherein the at least one personalized dictionary systematically adds information lost due to the degradation operator to the input image to obtain the reconstructed image.

10. The method of claim 6, wherein reference merging selectively merges the reconstructed image with the input image.

11. The method of claim 6, wherein in response to a portion of the reconstructed image being unequal to a portion of the input image, the portion of the input image is used to merge pixels in the portion of the reconstructed image.

12. The method of claim 6, wherein reference merging comprises using a difference map to determine areas of error in the higher quality image.

13. The method of claim 6, wherein the personalized dictionary comprises atoms for a single user.

14. The method of claim 6, wherein the multi-stage super-resolution results in an increase of image resolution of at least 36 times an original image resolution.

15. The method of claim 6, wherein a learning mechanism is used to generate the personalized dictionary.

16. A tangible, non-transitory, computer-readable medium comprising instructions that, when executed by a processor, direct the processor to:

generate a plurality of super-resolution stages, wherein each stage corresponds to a degradation operator and at least one personalized dictionary;

reconstruct an input image at each super-resolution stage, wherein the at least one personalized dictionary is applied to a sparse code to form a reconstructed image; and

apply reference merging to the reconstructed image and the input image to eliminate error in the reconstructed image.

17. The computer readable medium of claim 16, wherein a second personalized dictionary provides data to calculate a sparse code and reconstruct an image from the sparse code using the at least one personalized dictionary.

19. The computer readable medium of claim 16, wherein the at least one personalized dictionary systematically adds information lost due to the degradation operator to the input image to obtain the reconstructed image.

20. The computer readable medium of claim 16, wherein reference merging selectively merges the reconstructed image with the input image.

21. A system, comprising:

an image capture mechanism;

a memory that is to store instructions and that is communicatively coupled to the image capture mechanism and the display; and

a processor communicatively coupled to the image capture mechanism, the display, and the memory, wherein when the processor is to execute the instructions, the processor is to;

form a plurality of super-resolution stages, wherein each stage of the plurality of super-resolution stages correspond to at least one personalized dictionary and the personalized dictionary is applied to an input image that is sparse-coded to generate a reconstructed image; and

a reference merger to merge the reconstructed image and the input image to generate a super-resolved image for each stage.

18. The computer readable medium of claim 22, wherein the second personalized dictionary for each stage is generated by applying the degradation operator to atoms in the at least one dictionary for each stage.

22. The system of claim 21, wherein each stage of the plurality of super-resolution stages adds information to the input image lost to a distinct degradation operator.

23. The system of claim 21, wherein the input image is sparse-coded using a second, low-quality personalized dictionary.

24. The system of claim 21, wherein the at least one personalized dictionary for each stage is trained on images that are personalized to a user.

25. The system of claim 21, wherein the at least one personalized dictionary for each stage comprises an overcomplete basis of atoms.