Method and system of generating a high-resolution image from a set of low-resolution images

• US 20050244081A1
• Filed: 04/28/2004
• Published: 11/03/2005
• Est. Priority Date: 04/28/2004
• Status: Active Grant

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Clarke, Ian, Zhou, Hui, Sellers, Graham
• Granted Patent: US 7,352,919 B2
• Time in Patent Office: Days
• US Class Current: 382/299
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Abstract

A method of generating a high-resolution image from a set of source low-resolution images includes estimating a high-resolution image based on the set of source low-resolution images. The estimated high-resolution image is then transformed into a set of estimated low-resolution images. The set of source low-resolution images is compared with the set of estimated low-resolution images to generate a set of low-resolution errors. The set of low-resolution errors is transfomed into a set of high-resolution errors. A high-resolution error image based on the set of high-resolution errors is generated. The high-resolution error image is combined with the estimated high resolution image to yield an updated estimated high-resolution. The above steps are repeated until the updated estimated high-resolution image is of a desired quality.

Claims

1. A method of generating a high-resolution image from a set of source low-resolution images comprising:

(a) estimating a high-resolution image based on the set of source low-resolution images;

(b) transforming the estimated high-resolution image into a set of estimated low-resolution images;

(c) comparing the set of source low-resolution images with the set of estimated low-resolution images to generate a set of low-resolution errors;

(d) transforming the set of low-resolution errors into a set of high-resolution errors;

(e) generating a high-resolution error image based on the set of high-resolution errors;

(f) combining the high-resolution error image with the estimated high resolution image to yield an updated estimated high-resolution image; and

(g) repeating steps (b) to (f) until the updated estimated high-resolution image is of a desired quality.

2. The method of claim 1 wherein during said generating the high-resolution errors are averaged to generate said high-resolution error image.

3. The method of claim 1 wherein the medians of corresponding pixels in the high-resolution errors is determined and the medians used to generate said high-resolution error image.

4. The method of claim 2 wherein corresponding pixels in the high-resolution errors are sorted, selected sorted pixels are discarded and the remaining corresponding pixels averaged to generate said high-resolution error image.

5. The method of claim 4 wherein the selected sorted pixels are the highest and lowest sorted pixels.

6. The method of claim 1 wherein during said estimating, the source low-resolution images are registered, averaged and then scaled to high-resolution space thereby to yield said estimated high-resolution image.

7. The method of claim 6 wherein scaling is performed using a bilinear stretching method.

8. The method of claim 6 wherein during said registration, a transform matrix for each source low-resolution image in said set is generated that projects said source low-resolution image onto a designated source low-resolution image of said set.

9. The method of claim 8 wherein said transforming at step (b) comprises:

transforming the estimated high-resolution image using said transform matrices; and

downsampling the transformed estimated high-resolution images thereby to yield said set of estimated low-resolution images.

10. The method of claim 9 wherein said transforming at step (d) comprises:

upsampling the low-resolution errors; and

transforming the upsampled low-resolution errors using inverses of said transform matrices thereby to yield said set of high-resolution errors.

11. The method of claim 10 wherein during said generating, the high-resolution errors are averaged to generate said high-resolution error image.

12. The method of claim 10 wherein the medians of corresponding pixels in the high-resolution errors is determined and the medians used to generate said high-resolution error image.

13. The method of claim 10 wherein corresponding pixels in the high-resolution errors are sorted, selected sorted pixels are discarded and the remaining corresponding pixels averaged to generate said high-resolution error image.

14. The method of claim 13 wherein the selected sorted pixels are the highest and lowest sorted pixels.

15. The method of claim 6 wherein during registering of said low-resolution images, matching corners in adjoining low-resolution images I and I′

are determined and a transform between each adjoining pair of low-resolution images based on said matching corners is estimated.

16. The method of claim 15 wherein registering each pair of adjoining low-resolution images I and I′

comprises;

extracting features in each of said low-resolution images I and I′

corresponding to high curvature points therein;

determining corners adjacent said features; and

matching the corners of low-resolution image I to corresponding corners of low-resolution image I′

thereby to register said low-resolution images I and I′

.

17. The method of claim 16 wherein during said determining, corners within a neighbourhood surrounding said features are detected.

18. The method of claim 17 wherein said determining is performed until a threshold number of corners is detected.

19. The method of claim 18 wherein during said matching, each detected corner in low-resolution image I′

is compared with each detected corner in low-resolution image I to determine matching corners in said low-resolution images I and I′

.

20. The method of claim 19 wherein said comparing includes the steps of:

determining the correlation between each detected corner in low-resolution image I′

with each detected corner in low-resolution image I to yield a list of corners in which each corner in low-resolution image I has a set of candidate matching corners in low-resolution image I′

;

measuring the likelihood that each of the candidate matching corners in said set corresponds to the associated corner in low-resolution image I; and

selecting one of the candidate matching corners in said set.

21. The method of claim 20 wherein during said correlation determining, a normalized cross correlation is used to calculate a correlation score between each detected corner in low-resolution image I′

with each detected corner in low-resolution image I, correlation scores above a threshold level signifying a candidate matching corner.

22. The method of claim 21 wherein said correlation determining is performed until a threshold number of candidate matching corners is determined thereby to form said set.

23. The method of claim 22 wherein during said measuring, a score of matching is used to measure the likelihood that each of the candidate matching corners in said set corresponds to the associated corner in low-resolution image I based on other matching corner pairs within a neighbourhood surrounding the corners being matched.

24. A method of creating a high-resolution image comprising:

estimating an initial high-resolution image by interpolating the average of aligned source low-resolution images;

generating downsampled estimated low-resolution images from said initial high-resolution image;

comparing the estimated low-resolution images with the source low-resolution images and generating feedback error data; and

using the feedback error data to update the estimated high-resolution image, wherein the estimating, generating and comparing steps are iteratively performed until the estimated high-resolution image is of a desired resolution.

25. The method of claim 24 wherein the source low-resolution images are aligned using a matching features-based technique.

26. The method of claim 25 wherein said feedback error data is upsampled and averaged prior to being used to update the estimated high-resolution image.

27. The method of claim 26 wherein a modified averaging technique is used to modify the upsampled feedback error data prior to being used to update the estimated high-resolution image.

28. A digital image editing tool for generating a high-resolution image from a set of source low-resolution images comprising:

means for estimating a high-resolution image based on the set of source low-resolution images;

first means for transforming the estimated high-resolution image into a set of estimated low-resolution images;

means for comparing the set of source low-resolution images with the set of estimated low-resolution images to generate a set of low-resolution errors;

second means for transforming the set of low-resolution errors into a set of high-resolution errors;

means for generating a high-resolution error image based on said set of high-resolution errors; and

means for combining the high-resolution error image with the estimated high resolution image to yield an updated estimated high-resolution image, wherein said first transforming means, comparing means, second transforming means, generating means and combining means being operated iteratively until the updated estimated high-resolution image is of a desired quality.

29. A digital editing tool according to claim 28 wherein said generating means averages the high-resolution errors to generate said high-resolution error image.

30. A digital editing tool according to claim 28 wherein said generating means determines the medians of corresponding pixels in the high-resolution errors and uses the medians to generate said high-resolution error image.

31. A digital editing tool according to claim 29 wherein said generating means sorts corresponding pixels in the high-resolution errors, discards selected sorted pixels and averages the remaining corresponding pixels to generate said high-resolution error image.

32. A digital editing tool according to claim 28 wherein said estimating means registers, averages and then scales the set of source low-resolution images to high resolution space thereby to yield said estimated high-resolution image.

33. A digital editing tool according to claim 32 wherein scaling is performed using a bilinear stretching method.

34. A computer readable medium embodying a computer program for generating a high-resolution image from a set of source low-resolution images comprising:

computer program code for estimating a high-resolution image based on the set of source low-resolution images;

computer program code for first transforming the estimated high-resolution image into a set of estimated low-resolution images;

computer program code for comparing the set of source low-resolution images with the set of estimated low-resolution images to generate a set of low-resolution errors;

computer program code for second transforming the set of low-resolution errors into a set of high-resolution errors;

computer program code for generating a high-resolution error image based on said set of high-resolution errors;

computer program code for combining the high-resolution error image with the estimated high resolution image to yield an updated estimated high-resolution image; and

computer program code for iteratively conditioning the first transforming, comparing, second transforming, generating and combining computer program code until the updated estimated high-resolution image is of a desired quality.