Removing motion blur from unaligned multiple blurred images

• US 8,411,980 B1
• Filed: 02/24/2011
• Issued: 04/02/2013
• Est. Priority Date: 11/12/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Wang, Jue, Cho, Sunghyun, Lee, Seungyong
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US13/034,461
• Time in Patent Office: 768 Days
• Field of Search: 382260-269, 382/199, 382/274, 382254-255, 382128-132, 382/240, 382/162, 382/167, 382298-300, 3482084-2086, 348/208.99, 348/218.1, 348/246, 348/252, 348/584, 348/294, 348/242, 358/3.06, 358 326- 327
• US Class Current: 382/254
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/003 Deblurring; Sharpening vibration or motion blur correction for cameras comprising an electronic image sensor H04N23/682
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening

Abstract

Various embodiments of methods and apparatus for removing motion blur are disclosed. A homography is estimated for each of a plurality of images. Two or more images of the plurality are unaligned. An estimated latent image for the plurality of images selected for recovery of the latent image is deconvolved. A point spread function for the each of the plurality of images is calculated. The homography for the each of the plurality of images, the estimated latent image, and the point spread function for the each of the plurality of images are iteratively updated until the homography for each of the plurality of images stabilizes. The iteratively updating includes updating one of the homography for the each of the plurality of images, the estimated latent image, and the point spread function for the each of the plurality of images while holding the remainder constant.

Claims

1. A computer-implemented method, comprising:

estimating a respective homography for each of a plurality of images selected for recovery of a latent image of the plurality of images, wherein two or more images of the plurality are unaligned;

deconvolving an estimated latent image for the plurality of images selected for recovery of the latent image;

calculating a respective point spread function for each of the plurality of images;

iteratively updating a set comprising a plurality of members, wherein the plurality of members comprise the respective homography for each of the plurality of images, the estimated latent image, and the respective point spread function for each of the plurality of images, wherein the iteratively updating comprises updating one of the plurality of members while holding a remainder of the plurality of members of the set constant, and

the iteratively updating is performed until the respective homography for each of the plurality of images stabilizes.

2. The computer-implemented method of claim 1, wherein the calculating the point spread function further comprises estimating a point spread function based on a static homography assumption.

3. The computer-implemented method of claim 1, wherein the updating the homography further comprises:

calculating a homography update using a fixed blur kernel and a fixed estimate of the latent image, and

multiplying the homography by the homography update.

4. The computer-implemented method of claim 3, wherein the calculating the homography update further comprises minimizing an energy function of the homography update using an image registration method.

5. The computer-implemented method of claim 3, further comprising calculating a magnitude of a homography update by subtracting the homography from the product of the homography and the homography update.

6. The computer-implemented method of claim 1, wherein the updating the latent image further comprises optimizing an energy function of a latent image while fixing the blur kernel and the homography.

7. The computer-implemented method of claim 6, wherein the optimizing the energy function further comprises optimizing the energy function using a hyper-laplacian image prior.

8. A non-transitory computer-readable storage medium storing program instructions, wherein the program instructions are computer-executable to implement:

estimating a respective homography for each of a plurality of images selected for recovery of a latent image of the plurality of images, wherein two or more images of the plurality are unaligned;

deconvolving an estimated latent image for the plurality of images selected for recovery of the latent image;

calculating a respective point spread function for each of the plurality of images;

iteratively updating a set comprising a plurality of members, wherein the plurality of members comprise the respective homography for each of the plurality of images, the estimated latent image, and the respective point spread function for each of the plurality of images, wherein the iteratively updating comprises updating one of the plurality of members while holding a remainder the plurality of members of the set constant, and

the iteratively updating is performed until the respective homography for each of the plurality of images stabilizes.

9. The non-transitory computer-readable storage medium of claim 8, wherein the program instructions computer-executable to implement the calculating the point spread function further comprise program instructions computer-executable to implement estimating a point spread function based on a static homography assumption.

10. The non-transitory computer-readable storage medium of claim 8, wherein the program instructions computer-executable to implement the updating the homography further comprise program instructions computer-executable to implement:

calculating a homography update using a fixed blur kernel and a fixed estimate of the latent image, and

multiplying the homography by the homography update.

11. The non-transitory computer-readable storage medium of claim 10, wherein the program instructions computer-executable to implement the calculating the homography update further comprise program instructions computer-executable to implement minimizing an energy function of the homography update using an image registration method.

12. The non-transitory computer-readable storage medium of claim 10, further comprising instructions computer-executable to implement calculating a magnitude of a homography update by subtracting the homography from the product of the homography and the homography update.

13. The non-transitory computer-readable storage medium of claim 8, wherein the program instructions computer-executable to implement updating the latent image further comprise program instructions computer-executable to implement optimizing an energy function of a latent image while fixing the blur kernel and the homography.

14. The non-transitory computer-readable storage medium of claim 13, wherein the program instructions computer-executable to implement the optimizing the energy function further comprise program instructions computer-executable to implement optimizing the energy function using a hyper-laplacian image prior.

15. A system, comprising:

at least one processor; and

a memory comprising program instructions, wherein the program instructions are executable by the at least one processor to;

estimate a respective homography for each of a plurality of images selected for recovery of a latent image of the plurality of images, wherein two or more images of the plurality are unaligned;

deconvolve an estimated latent image for the plurality of images selected for recovery of the latent image;

calculate a respective point spread function for each of the plurality of images;

iteratively update a set comprising a plurality of members, wherein the plurality of members comprise the respective homography for each of the plurality of images, the estimated latent image, and the respective point spread function for each of the plurality of images, wherein

the program instructions executable by the at least one processor to iteratively update further comprise program instructions executable by the at least one processor to update one of a the plurality of members while holding a remainder of the plurality of members of the set constant, and

the program instructions executable by the at least one processor to iteratively update further comprise program instructions executable by the at least one processor iteratively update until the respective homography for each of the plurality of images stabilizes.

16. The system of claim 15, wherein the program instructions executable by the at least one processor to implement the calculating the point spread function further comprise program instructions executable by the at least one processor to estimate a point spread function based on a static homography assumption.

17. The system of claim 15, wherein the program instructions executable by the at least one processor to update the homography further comprise program instructions executable by the at least one processor to:

calculate a homography update using a fixed blur kernel and a fixed estimate of the latent image, and

multiply the homography by the homography update.

18. The system of claim 17, wherein the program instructions executable by the at least one processor to calculate the homography update further comprise program instructions executable by the at least one processor to minimize an energy function of the homography update using an image registration method.

19. The system of claim 17, further comprising program instructions executable by the at least one processor to calculate a magnitude of a homography update by subtracting the homography from the product of the homography and the homography update.

20. The system of claim 15, wherein the program instructions executable by the at least one processor to update the latent image further comprise program instructions executable by the at least one processor to optimize an energy function of a latent image while fixing the blur kernel and the homography.