Methods and Apparatus for De-blurring Images Using Lucky Frames

• US 20120121202A1
• Filed: 11/30/2010
• Published: 05/17/2012
• Est. Priority Date: 11/12/2010
• Status: Active Grant

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Simons, David P., Wang, Jue, Lee, Seungyong, Cho, Sunghyun
• Granted Patent: US 8,532,421 B2
• Time in Patent Office: Days
• US Class Current: 382/255
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/6845 by combination of a plurality of images sequentially taken

Abstract

A sharp frame and a blurred frame are detected from among a plurality of frames. A blur kernel is estimated. The blur kernel represents a motion-transform between the sharp frame and the blurred frame. Using the blur kernel, a static region measure for the sharp frame and the blurred frame is estimated. A de-blurred frame is generated by replacing one or more pixels of the blurred frame as indicated by the static region measure.

Claims

1. A computer-implemented method, comprising:

detecting a sharp frame and a blurred frame from among a plurality of frames;

estimating a blur kernel, wherein the blur kernel represents a motion-transform between one or more common regions of the sharp frame and the blurred frame;

identifying, using the blur kernel, a static region measure for the sharp frame and the blurred frame; and

generating a de-blurred frame by replacing one or more pixels of the blurred frame as indicated by the static region measure.

2. The method of claim 1, wherein the replacing the one or more pixels of the blurred frame as indicated by the static region measure further comprises:

calculating a replacement pixel for one of the one or more pixels as a composite of a first pixel from the sharp frame and a second pixel from a second sharp frame, wherein the sharp frame is a frame occurring prior to the blurred frame in the video stream comprising the plurality of frames, and the second sharp frame is a frame occurring after the blurred frame in the video stream comprising the plurality of frames.

3. The method of claim 1, wherein the identifying, using the blur kernel, the static region measure for the sharp frame and the blurred frame further comprises identifying areas of motion between the sharp frame and the blurred frame by comparing the blurred frame to a simulated blurred frame generated as a product of the sharp frame and the blur kernel.

4. The method of claim 1, wherein the detecting the sharp frame further comprises

detecting a set of feature points in a first frame;

tracking the set of feature points in a second frame;

calculating a motion of the set of feature points between the first frame and the second frame; and

identifying the second frame as the sharp frame if the motion of the set of feature points between the first frame and the second frame is less than a threshold value.

5. The method of claim 4, wherein the calculating the motion of the set of feature points between the first frame and the second frame further comprises estimating a homography based on a displacement of the set of feature points between the first frame and the second frame, and wherein the blur kernel is estimated based on an alignment between the sharp frame and the blurred frame based on the homography.

6. The method of claim 4, wherein the identifying the static region measure further comprises calculating a value of the static region measure for a pixel location such that the value is inversely correlated to a change of a status of the pixel location.

7. The method of claim 1, wherein the identifying the static region measure further comprises calculating a value of an energy minimization function comprising a regularization term and a structural similarity term.

8. A system, comprising:

one or more processors; and

memory coupled to the one or more processors, wherein the memory stores program instructions executable by the one or more processors to perform;

detecting a sharp frame and a blurred frame from among a plurality of frames;

estimating a blur kernel, wherein the blur kernel represents a motion-transform between one or more common regions of the sharp frame and the blurred frame;

identifying, using the blur kernel, a static region measure for the sharp frame and the blurred frame; and

generating a de-blurred frame by replacing one or more pixels of the blurred frame as indicated by the static region measure.

9. The system of claim 8, wherein the program instructions executable by the one or more processors to perform the replacing the one or more pixels of the blurred frame as indicated by the static region measure further comprise program instructions executable by the one or more processors to perform:

calculating a replacement pixel for one of the one or more pixels as a composite of a first pixel from the sharp frame and a second pixel from a second sharp frame, wherein the sharp frame is a frame occurring prior to the blurred frame in the video stream comprising the plurality of frames, and the second sharp frame is a frame occurring after the blurred frame in the video stream comprising the plurality of frames.

10. The system of claim 8, wherein the program instructions executable by the one or more processors to perform the identifying, using the blur kernel, the static region measure for the sharp frame and the blurred frame further comprise program instructions executable by the one or more processors to perform identifying areas of motion between the sharp frame and the blurred frame by comparing the blurred frame to a simulated blurred frame generated as a product of the sharp frame and the blur kernel.

11. The system of claim 8, wherein the program instructions executable by the one or more processors to perform the detecting the sharp frame further comprise program instructions executable by the one or more processors to perform:

detecting a set of feature points in a first frame;

tracking the set of feature points in a second frame;

calculating a motion of the set of feature points between the first frame and the second frame; and

identifying the second frame as the sharp frame if the motion of the set of feature points between the first frame and the second frame is less than a threshold value.

12. The system of claim 11, wherein the program instructions executable by the one or more processors to perform the calculating the motion of the set of feature points between the first frame and the second frame further comprise program instructions executable by the one or more processors to perform estimating a homography based on a displacement of the set of feature points between the first frame and the second frame, and wherein the blur kernel is estimated based on an alignment between the sharp frame and the blurred frame based on the homography.

13. The system of claim 8, wherein the program instructions executable by the one or more processors to perform the identifying the static region measure further comprise program instructions executable by the one or more processors to perform calculating a value of the static region measure for a pixel location such that the value is inversely correlated to a change of a status of the pixel location.

14. The system of claim 8, wherein the program instructions executable by the one or more processors to perform the identifying the static region measure further comprise program instructions executable by the one or more processors to perform calculating a value of an energy minimization function comprising a regularization term and a structural similarity term.

15. A non-transitory computer-readable storage medium storing program instructions computer-executable to perform:

detecting a sharp frame and a blurred frame from among a plurality of frames;

estimating a blur kernel, wherein the blur kernel represents a motion-transform between one or more common regions of the sharp frame and the blurred frame;

identifying, using the blur kernel, a static region measure for the sharp frame and the blurred frame; and

generating a de-blurred frame by replacing one or more pixels of the blurred frame as indicated by the static region measure.

16. The non-transitory computer-readable storage medium of claim 15, wherein the program instructions computer-executable to perform the replacing the one or more pixels of the blurred frame as indicated by the static region measure further comprise program instructions computer-executable to perform:

calculating a replacement pixel for one of the one or more pixels as a composite of a first pixel from the sharp frame and a second pixel from a second sharp frame, wherein the sharp frame is a frame occurring prior to the blurred frame in the video stream comprising the plurality of frames, and the second sharp frame is a frame occurring after the blurred frame in the video stream comprising the plurality of frames.

17. The non-transitory computer-readable storage medium of claim 15, wherein the program instructions computer-executable to perform the identifying, using the blur kernel, the static region measure for the sharp frame and the blurred frame further comprise program instructions computer-executable to perform identifying areas of motion between the sharp frame and the blurred frame by comparing the blurred frame to a simulated blurred frame generated as a product of the sharp frame and the blur kernel.

18. The non-transitory computer-readable storage medium of claim 15, wherein the program instructions computer-executable to perform the detecting the sharp frame further comprise program instructions computer-executable to perform:

detecting a set of feature points in a first frame;

tracking the set of feature points in a second frame;

calculating a motion of the set of feature points between the first frame and the second frame; and

identifying the second frame as the sharp frame if the motion of the set of feature points between the first frame and the second frame is less than a threshold value.

19. The non-transitory computer-readable storage medium of claim 18, wherein the program instructions computer-executable to perform the calculating the motion of the set of feature points between the first frame and the second frame further comprise program instructions computer-executable to perform estimating a homography based on a displacement of the set of feature points between the first frame and the second frame, and wherein the blur kernel is estimated based on an alignment between the sharp frame and the blurred frame based on the homography.

20. The non-transitory computer-readable storage medium of claim 18, wherein the program instructions computer-executable to perform the identifying the static region measure further comprise program instructions computer-executable to perform calculating a value of the static region measure for a pixel location such that the value is inversely correlated to a change of a status of the pixel location.