Constrained image deblurring for imaging devices with motion sensing

• US 20070009169A1
• Filed: 07/08/2005
• Published: 01/11/2007
• Est. Priority Date: 07/08/2005
• Status: Abandoned Application

• Current Assignee: Seiko Epson Corporation (Seiko Group)
• Original Assignee: Seiko Epson Corporation (Seiko Group)
• Inventors: Bhattacharjya, Anoop
• Application Number:
  US11/177,804
• Publication Number:
  US 20070009169A1
• Time in Patent Office: Days
• US Class Current: 382/255
• CPC Class Codes: G06T 2207/20056 Discrete and fast Fourier transform, [DFT, FFT]
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/10 using non-spatial domain filtering
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/684 performed by controlling the image sensor readout, e.g. by controlling the integration time

Abstract

Systems and methods are disclosed for deblurring a captured image using parametric deconvolution, instead of a blind, non-parametric deconvolution, by incorporating physical constraints derived from sensor inputs, such as a motion sensor, into the deconvolution process to constrain modifications to the point spread function. In an embodiment, a captured image is deblurred using a point spread function obtained from the cross-validation of information across a plurality of image blocks taken from the capture image, which image blocks are deconvolved using parametric deconvolution to constrain modifications to the point spread function.

Claims

1. A method for deblurring a captured image taken by an imaging device comprising an imaging sensor array for capturing the image during an exposure time and at least one motion sensor, the method comprising the steps of:

[a] obtaining the captured image;

[b] obtaining a set of motion parameters from the motion sensor related to the motion of the imaging sensor array during the exposure time and wherein at least one of the motion parameters within the set of motion parameters possesses associated interval values such that a family of motion paths may be defined by the set of motion parameters and associated interval values;

[c] obtaining an estimated point spread function that comprises the convolution of an optical point spread function of the imaging device and a motion path selected from the family of motion paths defined by the set of motion parameters and associated interval values;

[d] selecting an estimated deblurred image;

[e] computing a new estimated point spread function based upon the captured image, the estimated deblurred image, and the estimated point spread function;

[f] performing an optimization over the set of motion parameters and associated interval values to find a set of optimized parameter values within the set of motion parameters and associated interval values that yield an optimized point spread function that best fits the new estimated point spread function; and

[g] using the optimized point spread function to compute a new estimated deblurred image.

2. The method claim 1 further comprising the step of:

[h] adjusting pixel values within the new estimated deblurred image to keep the pixel values within a specified value range.

3. The method claim 2 further comprising the steps of:

selecting the optimized point spread function as the estimated point spread function;

selecting the new estimated deblurred image as the estimated deblurred image; and

iterating steps [e] through [h].

4. The method of claim 3 wherein the steps are iterated a set number of times.

5. The method claim 1 wherein the step of [f] performing an optimization over the set of motion parameters and associated interval values to find a set of optimized parameter values within the set of motion parameters and associated interval values that yield an optimized point spread function that best fits the new estimated point spread function further comprises the step of:

[f′

] mapping a motion parameter, from the set of motion parameters, and its associated interval values to an unconstrained variable to ensure that its optimized parameter value obtained from the optimization will produce a value that falls within the motion parameter'"'"'s associated interval values.

6. The method claim 1 wherein the captured image is a portion of a larger captured image.

7. The method of claim 6 further comprising the steps of:

obtaining a plurality of sets of optimized parameter values from a plurality of captured images that are portions of the larger captured image;

obtaining a best set of optimized parameters from the plurality of sets of optimized parameters; and

deblurring the larger captured image using the best set of optimized parameters.

8. The method of claim 1 wherein the associated interval values represent a measurement sensitivity value.

9. A computer readable medium comprising a set of instructions for performing the method of claim 1.

10. An imaging device comprising:

an imaging sensor array for capturing an image during an exposure time;

a motion sensor that measures a set of motion parameters related to the imaging sensor array'"'"'s motion during the exposure time;

a processor communicatively coupled to the imaging sensor array and adapted to perform the steps comprising;

[a] obtaining a captured image;

[b] obtaining a set of motion parameters from the motion sensor related to the motion of the imaging sensor array during the exposure time and wherein at least one of the motion parameters within the set of motion parameters possesses associated interval values such that a family of motion paths may be defined by the set of motion parameters and associated interval values;

[c] obtaining an estimated point spread function that comprises the convolution of an optical point spread function of the imaging device and a motion path selected from the family of motion paths defined by the set of motion parameters and associated interval values;

[d] obtaining an estimated deblurred image;

[e] computing a new estimated point spread function based upon the captured image, the estimated deblurred image, and the estimated point spread function;

[f] performing an optimization over the set of motion parameters and associated interval values to find a set of optimized parameter values within the set of motion parameters and associated interval values that yield an optimized point spread function that best fits the new estimated point spread function; and

[g] using the optimized point spread function to compute a new estimated deblurred image.

11. The imaging device of claim 10 wherein the processor is further adapted to perform the step comprising:

[h] adjusting pixel values within the new estimated deblurred image to keep the pixel values within a specified value range.

12. The imaging device of claim 11 wherein the processor is further adapted to perform the steps comprising:

selecting the optimized point spread function as the estimated point spread function;

selecting the new estimated deblurred image as the estimated deblurred image; and

iterating steps [e] through [h].

13. The imaging device of claim 12 wherein the steps are iterated a set number of times.

14. The imaging device of claim 10 wherein the step of [f] performing an optimization over the set of motion parameters and associated interval values to find a set of optimized parameter values within the set of motion parameters and associated interval values that yield an optimized point spread function that best fits the new estimated point spread function further comprises the step of:

[f′

] mapping a motion parameter, from the set of motion parameters, and its associated interval values to an unconstrained variable to ensure that its optimized parameter value obtained from the optimization will produce a value that falls within the motion parameter'"'"'s associated interval values.

15. The imaging device of claim 10 wherein the captured image is a portion of a larger captured image.

16. The imaging device of claim 15 wherein the processor is further adapted to perform the steps comprising:

obtaining a plurality of sets of optimized parameter values from a plurality of captured images that are portions of the larger captured image;

obtaining a best set of optimized parameters from the plurality of sets of optimized parameters; and

deblurring the larger captured image using the best set of optimized parameters.

17. A method for deblurring an image comprising:

[a] selecting a plurality of image blocks from a captured image, wherein the captured image was obtained from an imaging device with at least one motion sensor;

[b] estimating a point spread function within each of the plurality of image blocks, wherein each point spread function is consistent with a set of motion parameter values taken by the motion sensor during the capturing of the captured image;

[c] employing a deconvolution algorithm to deblur each of the plurality of image blocks wherein a modification to any of the point spread functions of the plurality of image blocks is consistent with the set of motion parameter values taken by the motion sensor during the capturing of the captured image;

[d] selecting a best point spread function from the point spread functions of the plurality of image blocks; and

[e] deblurring the captured image using the best point spread function.

18. The method of claim 17 wherein at least one of the values in the set of motion parameter values comprises a measurement sensitivity value.

19. The method of claim 17 wherein the step of [d] selecting a best point spread function from the point spread functions of the plurality of image blocks comprises using a cross-validation procedure.

20. A computer readable medium comprising a set of instructions for performing the method of claim 17.