Still image stabilization suitable for compact camera environments

• US 8,068,140 B2
• Filed: 08/07/2006
• Issued: 11/29/2011
• Est. Priority Date: 08/07/2006
• Status: Expired due to Fees

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: Avago Technologies General IP PTE Limited (Broadcom, Inc.)
• Inventors: Helbing, Rene
• Primary Examiner(s): Chan, Jason
• Assistant Examiner(s): OSINSKI, MICHAEL S
• Application Number:
  US11/500,148
• Publication Number:
  US 20080030587A1
• Time in Patent Office: 1,940 Days
• Field of Search: 348/208.4, 348/208, 348/362, 348/229.1, 348/221.1, 348/230.1, 348/207.99, 34820899-20816, 348/222.1, 348294-308
• US Class Current: 348/208.4
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode

Abstract

Still image apparatus and methods that are suitable for compact camera environments are described. In another aspect, image segments are captured during corresponding exposure intervals of a frame exposure period. Respective motion sensor signals are generated during the respective exposure intervals of the image segments. One or more corresponding movement measures are determined from the respective motion sensor signals for each of the exposure intervals. The image segments are combined into an output image based on the one or more corresponding movement measures.

Claims

1. An apparatus, comprising:

an image sensor operable to capture image segments during corresponding exposure intervals of a frame exposure period;

a motion sensor operable to generate respective motion sensor signals during the respective exposure intervals of the image segments, wherein the motion sensor comprises image sensing elements operable to capture images of a scene; and

a processing system operable to determine from the respective motion sensor signals one or more corresponding movement measures for each of the exposure intervals and combine the image segments into an output image based on the one or more corresponding movement measures, wherein the processing system is operable to determine the movement measures from comparisons of ones of the images captured by the image sensing elements of the motion sensor;

wherein at least a portion of the image sensing elements are further operable to capture the images in accordance with a rolling shutter process that is determined by;

E_FP=e_R+M*T_R,

t_E,i=(i−

1)*T_R+t_0,FP, and

t_R,i=i*T_R+e_R+t_0,FP,

wherein e_R denotes a scanline exposure interval, E_FP denotes a frame exposure period, T_R denotes readout time for each row of pixels, M is the number of rows of pixels in an active area of the image sensor, exposure of row i begins at time t_E,i, t_0,FP is a time at which frame exposure period FP begins, and readout of row i occurs at time t_R,i.

2. The apparatus of claim 1, wherein the image sensor comprises an array of pixels arranged in rows and columns in a capture plane and the image sensor captures each of the image segments by reading out values of the pixels across a respective subset of the rows of pixels integrated over the corresponding exposure interval.

3. The apparatus of claim 2, further comprising an exposure controller operable to control the capture of the image segments by the image sensor in accordance with a rolling shutter capture process.

4. The apparatus of claim 2, wherein the processing system combines the image segments by deblurring each of the image segments based on the one or more corresponding movement measures and arranging the deblurred image segments along respective rows of the output image.

5. The apparatus of claim 1, wherein the image sensor comprises an array of pixels arranged in rows and columns in a capture plane and the image sensor captures each of the image segments by reading out values of the pixels across the capture plane integrated over the corresponding exposure interval.

6. The apparatus of claim 5, wherein the processing system combines the image segments by warping the image segments to a common reference frame based on the one or more corresponding movement measures and combining values of corresponding pixels in the warped image segments into corresponding values of pixels in the output image.

7. The apparatus of claim 1, further comprising a telephone subsystem and a housing containing the image sensor, the motion sensor, the processing system, and the telephone subsystem.

15. The apparatus of claim 1, wherein the image sensing elements of the motion sensor are implemented separately from the image sensor.

16. The apparatus of claim 1, wherein the image sensing elements of the motion sensor are implemented by a region of the image sensor.

8. A method, comprising:

capturing image segments during corresponding exposure intervals of a frame exposure period;

generating respective motion sensor signals during the respective exposure intervals of the image segments, wherein the generating comprises capturing images of a scene;

determining from the respective motion sensor signals one or more corresponding movement measures for each of the exposure intervals, wherein the determining comprises deriving the movement measures from comparisons of ones of the captured images; and

combining the image segments into an output image based on the one or more corresponding movement measures;

wherein the movement measures are determined based on;

$U=\left(\begin{array}{c}u\\ v\end{array}\right)=\left(\begin{array}{ccc}{a}_{x<br><br><br><br>0}& a_{x<br><br><br><br>1}& a_{x<br><br><br><br>2}\\ a_{y<br><br><br><br>0}& a_{y<br><br><br><br>1}& a_{y<br><br><br><br>2}\end{array}\right)·<br><br>\left(\begin{array}{c}x\\ y\\ z\end{array}\right)=A·<br><br>\left(\begin{array}{c}x\\ y\\ z\end{array}\right),$
I_r(P)=I_t(P−

U(P)),

A=(X^TX)<sup>−

1</sup>X^TU,

9. The method of claim 8, wherein the capturing comprises capturing each of the image segments by reading out values of pixels across a respective subset of rows of a two-dimensional array of pixels integrated over the corresponding exposure interval.

10. The method of claim 9, wherein the combining comprises deblurring each of the image segments based on the one or more corresponding movement measures and arranging the deblurred image segments along respective rows of the output image.

11. The method of claim 8, wherein the capturing comprises capturing each of the image segments by reading out values of pixels across a two-dimensional array of pixels integrated over the corresponding exposure interval.

12. The method of claim 11, wherein the capturing is performed in accordance with a divided exposure capture process.

13. The method of claim 11, wherein the combining comprises warping the image segments to a common reference frame based on the one or more corresponding movement measures and combining values of corresponding pixels in the warped image segments into corresponding values of pixels in the output image.

17. The method of claim 8, further comprising:

using an image sensor to capture the image segments; and

using separate image sensing elements to capture the images of the scene for the motion sensor signals.

18. The method of claim 8, further comprising:

using an image sensor to capture the image segments; and

using a region of the image sensor to capture the images of the scene for the motion sensor signals.

14. An apparatus, comprising:

means for capturing image segments during corresponding exposure intervals of a frame exposure period;

means for generating respective motion sensor signals during the respective exposure intervals of the image segments, wherein the means for generating comprises means for capturing images of a scene;

means for determining from the respective motion sensor signals one or more corresponding movement measures for each of the exposure intervals, wherein the means for determining comprises means for deriving the movement measures from comparisons of ones of the captured images; and

means for combining the image segments into an output image based on the one or more corresponding movement measures;

wherein the means for capturing images of a scene captures images in accordance with a rolling shutter process that is determined by;

E_FP=e_R+M*T_R,

T_E,i=(i−

1)*T_R+t_0,FP, and

t_R,i=i*T_R+e_R+t_0,FP,

wherein e_R denotes a scanline exposure interval, E_FP denotes a frame exposure period, T_R denotes readout time for each row of pixels, M is the number of rows of pixels in an active area of the image sensor, exposure of row i begins at time t_E,i, t_0,FP is a time at which frame exposure period FP begins, and readout of row i occurs at time t_R,i.