Method and apparatus for image stabilization using multiple image captures

• US 8,081,224 B2
• Filed: 05/07/2008
• Issued: 12/20/2011
• Est. Priority Date: 05/07/2008
• Status: Active Grant

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Inventors: Lin, Peng
• Primary Examiner(s): Tran, Nhan T
• Application Number:
  US12/149,723
• Publication Number:
  US 20090278945A1
• Time in Patent Office: 1,322 Days
• Field of Search: 348/208.1, 348/208.4, 348/208.12, 348/208.13, 348/208.99, 348/241, 348/218.1, 348/239, 348/362, 348/229.1, 348/221.1, 382/254, 382/260, 375/240.16
• US Class Current: 348/208.99
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

A method and apparatus for image stabilization while mitigating the amplification of image noise by using a motion adaptive system employing spatial and temporal filtering of pixel signals from multiple captured frames of a scene.

Claims

1. A method of image processing for image stabilization, comprising:

capturing an image of a scene using a plurality of frames, each frame captured using an exposure time that is a fraction of a determined exposure time for the scene;

spatially filtering pixels in a current one of the captured frames;

temporally filtering pixels in the current captured frame;

detecting motion of the pixels in the current frame relative to the pixels in a previous, temporarily-stored, filtered frame;

combining results of the spatial and temporal filtering of the current captured frame based on detected motion; and

storing the filtered current frame, wherein the motion detecting includes the steps of;

detecting block motion between the current frame and the previous, temporarily-stored, filtered frame; and

detecting pixel motion between the current frame and the previously-filtered frame, and wherein;

the current input frame is the defined by a function ƒ

(x,y,k);

the spatial filtering of the current input frame produces a result ƒ

_sp(x,y,k);

the pixel motion detecting of the current input frame produces a result pm(x,y,k);

the temporal filtering of the current input frame produces a result ƒ

_tp(x,y,k); and

the results of the spatial and temporal filtering are combined as a result ƒ

_out(x,y,k) defined by the equation;

ƒ

_out(x,y,k)=(1−

pm(x,y,k))·

ƒ

_tp(x,y,k) +pm(x,y,k)·

ƒ

_sp(x,y,k).

2. The method of claim 1, wherein the previous filtered frame is temporarily stored in a buffer, and the method further comprises:

replacing the previous filtered frame with the filtered current frame.

3. The method of claim 1, wherein the spatial filtering and motion detecting are functions of an input filter strength variable σ

_n, the input filter strength variable σ

_n relating to the noise in the plurality of frames.

4. The method of claim 3, wherein the input filter strength variable σ

_n corresponds to sensor characteristics and gain applied to the captured image frame.

5. The method of claim 1, wherein a result of the block motion detection is used in detecting pixel motion.

6. The method of claim 1, wherein the spatial filtering applied is adaptive spatial filtering with an output approaching the local mean value in areas of the current captured image frame having a low variance.

7. A method of image processing for image stabilization, comprising:

capturing an image of a scene using a plurality of frames, wherein each captured frame is captured using an exposure time that is a fraction of a determined exposure time for the scene;

spatially filtering pixels in a first one of the captured frames;

buffering the filtered first captured frame; and

for each successive captured frame in the plurality of frames;

spatially filtering each pixel in the captured frame;

temporally filtering each pixel in the captured frame relative to the buffered frame;

detecting motion in the current captured frame relative to the buffered frame;

combining results of the spatial and temporal filtering of the captured frame based on detected motion; and

buffering the combined result as a filtered frame, replacing the previous buffered frame, wherein the motion detecting includes the steps of;

detecting block motion between the current frame and buffered frame; and

detecting pixel motion between the current frame and the buffered frame, and wherein;

the current input frame is the defined by a function ƒ

(x,y,k);

the spatial filtering of the current input frame produces a result ƒ

_sp(x,y,k);

the pixel motion detecting of the current input frame produces a result pm(x,y,k);

the temporal filtering of the current input frame produces a result ƒ

_tp(x,y,k); and

the results of the spatial and temporal filtering are combined as a result ƒ

_out(x,y,k) defined by the equation;

ƒ

_out(x,y,k)=(1−

pm(x,y,k))·

ƒ

_tp(x,y,k)+pm(x,y,k)·

ƒ

_sp(x,y,k).

8. The method of claim 7, wherein a final filtered frame of the plurality of frames is output.

9. The method of claim 7, wherein the spatial filtering and motion detecting are functions of an input filter strength variable σ

_n, the input filter strength variable σ

_n relating to the noise in the plurality of frames.

10. The method of claim 7, wherein the input filter strength variable σ

_n corresponds to sensor characteristics and gain applied to the captured image frame.

11. The method of claim 7, wherein a result of the block motion detection is used in detecting pixel motion.

12. An imager, comprising:

a pixel array that outputs pixel values for a current image frame; and

an image processor electrically connected to the pixel array, the image processor including;

a buffer to store a previously filtered image frame;

a spatial filter to filter the current image frame;

a temporal filter to average the current image frame with the previously filtered image frame;

a motion detector to detect motion between the current image frame and the previously filtered image frame; and

a controller to combine outputs of the spatial filter and the temporal filter for each pixel of the current image frame in response to the motion detected by the motion detector, wherein the motion detector further comprises;

a block motion detector; and

a pixel motion detector, wherein the result of the block motion detector is input to the pixel motion detector, and wherein the current input frame is the defined by a function ƒ

(x,y,k), and;

the result of the spatial filter is ƒ

_sp(x,y,k);

the result of the pixel motion detector is pm(x,y,k);

the result of temporal filter is ƒ

_tp(x,y,k); and

the result of the controller is ƒ

_out(x,y,k) and is defined by the equation;

ƒ

_out(x,y,k)=(1−

pm(x,y,k))·

ƒ

_tp(x,y,k)+pm(x,y,k)·

ƒ

_sp(x,y,k),

wherein a control circuit is electrically connected to the pixel array for controlling the output of pixel values by the pixel array, wherein the control circuit triggers the pixel array to capture an image using a plurality of frames, each frame is captured using an exposure time that is a fraction of a determined exposure time for the scene.

13. The imager of claim 12, wherein the spatial filter and motion detector receive an input filter strength variable σ

_n, the input filter strength variable σ

_nrelating to the noise in the plurality of frames.

14. The imager of claim 13, wherein the input filter strength variable σ

_n corresponds to sensor characteristics and gain applied to the current image frame.

15. The imager of claim 12, wherein the result of the controller ƒ

_out(x,y,k) is input into the buffer.

16. An image processor that receives a plurality of image frames of a captured image, the image processor comprising:

a buffer to store a previously filtered image frame;

a spatial filter to filter a current image frame;

a temporal filter to average the current image frame with the previously filtered image frame;

a motion detector to detect motion between the current image frame and the previously filtered image frame; and

a controller to combine outputs of the spatial filter and the temporal filter for each pixel of the current image frame in response to the motion detected by the motion detector, wherein the motion detector further comprises;

a block motion detector; and

a pixel motion detector, wherein the result of the block motion detector is input to the pixel motion detector, and wherein the current input frame is the defined by a function ƒ

(x,y,k), and;

the result of the spatial ƒ

_sp(x,y,k);

the result of the pixel motion detector is pm(x,y,k);

the result of the temporal filter is ƒ

_tp(x,y,k); and

the result of the controller is ƒ

_out(x,y,k) and is defined by the equation;

ƒ

_out(x,y,k)=(1−

pm(x,y,k)·

ƒ

_tp(x,y,k)+pm(x,y,k)·

ƒ

_sp(x,y,k);

wherein each of the plurality of image frames is captured using an exposure time that is a fraction of a determined exposure time for the scene.

17. The image processor of claim 16, wherein the spatial filter and motion detector receive an input filter strength variable σ

_n, the input filter strength variable σ

_n relating to the noise in the plurality of image frames.

18. The image processor of claim 17, wherein the input filter strength variable σ

_n corresponds to sensor characteristics and gain applied to the current image frame.

19. The image processor of claim 16, wherein the result of the controller ƒ

_out(x,y,k) is input into the buffer.