Method and apparatus for image stabilization using multiple image captures

• US 20090278945A1
• Filed: 05/07/2008
• Published: 11/12/2009
• Est. Priority Date: 05/07/2008
• Status: Active Grant

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Micron Technology, Inc.
• Inventors: Lin, Peng
• Granted Patent: US 8,081,224 B2
• Time in Patent Office: Days
• US Class Current: 348/208.100
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory

Abstract

A method and apparatus for image stabilization while mitigating the amplification of image noise by using a motion adaptive system employing spatial and temporal filtering of pixel signals from multiple captured frames of a scene.

Claims

1. A method of image processing for image stabilization, comprising:

capturing an image of a scene using a plurality of frames, each frame captured using an exposure time that is a fraction of a determined exposure time for the scene;

spatially filtering pixels in a current one of the captured frames;

temporally filtering pixels in the current captured frame;

detecting motion of the pixels in the current frame relative to the pixels in a previous, temporarily-stored, filtered frame;

combining results of the spatial and temporal filtering of the current captured frame based on detected motion; and

storing the filtered current frame.

2. The method of claim 1, wherein the previous filtered frame is temporarily stored in a buffer, and the method further comprises:

replacing the previous filtered frame with the filtered current frame.

3. The method of claim 1, wherein the results of the spatial and temporal filtering are combined as a weighted function.

4. The method of claim 1, wherein the spatial filtering and motion detecting are functions of an input filter strength variable σ

_n, the input filter strength variable σ

_n relating to the noise in the plurality of frames.

5. The method of claim 4, wherein the input filter strength variable σ

_n corresponds to sensor characteristics and gain applied to the captured image frame.

6. The method of claim 1, wherein the motion detecting includes the steps of:

detecting block motion between the current frame and the previous, temporarily-stored, filtered frame; and

detecting pixel motion between the current frame and the previously-filtered frame.

7. The method of claim 6, wherein a result of the block motion detection is used in detecting pixel motion.

8. The method of claim 6, wherein:

the current input frame is the defined by a function ƒ

(x,y,k);

the spatial filtering of the current input frame produces a result f_sp(x,y,k);

the pixel motion detecting of the current input frame produces a result pm(x,y,k);

the temporal filtering of the current input frame produces a result f_tp(x,y,k); and

the results of the spatial and temporal filtering are combined as a result f_out(x,y,k) defined by the equation;

f_out(x,y,k)=(1−

pm(x,y,k))·

f_tp(x,y,k)+pm(x,y,k)·

f_sp(x,y,k).

9. The method of claim 8, wherein the spatial filtering applied is adaptive spatial filtering with an output approaching the local mean value in areas of the current captured image frame having a low variance.

10. A method of image processing for image stabilization, comprising:

capturing an image of a scene using a plurality of frames, wherein each captured frame is captured using an exposure time that is a fraction of a determined exposure time for the scene;

spatially filtering pixels in a first one of the captured frames;

buffering the filtered first captured frame; and

for each successive captured frame in the plurality of frames;

spatially filtering each pixel in the captured frame;

temporally filtering each pixel in the captured frame relative to the buffered frame;

detecting motion in the current captured frame relative to the buffered frame;

combining results of the spatial and temporal filtering of the captured frame based on detected motion; and

buffering the combined result as a filtered frame, replacing the previous buffered frame.

11. The method of claim 10, wherein a final filtered frame of the plurality of frames is output.

12. The method of claim 10, wherein the results of the spatial and temporal filtering are combined as a weighted function.

13. The method of claim 10, wherein the spatial filtering and motion detecting are functions of an input filter strength variable σ

_n, the input filter strength variable σ

_n relating to the noise in the plurality of frames.

14. The method of claim 10, wherein the input filter strength variable σ

_n corresponds to sensor characteristics and gain applied to the captured image frame.

15. The method of claim 10, wherein the motion detecting includes the steps of:

detecting block motion between the current frame and the buffered frame; and

detecting pixel motion between the current frame and the buffered frame.

16. The method of claim 15, wherein a result of the block motion detection is used in detecting pixel motion.

17. The method of claim 15, wherein:

the current input frame is the defined by a function ƒ

(x,y,k);

the spatial filtering of the current input frame produces a result f_sp(x,y,k);

the pixel motion detecting of the current input frame produces a result pm(x,y,k);

the temporal filtering of the current input frame produces a result f_tp(x,y,k); and

the results of the spatial and temporal filtering are combined as a result f_out(x,y,k) defined by the equation;

f_out(x,y,k)=(1−

pm(x,y,k))·

f_tp(x,y,k)+pm(x,y,k)·

f_sp(x,y,k).

18. An imager, comprising:

a pixel array that outputs pixel values for a current image frame; and

an image processor electrically connected to the pixel array, the image processor including;

a buffer to store a previously filtered image frame;

a spatial filter to filter the current image frame;

a temporal filter to average the current image frame with the previously filtered image frame;

a motion detector to detect motion between the current image frame and the previously filtered image frame; and

a controller to combine outputs of the spatial filter and the temporal filter for each pixel of the current image frame in response to the motion detected by the motion detector.

19. The imager of claim 18, further comprising:

a control circuit electrically connected to the pixel array for controlling the output of pixel values by the pixel array, wherein the control circuit triggers the pixel array to capture an image using a plurality of frames, each frame captured using an exposure time that is a fraction of a determined exposure time for the scene.

20. The imager of claim 18, wherein the spatial filter and motion detector receive an input filter strength variable σ

_n, the input filter strength variable σ

_n relating to the noise in the plurality of frames.

21. The imager of claim 20, wherein the input filter strength variable σ

_n corresponds to sensor characteristics and gain applied to the current image frame.

22. The imager of claim 18, wherein the motion detector further comprises:

a block motion detector; and

a pixel motion detector, wherein the result of the block motion detector is input to the pixel motion detector.

23. The imager of claim 22, wherein the current input frame is the defined by a function ƒ

(x,y,k), and;

the result of the spatial filter is f_sp(x,y,k);

the result of the pixel motion detector is pm(x,y,k);

the result of the temporal filter is f_tp(x,y,k); and

the result of the controller is f_out(x,y,k) and is defined by the equation;

f_out(x,y,k)=(1−

pm(x,y,k))·

f_tp(x,y,k)+pm(x,y,k)·

f_sp(x,y,k).

24. The imager of claim 23, wherein the result of the controller f_out(x,y,k) is input into the buffer.

25. An image processor that receives a plurality of image frames of a captured image, the image processor comprising:

a buffer to store a previously filtered image frame;

a spatial filter to filter a current image frame;

a temporal filter to average the current image frame with the previously filtered image frame;

a motion detector to detect motion between the current image frame and the previously filtered image frame; and

a controller to combine outputs of the spatial filter and the temporal filter for each pixel of the current image frame in response to the motion detected by the motion detector.

26. The image processor of claim 25, wherein the spatial filter and motion detector receive an input filter strength variable σ

_n, the input filter strength variable σ

_n relating to the noise in the plurality of image frames.

27. The image processor of claim 26, wherein the input filter strength variable σ

_n corresponds to sensor characteristics and gain applied to the current image frame.

28. The image processor of claim 25, wherein the motion detector further comprises:

a block motion detector; and

a pixel motion detector, wherein the result of the block motion detector is input to the pixel motion detector.

29. The image processor of claim 28, wherein the current input frame is the defined by a function ƒ

(x,y,k), and;

the result of the spatial filter is f_sp(x,y,k);

the result of the pixel motion detector is pm(x,y,k);

the result of the temporal filter is f_tp(x,y,k); and

the result of the controller is f_out(x,y,k) and is defined by the equation;

f_out(x,y,k)=(1−

pm(x,y,k))·

f_tp(x,y,k)+pm(x,y,k)·

f_sp(x,y,k).

30. The image processor of claim 29, wherein the result of the controller f_out(x,y,k) is input into the buffer.