Efficient method for reducing noise and blur in a composite still image from a rolling shutter camera

• US 8,698,908 B2
• Filed: 02/11/2008
• Issued: 04/15/2014
• Est. Priority Date: 02/11/2008
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Slavin, Keith R.
• Primary Examiner(s): Moe, Aung S
• Assistant Examiner(s): HSU, AMY R
• Application Number:
  US12/069,669
• Publication Number:
  US 20090201383A1
• Time in Patent Office: 2,255 Days
• Field of Search: 348/208.2, 348/208.4, 348/208.7, 348/208.11, 348/222.1
• US Class Current: 348/222.1
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise

Abstract

A rolling shutter digital camera. Each photographic image of a given exposure duration is captured as a multi-frame burst of frames each having a shorter exposure duration to minimize motion blur and to reduce sensor noise by averaging. Each frame is quantized into swaths, captured sequentially by the rolling shutter. Swaths of the first frames are analyzed to select a set of best motion detection reference regions. Swaths of subsequent frames are analyzed versus only those regions, to reduce required computation, and are re-registered accordingly. Corresponding swaths of each frame are accumulated. The accumulator is normalized to the desired bit depth and written as the final image. Averaging of the multiple frames is improved by re-registering swaths rather than entire frames, because of the time delta caused by the rolling shutter. Computation is further reduced by selecting candidate points only along a key line of each swath of the first frame, and still further by pre-limiting the search by using a thumbnail version of the first frame to cull most candidate points. Thresholds are used to ensure adequate motion detection regions are used for each swath.

Claims

1. A method of compositing a plurality of frames of a multi-frame image captured from a sensor having a rolling shutter in a digital camera, the method comprising:

storing a first frame of a plurality of frames of a multi-frame image;

generating a reference copy of the first frame;

analyzing the reference copy to determine a plurality of motion detection reference regions;

for each subsequent frame of the plurality of fames after the first frame,

comparing portions of the subsequent frames of the plurality of frames to the plurality of motion detection reference regions of the reference copy to identify frames having activity levels above a predetermined threshold; and

accumulating frames having activity levels above a predetermined threshold into an output image,

wherein the reference copy of the first frame comprises a lower resolution than the first frame.

2. The method according to claim 1, wherein the method further comprises quantizing each of the plurality of frames into a correspond plurality of swaths.

3. The method according to claim 1, wherein the reference copy comprises a luminance component of a Y′

CbCr color space corresponding to the first frame.

4. The method according to claim 1, wherein the first frame is stored in an accumulator.

5. The method according to claim 1, wherein the reference copy is generated by a processor comprised in the digital camera.

6. A method of generating a final output image from a burst of at least three frames of a multi-frame image from a sensor of a digital camera having a rolling shutter, the method comprising:

accumulating a first frame of the multi-frame image into an accumulating image;

dividing the first frame into a plurality of swaths;

generating a reference copy corresponding to the first frame;

identifying pixel locations in the reference copy having activity levels above a predetermined threshold in the generated thumbnail;

for each subsequent frame in the multi-frame image,

(i) dividing the subsequent frame into a corresponding plurality of swaths,

(ii) for each swath in the subsequent frame,

(a) comparing the swath of the subsequent frame to the corresponding swath of the first frame to identify a motion vector for the swath of the subsequent frame, and

(b) re-registering the swath of the subsequent frame according to the motion vector,

(iii) accumulating the subsequent frame into the accumulated image;

(D) outputting the accumulated image as the final output image,

wherein the reference copy corresponding to the first frame comprises a lower resolution than the first frame.

7. The method of claim 6 further comprising:

analyzing the selected pixel region along a key line of the plurality of swaths of the first frame.

8. The method of claim 7 further comprising:

for each swath of the first frame, selecting a selected plurality of pixel regions as a predetermined number of stride points.

9. The method of claim 8 further comprising:

analyzing the selected pixel regions by constructing a histogram of activity levels of each stride point along the key line.

10. The method of claim 7 further comprising:

selecting the key line to be substantially midway between a top of the swath and a bottom of the swath.

11. The method of claim 7 further comprising:

for each stride point along the key line of each swath of the plurality of swaths, inspecting the lower resolution reference frame of the first frame to identify a predetermined number of candidate points, and searching for motion at a stride point only if the stride point is identified as a candidate point.