Efficient method for reducing noise and blur in a composite still image from a rolling shutter camera

• US 20090201383A1
• Filed: 02/11/2008
• Published: 08/13/2009
• Est. Priority Date: 02/11/2008
• Status: Active Grant

• Current Assignee: NVIDIA Corporation
• Original Assignee: NVIDIA Corporation
• Inventors: Slavin, Keith R.
• Granted Patent: US 8,698,908 B2
• Time in Patent Office: Days
• US Class Current: 348/222.100
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 25/60 Noise processing, e.g. detecting, correcting, reducing or removing noise

Abstract

A rolling shutter digital camera. Each photographic image of a given exposure duration is captured as a multi-frame burst of frames each having a shorter exposure duration to minimize motion blur and to reduce sensor noise by averaging. Each frame is quantized into swaths, captured sequentially by the rolling shutter. Swaths of the first frames are analyzed to select a set of best motion detection reference regions. Swaths of subsequent frames are analyzed versus only those regions, to reduce required computation, and are re-registered accordingly. Corresponding swaths of each frame are accumulated. The accumulator is normalized to the desired bit depth and written as the final image. Averaging of the multiple frames is improved by re-registering swaths rather than entire frames, because of the time delta caused by the rolling shutter. Computation is further reduced by selecting candidate points only along a key line of each swath of the first frame, and still further by pre-limiting the search by using a thumbnail version of the first frame to cull most candidate points. Thresholds are used to ensure adequate motion detection regions are used for each swath.

Claims

1. A method of compositing a plurality of frames of a multi-frame image captured from a sensor having a rolling shutter in a digital camera, the method comprising:

storing a first frame of the plurality into an accumulator,

storing a reference copy of the first frame;

identifying at least a predetermined number of motion estimation reference points in each of a plurality of swaths of the first image, wherein the swaths extend perpendicular to a direction of motion of the rolling shutter;

for each subsequent frame after the first frame,

(a) determining an overall motion vector for each swath of the subsequent image by determining a motion vector for at least some of the identified motion estimation points of a corresponding swath in the first image, and

(b) re-registering each swath of the subsequent frame according to its overall motion vector, and

(c) adding each re-registered swath to-the accumulator; and

after a final frame of the plurality, writing the accumulator as an output image.

2. The method of claim 1 further comprising:

converting the first frame to a Y′

CbCr type color space;

storing the reference copy of the first frame comprises storing at least a luma component of the converted first frame;

converting each subsequent frame to the Y′

CbCr type color space; and

determining the motion vectors is performed by looking for motion of pixel regions using only the luma component of the subsequent frame versus the stored reference copy luma component of the first frame.

3. The method of claim 1 further comprising:

generating a reduced resolution thumbnail of at least the luma component of the first frame;

performing a preliminary screen of the thumbnail to identify locations having high activity levels;

identifying the motion estimation reference points includes analyzing, in the stored reference frame, only pixel regions corresponding to the identified thumbnail locations having high activity levels.

4. A method of generating a final output image from a burst of at least three frames of a multi-frame image from a sensor of a digital camera having a rolling shutter, the method comprising:

(A) accumulating a first frame of the multi-frame image into an accumulating image;

(B) dividing the first frame into a plurality of swaths each extending perpendicular to a direction of motion of the rolling shutter such that the swaths are captured in series;

(C) for each subsequent frame in the multi-frame image,

(i) dividing the subsequent frame into the plurality of swaths,

(ii) for each swath in the subsequent frame,

(a) comparing the swath of the subsequent frame to the corresponding swath of the first frame to identify a motion vector for the swath of the subsequent frame, and

(b) re-registering the swath of the subsequent frame according to the motion vector,

(iii) accumulating the subsequent frame into the accumulated image;

(D) outputting the accumulated image as the final output image.

5. The method of claim 4 further comprising:

(C)(ii)(c) for any missing pixels in the re-registered swath of the subsequent frame, accumulating corresponding pixels from the first image.

6. The method of claim 4 wherein comparing the swath of the subsequent frame to the corresponding swath of the first frame comprises:

searching for motion of pixel regions around a proper subset of pixels along a key line of the swath of the first frame.

7. The method of claim 6 further comprising:

for each swath of the first frame, selecting the proper subset as a predetermined number of stride points.

8. The method of claim 7 further comprising:

selecting the proper subset by constructing a histogram of activity levels of each stride point along the key line.

9. The method of claim 5 further comprising:

selecting the key line to be substantially midway between a top of the swath and a bottom of the swath.

10. The method of claim 5 further comprising:

for each stride point along the key line of each swath, inspecting a thumbnail version of the first frame to identify a predetermined number of candidate points, and searching for motion at a stride point only if the stride point is identified as a candidate point.

11. A method of compositing a plurality of at least three frames of a multi-frame image into a final output image, wherein the multi-frame image is captured by a sensor having a rolling shutter, the method comprising:

(A) for a first frame of the plurality of frames,

(1) if the first frame is in an RGB style color space, converting the first frame to a Y′

CrCb style color space,

(2) saving the converted first frame to an accumulator,

(3) downsampling a Y′

luma component of the first frame to generate a thumbnail,

(4) dividing the first frame into N swaths,

(5) for each of the N swaths,

(a) selecting a key line of pixels in the swath,

(b) selecting a plurality of points along the key line,

(c) for each selected point,

(i) mapping the point into a location in the thumbnail,

(ii) performing a first activity level calculation of a pixel at the mapped location in the thumbnail,

(d) selecting a first predetermined number of candidate points from the plurality of points according to their first calculated activity levels,

(e) saving the selected candidate points in the swath'"'"'s row in a candidate matrix, and

(6) for each swath'"'"'s row in the candidate matrix,

(a) for each saved candidate point in the row,

(i) mapping the saved candidate point into a location in the first frame, and

(ii) performing a second activity level calculation of a pixel at the mapped location in the first frame,

(b) selecting a first predetermined number of motion detection points in the swath according to their second calculated activity levels,

(c) saving the selected motion detection points in the swath'"'"'s row in an edge matrix,

(B) for each subsequent frame after the first frame,

(1) dividing the subsequent frame in to N swaths according to substantially a same division as for the first frame,

(2) for each of the swaths of the subsequent frame,

(a) for each saved motion detection point in the swath'"'"'s row of the edge matrix,

(i) calculating a motion vector of a pixel region at the motion detection point in the subsequent frame versus the corresponding frame of the first frame,

(b) from the calculated motion vectors, calculating an overall motion vector of the swath;

(3) for each of the swaths of the subsequent frame,

(a) re-registering the swath according to the overall motion vector of the swath, and

(b) adding the re-registered swath to the accumulator; and

(C) after a final frame of the plurality,

(1) scaling the accumulator according to a number of frames in the multi-frame image,

(2) writing the scaled accumulator to the final output image.

12. The method of claim 11 further comprising:

if the first frame was in an RGB style color space, one of (i) converting the accumulator back to the RGB style color space before writing the accumulator to the final output stage, and (ii) converting the final output image to the RGB style color space after writing the accumulator to the final output image.