Methods and system for digitally stabilizing video captured from rolling shutter cameras

• US 8,054,335 B2
• Filed: 12/20/2007
• Issued: 11/08/2011
• Est. Priority Date: 12/20/2007
• Status: Active Grant

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Inventors: Deng, Yining, Silverstein, D. Amnon
• Primary Examiner(s): Chan; Jason
• Assistant Examiner(s): Vieaux; Gary C
• Application Number:
  US12/004,056
• Publication Number:
  US 20090160957A1
• Time in Patent Office: 1,419 Days
• Field of Search: 348/208.99, 396/55
• US Class Current: 348/208.1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 7/238 using non-full search, e.g. three-step search
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo

Abstract

An imaging device and methods of stabilizing video captured using a rolling shutter operation. Motion estimation is used to determine a row motion vector for individual rows of a current frame of a pixel array. The row motion vectors are used to map the location of pixels in the current frame to a mapped frame representing a single acquisition time for the current frame.

Claims

1. A method of determining row motion vectors for each row in a current frame of a pixel array, the method comprising:

dividing the current frame into strips, each strip including a plurality of rows;

identifying blocks of pixels within each strip;

determining block motion vectors for each block;

using the block motion vectors for each block to determine a strip motion vector for each strip; and

using the strip motion vectors to determine the row motion vectors.

2. The method of claim 1, wherein block motion vectors are only determined for a portion of the blocks that have distinguishing features that are able to be matched with blocks from previous and/or subsequent frames.

3. The method of claim 1, wherein a strip motion vector is determined by averaging the block motion vectors for the strip.

4. The method of claim 1, wherein a strip motion vector is determined by finding a median block motion vector of the block motion vectors for the strip and then averaging a portion of the block motion vectors that are closest to the median block motion vector.

5. The method of claim 1, wherein a middle row of a strip is assigned a row motion vector equal to the strip motion vector of the strip, and wherein row motion vectors for the remaining rows of the strip are interpolated from strip motion vectors of neighboring strips.

6. A method of selecting pixel blocks for motion estimation from a strip of n pixel rows using only two rows of memory, wherein n is any power of two greater than zero, the method comprising:

inputting the first two rows of the strip into the memory;

dividing the memory into two-by-n pixel blocks;

determining a feature score for each block;

selecting half of the blocks with a feature score greater than determined for the other half of the blocks;

discarding the unselected blocks;

filling the available memory with portions of additional rows in the strip that correspond to the selected blocks; and

repeating the determining, the selecting, the discarding and the filling steps until the memory contains n-by-n blocks.

7. The method of claim 6, further comprising pre-seeding the locations of the two-by-n pixel blocks by ranking the two-by-n pixel blocks based on feature scores of neighboring pixel blocks and organizing the two-by-n pixel blocks so that higher-ranked pixel blocks are paired, single-elimination tournament style, with lower-ranked pixel blocks.

8. The method of claim 7, wherein the selecting step includes comparing the feature scores of the pixel block in each pair and selecting one block of each pair.

9. The method of claim 8, wherein if a lower-ranked block is selected over a higher-ranked block, the contents of the lower-ranked block are shuffled into the location of the unselected higher-ranked block.

10. The method of claim 6, wherein the selecting step includes comparing the feature scores of neighboring pairs of pixel blocks and selecting one block of each pair.