Methods and system for digitally stabilizing video captured from rolling shutter cameras

• US 20090160957A1
• Filed: 12/20/2007
• Published: 06/25/2009
• Est. Priority Date: 12/20/2007
• Status: Active Grant

• Current Assignee: Aptina Imaging Corporation (ON Semiconductor Corporation)
• Original Assignee: Micron Technology, Inc.
• Inventors: Silverstein, D. A., Deng, Yining
• Granted Patent: US 8,054,335 B2
• Time in Patent Office: Days
• US Class Current: 348/208.990
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 7/238 using non-full search, e.g. three-step search
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS
• CPC Class Codes: H04N 5/144 Movement detection for video coding H04N19/503; analysis of motion in general G06T7/20
• CPC Class Codes: H04N 5/21 Circuitry for suppressing or minimising disturbance, e.g. moiré or halo

Abstract

An imaging device and methods of stabilizing video captured using a rolling shutter operation. Motion estimation is used to determine a row motion vector for individual rows of a current frame of a pixel array. The row motion vectors are used to map the location of pixels in the current frame to a mapped frame representing a single acquisition time for the current frame.

Claims

1. A method of stabilizing video captured using a pixel array and a rolling shutter, the method comprising:

determining a motion vector for individual rows of a current frame of the pixel array;

identifying a reference time during the acquisition of the current frame to which all pixels of the current frame are mapped; and

using the motion vectors, mapping the location of pixels in the current frame to a mapped frame associated with the reference time.

2. The method of claim 1, wherein determining a motion vector further comprises:

dividing the current frame into strips, each strip including a plurality of rows;

identifying blocks of pixels within each strip, the blocks having feature scores indicative of feature characteristics within each block;

selecting a portion of the identified blocks with the highest feature scores;

determining block motion vectors by matching features of each selected block with selected blocks in previous and/or subsequent frames;

using the block motion vectors for each strip to determine a strip motion vector for each strip; and

using the strip motion vectors to determine motion vectors for the individual rows.

3. The method of claim 2, wherein the selected blocks are identified and selected using a line buffer memory that is approximately equal in capacity to two pixel rows of the pixel array.

4. The method of claim 1, wherein pixels of the current frame that are acquired before the reference time are mapped to the mapped frame using a row motion vector determined from the current frame and a next frame.

5. The method of claim 1, wherein pixels of the current frame that are acquired after the reference time are mapped to the mapped frame using a row motion vector determined from the current frame and a previous frame.

6. A method of determining row motion vectors for each row in a current frame of a pixel array, the method comprising:

dividing the current frame into strips, each strip including a plurality of rows;

identifying blocks of pixels within each strip;

determining block motion vectors for each block;

using the block motion vectors for each strip to determine a strip motion vector for each strip; and

using the strip motion vectors to determine the row motion vectors.

7. The method of claim 6, wherein block motion vectors are only determined for a portion of the blocks that have distinguishing features that are able to be matched with blocks from previous and/or subsequent frames.

8. The method of claim 6, wherein a strip motion vector is determined by averaging the block motion vectors for the strip.

9. The method of claim 6, wherein a strip motion vector is determined by finding a median block motion vector of the block motion vectors for the strip and then averaging a portion of the block motion vectors that are closest to the median block motion vector.

10. The method of claim 6, wherein a middle row of a strip is assigned a row motion vector equal to the strip motion vector of the strip, and wherein row motion vectors for the remaining rows of the strip are interpolated from strip motion vectors of neighboring strips.

11. A method of selecting pixel blocks for motion estimation from a strip of n pixel rows using only two rows of memory, wherein n is any power of two, the method comprising:

inputting the first two rows of the strip into the memory;

dividing the memory into two-by-n pixel blocks;

determining a feature score for each block;

selecting half of the blocks with a feature score greater than determined for the other half of the blocks;

discarding the unselected blocks;

filling the available memory with portions of additional rows in the strip that correspond to the selected blocks; and

repeating the determining, the selecting, the discarding and the filling steps until the memory contains n-by-n blocks.

12. The method of claim 11, further comprising pre-seeding the locations of the two-by-n pixel blocks by ranking the two-by-n pixel blocks based on feature scores of neighboring pixel blocks and organizing the two-by-n pixel blocks so that higher-ranked pixel blocks are paired, single-elimination tournament style, with lower-ranked pixel blocks.

13. The method of claim 12, wherein the selecting step includes comparing the feature scores of the pixel block in each pair and selecting one block of each pair.

14. The method of claim 13, wherein if a lower-ranked block is selected over a higher-ranked block, the contents of the lower-ranked block are shuffled into the location of the unselected higher-ranked block.

15. The method of claim 11, wherein the selecting step includes comparing the feature scores of neighboring pairs of pixel blocks and selecting one block of each pair.

16. A device for removing rolling shutter artifacts from captured video, comprising:

a motion estimation unit for determining a row motion vector for individual rows of a current frame of a pixel array; and

a rolling shutter artifact removal unit for using the row motion vectors to map the location of pixels in the current frame to a mapped frame representing a single acquisition time for the current frame.

17. The device of claim 16, wherein the pixel array is divided into strips of pixel rows.

18. The device of claim 17, further comprising a memory buffer for determining pixel blocks from each strip that have distinguishing features useful in motion estimation.

19. The device of claim 18, further comprising a motion vector calculating unit, used to determine the block motion vectors of the pixel blocks and to determine strip motion vectors from the block motion vectors in each strip, and row motion vectors from the determined strip motion vectors.

20. The device of claim 16, further comprising a gyro for determining strip motion vectors for a plurality of strips of pixel rows, the strip motion vectors being used to determine the row motion vectors.

21. An image processing system for rolling shutter video capture, comprising:

a pixel array; and

a processor for processing video frames acquired using the pixel array, the processor coupled to a motion stabilization unit, the motion stabilization unit causing a current video frame to be motion stabilized and comprising;

a motion estimation unit for determining a row motion vector for individual rows of a current frame of a pixel array; and

a rolling shutter artifact removal unit for using the row motion vectors to map the location of pixels in the current frame to a mapped frame representing a single acquisition time for the current frame.

22. The system of claim 21, wherein the motion estimation unit further comprises a gyro for determining row motion vectors.

23. The system of claim 21, wherein the motion estimation unit determines row motion vectors by first determining strip motion vectors for a plurality of strips in the pixel array.

24. The system of claim 23, wherein the motion estimation unit determines strip motion vectors by first determining block motion vectors for pixel blocks in each strip.

25. The system of claim 24, further comprising a two line buffer memory for selecting the pixel blocks used by the motion estimation unit.