Rolling Shutter Reduction Based on Motion Sensors

• US 20130044241A1
• Filed: 08/15/2011
• Published: 02/21/2013
• Est. Priority Date: 08/15/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Zhou, Jianping
• Granted Patent: US 8,786,716 B2
• Time in Patent Office: Days
• US Class Current: 348/241
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode

Abstract

This disclosure pertains to devices, methods, and computer readable media for reducing rolling shutter distortion effects in captured video frames based on timestamped positional information obtained from positional sensors in communication with an image capture device. In general, rolling shutter reduction techniques are described for generating and applying image segment-specific perspective transforms to already-captured segments (i.e., portions) of images in a video sequence, so as to counter or compensate for unwanted distortions that occurred during the read out of the image sensor. Such distortions may be due to, for example, the use of CMOS sensors combined with the rapid movement of the image capture device. In contrast to the prior art, rolling shutter reduction techniques described herein may be applied to captured images in real-time or near real-time using positional sensor information and without intensive image processing that would require an analysis of the content of the underlying image data.

Claims

1. A motion sensor-based rolling shutter reduction method, comprising:

obtaining, by a device comprising an image sensor, a sequence of images, wherein each image in the sequence comprises a plurality of sequentially read out rows, and wherein the device has one or more image capture parameters;

selecting a first plurality of rows from a first age in the sequence;

for each of the first plurality of rows;

obtaining motion information corresponding to motion of the device at approximately the time the row was obtained; and

determining a motion estimate for the row based, at least in part, on the obtained motion information;

defining a first plurality of segments, the first plurality of segments comprising the first image;

for each of the first plurality of segments;

generating a perspective transformation for the segment based, at least in part, on the motion estimates for one or more of the first plurality of rows; and

applying the generated perspective transformation to the segment to generate a corrected image segment;

assembling the plurality of corrected image segments into a first corrected mage; and

storing the first corrected image in a memory.

2. The method of claim 1, wherein the image capture parameters comprise one or more of a principal point, a focus position, and a focal length.

3. The method of claim 1, wherein the act of generating a perspective transformation for the segment is further based, at least in part, on an interpolation of the motion estimates for one or more of the first plurality of rows.

4. The method of claim 1, wherein the act of generating a perspective transformation for the segment is further based, at least in part, on the one or more image capture parameters.

5. The method of claim 1, wherein the act of obtaining motion information comprises obtaining motion data from a gyroscopic sensor in communication with the device.

6. The method of claim 5, wherein the act of obtaining motion information further comprises obtaining acceleration data from an accelerometer sensor in communication with the device.

7. The method of claim 1, wherein the obtained motion information corresponding to motion of the device at approximately the time a row was obtained comprises one or more motion samples.

8. The method of claim 7, wherein the act of determining a motion estimate for the row based on the obtained motion information comprises interpolating between the one or more motion samples comprising the obtained motion information.

9. The method of claim 1, wherein the act of determining a motion estimate for a row further comprises determining rotational information for the row based on the motion information obtained for the corresponding row in one or more preceding images in the sequence and the motion information obtained for the corresponding row in one more successive images in the sequence.

10. The method of claim 1, wherein the act of determining a motion estimate for a row further comprises estimating unwanted motion of the device at approximately the time the row was obtained.

11. The method of claim 10, wherein the act of estimating unwanted motion for a specified row comprises:

identifying a motion of the device during the read out of the corresponding row in a first specified number of images in the sequence obtained before the image having the specified row and a second specified number of images in the sequence obtained after the image having the specified row;

filtering the identified motion of the device to generate a filtered motion, the filtered motion having a value corresponding to the specified row; and

determining the difference between the value of the filtered motion corresponding to the specified row and a position of the device during the read out of the specified row based on the obtained motion information corresponding to the specified row.

12. The method of claim 11, wherein the first specified number is equal to a total number of images that were obtained from the sequence before the image having the specified row was obtained.

16. A non-transitory storage device having instructions stored thereon for causing a programmable control device to perform the method of claim 1.

17. The non-transitory storage device of claim 16, wherein the instructions for obtaining motion data comprise instructions for obtaining motion data from a gyroscopic sensor.

13. A motion sensor-based rolling shutter reduction method, comprising:

obtaining a sequence of images from an image sensor in a device, wherein each image in the sequence comprises a plurality of rows, and wherein the device has image capture parameters;

selecting a first plurality of rows from a first image in the sequence;

determining a motion for each of the first plurality of rows based on motion data from one or more motion sensors of the device;

determining a correction motion for each of the first plurality of rows based, at least in part, on the determined motion for the row;

identifying a first plurality of segments for the first image, wherein each segment comprises a second plurality of rows;

determining a correction motion for each of the first plurality of segments based, at least in part, on the determined correction motion for one or more of the first plurality of rows;

generating a transformation for each of the first plurality of segments based, at least in part, on the segment'"'"'s determined correction motion;

independently applying each of the generated transformations to its corresponding segment to generate a corrected image; and

storing the corrected image in a memory.

14. The method of claim 13, wherein the act of determining a correction motion for each of the first plurality of rows is further based, at least in part, on a determined motion for a corresponding row in one or more other images in the sequence.

15. The method of claim 13, wherein the act of determining a correction motion for each of the first plurality of rows is further based, at least in part, on the image capture parameters.

18. A non-transitory storage device having instructions stored thereon for causing a programmable control device to perform the method of claim 13.

19. An electronic device, comprising:

an image sensor;

a positional sensor;

a memory communicatively coupled to the image sensor;

a programmable control device communicatively coupled to the memory and the positional sensor,

wherein the memory has computer program code stored thereon for causing the programmable control device to;

receive a plurality of sequential images captured by the image sensor, the electronic device having image capture parameters, wherein each of the plurality of sequential images is associated with values corresponding to the image capture parameters at the time each of the images was captured;

obtain motion information from the positional sensor for each of the plurality of sequential images, wherein the motion information for each image in the plurality of sequential images is obtained at approximately the same time as each image was captured, and wherein the motion information comprises a plurality of motion samples;

divide each image from the plurality of sequential images into a plurality of segments;

generate a perspective transformation for each segment of each image based, at least in part, on one or more motion samples corresponding to the segment and the one or more image capture parameters associated with the image;

apply the generated perspective transformations independently to each segment of each of the plurality of sequential images to substantially remove rolling shutter distortions; and

store each of the perspective transformed plurality of sequential images in the memory.

20. The electronic device of claim 19, wherein the motion sensor comprises a gyroscopic sensor.

21. The electronic device of claim 19, wherein the memory has further computer program code stored thereon for causing the programmable control device to determine a timestamp for each of a plurality of rows in each received image.

22. The electronic device of claim 21, wherein the memory has further computer program code stored thereon for causing the programmable control device to determine a timestamp for each of the plurality of motion samples for an image.

23. The electronic device of claim 22, wherein the memory has further computer program code stored thereon for causing the programmable control device to estimate a motion for each of the plurality of rows in each received image based, at least in part, on an interpolation of one or more motion samples whose determined timestamps are closest to the determined timestamp of the row.

24. The electronic device of claim 19, wherein each of the plurality of segments comprises a number of consecutive rows or columns of the image.

25. The electronic device of claim 24, wherein the motion samples corresponding to a segment comprise the one or more motion samples whose determined timestamps are closest to the determined timestamps of the consecutive rows or columns of the image comprising the segment.