Rolling shutter reduction based on motion sensors

• US 8,913,140 B2
• Filed: 08/29/2012
• Issued: 12/16/2014
• Est. Priority Date: 08/15/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Zhou, Jianping
• Primary Examiner(s): Jerabek, Kelly L
• Application Number:
  US13/598,268
• Publication Number:
  US 20130044230A1
• Time in Patent Office: 839 Days
• Field of Search: 348/208.99, 3482081-2084
• US Class Current: 348/208.4
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode
• CPC Class Codes: H04N 25/531 by controlling rolling shutters in CMOS SSIS

Abstract

This disclosure pertains to devices, methods, and computer readable media for reducing rolling shutter distortion effects in captured video frames based on timestamped positional information obtained from positional sensors in communication with an image capture device. In general, rolling shutter reduction techniques are described for generating and applying image segment-specific perspective transforms to already-captured segments of a single image or images in a video sequence, to compensate for unwanted distortions that occurred during the read out of the image sensor. Such distortions may be due to, for example, the use of CMOS sensors combined with the movement of the image capture device. In contrast to the prior art, rolling shutter reduction techniques described herein may be applied to captured images or videos in real-time or near real-time using positional sensor information and without intensive image processing that would require an analysis of the content of the underlying image data.

Claims

1. A method for reducing rolling shutter distortion in a frame of image data, comprising:

estimating motion of a base row of the frame based on motion sensor data representing motion of image sensor during capture;

for a plurality of other rows of the frame, estimating a motion difference between motion of each other row and the base row based on the motion sensor data;

generating a perspective transform for each other row based on the estimated motion difference; and

applying the generated perspective transforms to image data of the rows to generate corrected frame data.

2. The method of claim 1, wherein the base row is a center row of the image.

3. The method of claim 1, wherein the base row is a center row located in an identified object of the image.

4. The method of claim 1, wherein estimating the motion difference, generating the perspective transformation and applying the generated perspective transform is performed on each row of the frame.

5. The method of claim 1, wherein estimating the motion difference, generating the perspective transformation and applying the generated perspective transform is performed on each anchor row of the frame.

6. The method of claim 1, wherein generating a perspective transform is based on the motion difference and image capture parameters comprising one or more of a principal point, a focus position, and a focal length.

7. The method of claim 1, wherein estimating the motion difference of at least one row of the other rows includes interpolating between a plurality of samples of motion data that are associated with the at least one row to obtain motion information for the at least one row.

8. The method of claim 1, wherein estimating the motion difference of the row of the frame includes compensating for motion of the row by finding an average of the motion of the row of the frame in the current frame and motion of corresponding rows in one or more adjacent images.

9. A method for reducing rolling shutter distortion in an input frame, comprising:

determining motion of each input row of the input frame;

finding a motion difference between the motion of a base row of the input frame and each of the input rows; and

generating a plurality of transformed rows according to a plurality of corresponding perspective transforms, wherein the perspective transforms correct for shutter distortion associated with input rows based on the motion difference.

10. The method of claim 9, further comprising assembling the transformed rows to provide an output frame.

11. The method of claim 9, wherein the base row is a center row of the input frame.

12. The method of claim 9, wherein the perspective transforms are based on the motion difference and image capture parameters comprising one or more of a principal point, a focus position, and a focal length.

13. The method of claim 9, wherein determining motion of each input row of the input frame includes interpolating between a plurality of samples of motion data.

14. The method of claim 9, wherein determining motion includes using motion data from a motion sensor.

15. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed, cause a processor to execute a method comprising:

determining motion of each input row of an input frame;

finding a motion difference between the motion of a base row of the input frame and each of the input rows; and

generating a plurality of transformed rows according to a plurality of corresponding perspective transforms, wherein the perspective transforms correct for shutter distortion associated with input rows based on the motion difference.

16. The non-transitory computer-readable storage medium of claim 15, wherein the base row is a center row of the input frame.

17. The non-transitory computer-readable storage medium of claim 15, wherein the perspective transforms are based on the motion difference and image capture parameters comprising one or more of a principal point, a focus position, and a focal length.

18. The non-transitory computer-readable storage medium of claim 15, wherein determining motion of each input row of the input frame includes interpolating between a plurality of samples of motion data.

19. An apparatus comprising:

a image sensor to capture and output a sequence of input rows of an input frame;

a motion sensor to output motion data corresponding to motion of the apparatus at approximately the time the image sensor captures the sequence of input rows;

a memory to store a corrected frame; and

a controller configured to;

estimate motion of each input row based on the motion data;

find a motion difference for each row between the motion of the row and a base row of the input frame;

generate a perspective transform for each row based on the motion difference of the corresponding row;

apply the corresponding generated perspective transform to each row to generate a corrected row;

assemble the corrected rows into the corrected frame; and

store the corrected frame in the memory.

20. The apparatus of claim 19, wherein the base row is a center row of the input frame.

21. The apparatus of claim 19, wherein the perspective transforms are based on the motion difference and intrinsic camera parameters comprising one or more of a principal point, a focus position, and a focal length.

22. The apparatus of claim 19, wherein estimating motion of each input row includes interpolating between a plurality of samples of the motion data.

23. A chip comprising:

a video device driver configured to receive a captured frame including a plurality of captured rows;

a motion sensor driver configured to receive motion data associated with the captured rows; and

a rolling shutter distortion processor configured to;

determine motion of each captured row of the captured frame;

find a motion difference between the motion of a base row of the captured frame and each of the captured rows; and

generate a plurality of transformed rows according to a plurality of corresponding perspective transforms, wherein the perspective transforms correct for shutter distortion associated with input rows based on the motion difference.

24. The chip of claim 23, wherein the base row is a center row of the captured frame.

25. The chip of claim 23, wherein the perspective transforms are based on the motion difference and intrinsic camera parameters comprising one or more of a principal point, a focus position, and a focal length.

26. A non-transitory computer-readable storage medium storing computer-executable instructions that, when executed, cause a processor to perform a method comprising:

estimating motion of a base row of the frame based on motion sensor data representing motion of image sensor during capture;

for a plurality of other rows of the frame, estimating a motion difference between motion of each other row and the base row based on the motion sensor data;

generating a perspective transform for each other row based on the estimated motion difference; and

applying the generated perspective transforms to image data of the rows to generate corrected frame data.

27. The non-transitory computer-readable storage medium of claim 26, wherein the base row is a center row of the image.

28. The non-transitory computer-readable storage medium of claim 26, wherein generating a perspective transform is based on the motion difference and image capture parameters comprising one or more of a principal point, a focus position, and a focal length.

29. The non-transitory computer-readable storage medium of claim 26, wherein estimating the motion difference of at least one row of the other rows includes interpolating between a plurality of samples of motion data that are associated with the at least one row to obtain motion information for the at least one row.

30. An apparatus comprising:

a image sensor to capture and output a sequence of rows of a frame of image data;

a motion sensor to output motion data corresponding to motion of the apparatus at approximately the time the image sensor captures the sequence of rows;

a memory to store a corrected frame; and

a controller configured to;

estimate motion of a base row of the frame based on the motion data representing motion of the apparatus during capture;

for a plurality of other rows of the frame, estimate a motion difference between motion of each other row and the base row based on the motion data;

generate a perspective transform for each other row based on the estimated motion difference; and

apply the generated perspective transforms to image data of the rows to generate corrected frame data.

31. The apparatus of claim 30, wherein the base row is a center row of the frame.

32. The apparatus of claim 30, wherein generating a perspective transform is based on the motion difference and image capture parameters comprising one or more of a principal point, a focus position, and a focal length.

33. The apparatus of claim 30, wherein estimating the motion difference of at least one row of the other rows includes interpolating between a plurality of samples of motion data that are associated with the at least one row to obtain motion information for the at least one row.

34. A chip comprising:

a video device driver configured to receive a captured frame including a plurality of captured rows;

a motion sensor driver configured to receive motion data associated with the captured rows; and

a rolling shutter distortion processor configured to;

estimate motion of a base row of the frame based on the motion data;

for a plurality of other rows of the frame, estimate a motion difference between motion of each other row and the base row based on the motion data;

generate a perspective transform for each other row based on the estimated motion difference; and

apply the generated perspective transforms to image data of the rows to generate corrected frame data.

35. The chip of claim 34, wherein the base row is a center row of the frame.

36. The chip of claim 34, wherein generating a perspective transform is based on the motion difference and image capture parameters comprising one or more of a principal point, a focus position, and a focal length.

37. The chip of claim 34, wherein estimating the motion difference of at least one row of the other rows includes interpolating between a plurality of samples of motion data that are associated with the at least one row to obtain motion information for the at least one row.