CORRECTING ROLLING SHUTTER USING IMAGE STABILIZATION

• US 20120307084A1
• Filed: 06/06/2011
• Published: 12/06/2012
• Est. Priority Date: 06/06/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Greenebaum, Kenneth I., Mantzel, William E., Mullins, Gregory Keith
• Granted Patent: US 8,823,813 B2
• Time in Patent Office: Days
• US Class Current: 348/208.2
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode

Abstract

Several methods, devices and systems for correcting rolling shutter artifacts are described. In one embodiment, an image capturing system includes a rolling shutter image sensor that may cause a rolling shutter artifact (e.g., warping). The system includes a processing system that is configured to perform an automatic rolling shutter correction mechanism that utilizes calibration data based on a relationship between pixel locations in an image plane of the image sensor and their corresponding rays of light in a coordinate space. The rolling shutter mechanism determines pixel velocity components based on the calibration data and estimates for each image an aggregate pixel velocity based on an aggregation of the pixel velocity components.

Claims

1. A computer implemented method to perform a rolling shutter correction of images captured with an image-capturing device, comprising:

utilizing calibration data for a camera of the image-capturing device to transform image coordinates in an image plane of an image sensor of the camera into a direction vector;

determining pre-computed pixel velocity components based on the calibration data;

capturing a sequence of images with the image-capturing device; and

estimating for each image an aggregate pixel velocity based on an aggregation of the pre-computed pixel velocity components.

2. The method of claim 1, wherein the pre-computed pixel velocity components are pre-computed prior to capturing the sequence of images.

3. The method of claim 1, wherein determining the pre-computed pixel velocity components comprises:

constructing pixel velocity maps from the calibration data.

4. The method of claim 3, wherein estimating for each image the aggregate pixel velocity based on the aggregation of the pre-computed pixel velocity components comprises aggregating a weighted sum of each pre-computed pixel velocity with weights corresponding to the rotational velocity value for each dimension.

5. The method of claim 4, further comprising:

resampling each image to generate a new image to perform the rolling shutter correction.

6. The method of claim 4, further comprising:

determining motion data in three dimensions with a motion-estimating device; and

determining the rotational velocity values based on the motion data.

7. The method of claim 6, wherein the motion-estimating device further comprises a gyroscope, an accelerometer, or any combination thereof.

8. A computer readable medium containing executable computer program instructions which when executed by a data processing system cause said system to perform a method, the method comprising:

utilizing calibration data for the system having an image sensor to transform two-dimensional pixel coordinates in an image plane of the image sensor into a direction vector;

determining pixel velocity components based on the calibration data;

capturing a sequence of images with the system;

determining motion data for captured images with a motion-estimating device of the system; and

estimating for each image an aggregate pixel velocity based on the pixel velocity components and corresponding motion data.

9. The computer readable medium of claim 8, wherein the pixel velocity components comprise pre-computed velocity components that are computed prior to capturing the sequence of images.

10. The computer readable medium of claim 8, wherein determining the pixel velocity components comprises:

constructing pixel velocity maps from the calibration data.

11. The computer readable medium of claim 8, wherein estimating for each image the aggregate pixel velocity based on the aggregation of the pre-computed pixel velocity components comprises aggregating a weighted sum of each pre-computed pixel velocity with weights corresponding to the rotational velocity value for each dimension.

12. The computer readable medium of claim 11, further comprising:

resampling each image to generate a new image to perform a rolling shutter correction.

13. The computer readable medium of claim 12, wherein resampling each image using a binary interpolation to generate a new image is based on a current image and the aggregate pixel velocity.

14. The computer readable medium of claim 8, wherein the motion-estimating device further comprises a gyroscope, an accelerometer, or any combination thereof.

15. An image capturing system, comprising:

a rolling shutter image sensor to sense images;

a storage device coupled to the image sensor, the storage device to store captured images;

a motion-estimating device to detect motion data; and

a processing system coupled to the storage device and the motion-estimating device, the processing system is configured to perform an automatic rolling shutter correction mechanism that utilizes calibration data based on a relationship between pixel locations in an image plane of the image sensor and their corresponding rays of light in a coordinate space, to determine pixel velocity components based on the calibration data; and

to estimate for each image an aggregate pixel velocity based on an aggregation of the pixel velocity components.

16. The system of claim 15, wherein the pixel velocity components comprise pre-computed velocity components that are computed prior to sensing the images.

17. The system of claim 15, wherein determining the pixel velocity components comprises:

constructing pixel velocity maps from the calibration data.

18. The system of claim 17, wherein estimating for each image the aggregate pixel velocity based on the aggregation of the pixel velocity components comprises aggregating a weighted sum of each pre-computed pixel velocity with weights corresponding to the rotational velocity value for each dimension.

19. The system of claim 18, further comprising:

resampling each image to generate a new image.

20. The system of claim 19, wherein resampling each image to generate a new image is based on a current image and the aggregate pixel velocity.

21. The system of claim 15, wherein the motion-estimating device further comprises a gyroscope, an accelerometer, or any combination thereof.

22. The system of claim 18, wherein the processing system is configured to determine the rotational velocity value for each dimension based on the motion data.