Correcting rolling shutter using image stabilization

• US 8,823,813 B2
• Filed: 06/06/2011
• Issued: 09/02/2014
• Est. Priority Date: 06/06/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Mantzel, William E., Greenebaum, Kenneth I., Mullins, Gregory Keith
• Primary Examiner(s): CUTLER, ALBERT H
• Application Number:
  US13/154,389
• Publication Number:
  US 20120307084A1
• Time in Patent Office: 1,184 Days
• Field of Search: None
• US Class Current: 348/208.6
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode

Abstract

Several methods, devices and systems for correcting rolling shutter artifacts are described. In one embodiment, an image capturing system includes a rolling shutter image sensor that may cause a rolling shutter artifact (e.g., warping). The system includes a processing system that is configured to perform an automatic rolling shutter correction mechanism that utilizes calibration data based on a relationship between pixel locations in an image plane of the image sensor and their corresponding rays of light in a coordinate space. The rolling shutter mechanism determines pixel velocity components based on the calibration data and estimates for each image an aggregate pixel velocity based on an aggregation of the pixel velocity components.

Claims

1. A computer implemented method to perform a rolling shutter correction of images captured with an image-capturing device, comprising:

utilizing calibration data for a camera of the image-capturing device to transform image coordinates in an image plane of an image sensor of the camera into a direction vector;

determining pre-computed pixel velocity components based on the calibration data;

capturing a sequence of images with the image-capturing device;

estimating for each image an aggregate pixel velocity based on aggregating a weighted sum of each pre-computed pixel velocity component with weights corresponding to a rotational velocity vector for each dimension; and

resampling each image to generate a new image to perform the rolling shutter correction.

2. The method of claim 1, wherein the pre-computed pixel velocity components are pre-computed prior to capturing the sequence of images.

3. The method of claim 1, wherein determining the pre-computed pixel velocity components comprises:

constructing pixel velocity maps from the calibration data.

4. The method of claim 1, further comprising:

determining motion data in three dimensions with a motion-estimating device; and

determining rotational velocity values for the rotational velocity vector for each dimension based on the motion data.

5. The method of claim 4, wherein the motion-estimating device further comprises a gyroscope, an accelerometer, or any combination thereof.

6. A non-transitory computer readable medium containing executable computer program instructions which when executed by a data processing system cause said system to perform a method, the method comprising:

utilizing calibration data for the system having an image sensor to transform two-dimensional pixel coordinates in an image plane of the image sensor into a direction vector;

determining pixel velocity components based on the calibration data;

capturing a sequence of images with the system;

determining motion data for captured images with a motion-estimating device of the system;

estimating for each image an aggregate pixel velocity based on aggregating a weighted sum of each pre-computed pixel velocity component with weights corresponding to a rotational velocity vector for each dimension; and

resampling each image to generate a new image to perform the rolling shutter correction.

7. The non-transitory computer readable medium of claim 6, wherein the pixel velocity components comprise pre-computed velocity components that are computed prior to capturing the sequence of images.

8. The non-transitory computer readable medium of claim 6, wherein determining the pixel velocity components comprises:

constructing pixel velocity maps from the calibration data.

9. The non-transitory computer readable medium of claim 6, wherein resampling each image comprises using a binary interpolation to generate a new image based on a current image and the aggregate pixel velocity.

10. The non-transitory computer readable medium of claim 6, wherein the motion-estimating device further comprises a gyroscope, an accelerometer, or any combination thereof.

11. An image capturing system, comprising:

a rolling shutter image sensor to sense images;

a storage device coupled to the image sensor, the storage device to store captured images;

a motion-estimating device to detect motion data; and

a processing system coupled to the storage device and the motion-estimating device, the processing system is configured to perform an automatic rolling shutter correction mechanism that utilizes calibration data based on a relationship between pixel locations in an image plane of the image sensor and their corresponding rays of light in a coordinate space, to determine pixel velocity components based on the calibration data;

to estimate for each image an aggregate pixel velocity based on aggregating a weighted sum of each pixel velocity component with weights corresponding to a rotational velocity vector for each dimension; and

to resample each image to generate a new image.

12. The system of claim 11, wherein the pixel velocity components comprise pre-computed velocity components that are computed prior to sensing the images.

13. The system of claim 11, wherein determining the pixel velocity components comprises:

constructing pixel velocity maps from the calibration data.

14. The system of claim 11, wherein resampling each image to generate a new image is based on a current image and the aggregate pixel velocity.

15. The system of claim 11, wherein the motion-estimating device further comprises a gyroscope, an accelerometer, or any combination thereof.

16. The system of claim 11, wherein the processing system is configured to determine rotational velocity values for the rotational velocity vector for each dimension based on the motion data.

17. A computer implemented method to perform a rolling shutter correction of images captured with an image-capturing device, comprising:

utilizing calibration data for a camera of the image-capturing device to transform image coordinates in an image plane of an image sensor of the camera into a direction vector;

determining pre-computed pixel velocity components based on the calibration data;

capturing a sequence of images with the image-capturing device; and

estimating for each image an aggregate pixel velocity based on aggregating a weighted sum of each pre-computed pixel velocity component with a rotational velocity weight for each dimension.

18. The method of claim 17, further comprising:

determining motion data in three dimensions with a motion-estimating device; and

determining rotational velocity values for the rotational velocity vector for each dimension based on the motion data.

19. A non-transitory computer readable medium containing executable computer program instructions which when executed by a data processing system cause said system to perform a method, the method comprising:

utilizing calibration data for the system having an image sensor to transform two-dimensional pixel coordinates in an image plane of the image sensor into a direction vector;

determining pixel velocity components based on the calibration data;

capturing a sequence of images with the system;

determining motion data for captured images with a motion-estimating device of the system; and

estimating for each image an aggregate pixel velocity based on aggregating a weighted sum of each pre-computed pixel velocity component with a rotational velocity weight for each dimension.

20. The non-transitory computer readable medium of claim 19, wherein the method further comprising:

determining motion data in three dimensions with a motion-estimating device; and

determining rotational velocity values for the rotational velocity vector for each dimension based on the motion data.

21. An image capturing system, comprising:

a rolling shutter image sensor to sense images;

a storage device coupled to the image sensor, the storage device to store captured images;

a motion-estimating device to detect motion data; and

a processing system coupled to the storage device and the motion-estimating device, the processing system is configured to perform an automatic rolling shutter correction mechanism that utilizes calibration data based on a relationship between pixel locations in an image plane of the image sensor and their corresponding rays of light in a coordinate space, to determine pixel velocity components based on the calibration data; and

to estimate for each image an aggregate pixel velocity based on aggregating a weighted sum of each pre-computed pixel velocity component with a rotational velocity weight for each dimension.

22. The image capturing system of claim 21, wherein the pixel velocity components comprise pre-computed velocity components that are computed prior to sensing the images.