Correcting rolling shutter using image stabilization

• US 9,602,725 B2
• Filed: 06/10/2014
• Issued: 03/21/2017
• Est. Priority Date: 06/06/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Mantzel, William E., Greenebaum, Kenneth I., Mullins, Gregory Keith
• Primary Examiner(s): CUTLER, ALBERT H
• Application Number:
  US14/300,890
• Publication Number:
  US 20140320681A1
• Time in Patent Office: 1,015 Days
• US Class Current: 1/1
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode

Abstract

Several methods, devices and systems for correcting rolling shutter artifacts are described. In one embodiment, an image capturing system includes a rolling shutter image sensor that may cause a rolling shutter artifact (e.g., warping). The system includes a processing system that is configured to perform an automatic rolling shutter correction mechanism that utilizes calibration data based on a relationship between pixel locations in an image plane of the image sensor and their corresponding rays of light in a coordinate space. The rolling shutter mechanism determines pixel velocity components based on the calibration data and estimates for each image an aggregate pixel velocity based on an aggregation of the pixel velocity components.

Claims

1. A computer implemented method for image stabilization for an image-capturing device with associated calibration data, the method comprising:

determining motion data for the image-capturing device using a motion-estimating device after utilizing the calibration data of the image-capturing device to map image coordinates, which represent two dimensional pixels of an image plane of an image sensor of the image-capturing device, into image coordinates of a 3D coordinate space;

matching motion data to a sequence of frames captured by the image-capturing device to determine motion data for each frame;

computing a desired motion correction from a motion path observed in the motion data to a target motion path;

correcting image coordinates of the image plane based on the calibration data and the desired motion correction by;

applying a desired rotation to image coordinates of the 3D coordinate space; and

utilizing the calibration data of the image-capturing device to map the rotated image coordinates of the 3D coordinate space back to the image plane.

2. The method of claim 1, further comprising:

utilizing calibration data of the image-capturing device to resample each frame to generate a corrected sequence of stabilized frames according to the desired motion correction; and

cropping and filling unknown regions of the stabilized frames.

3. The method of claim 1, wherein the image stabilization to correct for rotational motion and vibration of the image-capturing device.

4. The method of claim 3, wherein the calibration data comprises at least one of radial distortion, field of view of the image, and center point of the capturing device, or any combination thereof.

5. The method of claim 1, further comprising:

constructing the target motion path of the image-capturing device based on the motion data for each frame.

6. The method of claim 1, wherein determining the motion data for the image-capturing device comprises determining rotational velocity vectors in three dimensions for each frame.

7. The method of claim 1, wherein matching motion data to the sequence of frames captured by the image-capturing device to determine motion data for each frame comprises translating time stamps of the motion-estimating device into video time of the frames.

8. The method of claim 7, wherein matching motion data to the sequence of frames captured by the image-capturing device to determine motion data for each frame further comprises integrating rotational velocity data received from the motion-estimating device to estimate inter-frame rotation.

9. A computer readable non-transitory medium containing executable computer program instructions which when executed by a data processing system cause said system to perform a method, the method comprising:

determining motion data for the system using a motion-estimating device after utilizing calibration data of an image-capturing device to map image coordinates, which represent two dimensional pixels of an image plane of an image sensor of the system into image coordinates of a three dimensional (3D) coordinate space;

matching motion data to a sequence of frames captured by the system to determine motion data for each frame;

computing a desired motion correction from a motion path observed in the motion data to a target motion path;

correcting image coordinates of the image plane based on the calibration data and the desired motion correction by;

applying a desired rotation to image coordinates of the 3D coordinate space; and

utilizing the calibration data to map the rotated image coordinates of the 3D coordinate space back to the image plane.

10. The computer readable non-transitory medium of claim 9, the method further comprising:

generating stabilized frames based on the corrected image coordinates.

11. The computer readable non-transitory medium of claim 10, the method further comprising:

adaptively cropping and filling any unknown region of the stabilized frames.

12. The computer readable non-transitory medium of claim 9, wherein calibration data comprises at least one of radial distortion, field of view of the image, center point of the capturing device, or any combination thereof.

13. The computer readable non-transitory medium of claim 9, the method further comprising:

constructing a target motion path of the image-capturing device based on the motion data for each frame, wherein constructing the target motion path of the system comprises applying a lowpass filter in each dimension to the observed motion path.

14. The computer readable non-transitory medium of claim 9, wherein determining the motion data for each frame comprises determining rotational velocity vectors in three dimensions for each frame.

15. The computer readable non-transitory medium of claim 9, wherein matching motion data to a sequence of frames captured by the system to determine motion data for each frame comprises translating time stamps of the motion-estimating device into video time of the frames.

16. An image-capturing device, comprising:

an image sensor with associated calibration data to sense images;

a memory coupled to the image sensor, the memory to store captured images;

a motion-estimating device; and

a processing system coupled to the memory and the motion-estimating device, the processing system including hardware that invokes processing logic to perform an automatic image stabilization mechanism by;

determining motion data for the image-capturing device using the motion estimating device after utilizing calibration data to map image coordinates, which represent two dimensional pixels of an image plane of the image sensor of the image capturing device, into image coordinates of a three dimensional (3D) coordinate space,

matching motion data to a sequence of frames captured by the image-capturing device to determine motion data for each frame,

computing a desired motion correction from a motion path observed in the motion data to a target motion path;

correcting image coordinates of the image plane based on the calibration data and the desired motion correction by;

applying a desired rotation to image coordinates of the 3D coordinate space; and

utilizing the calibration data to map the rotated image coordinates of the 3D coordinate space back to the image plane.

17. The image-capturing device of claim 16, wherein the processing system is further configured to:

construct a target motion path of the system based on the estimated motion path of the system;

apply resampling for each frame from the original image coordinates to the corrected image coordinates to generate stabilized frames; and

crop and fill an unknown region of the image plane.

18. The image-capturing device of claim 17, wherein constructing the target motion path comprises filtering the estimated motion path.

19. The image-capturing device of claim 16, wherein comprises:

at least one of radial distortion, field of view of the image, center point of the capturing device, or any combination thereof.

20. The image-capturing device of claim 16, wherein determining the motion data for each frame comprises determining rotational velocity vectors in three dimensions for each frame.

21. The image-capturing device of claim 16, wherein matching motion data to a sequence of frames captured by the image-capturing device to determine motion data for each frame comprises translating time stamps of the motion-estimating device into video time of the frames.

22. The image-capturing device of claim 21, wherein matching motion data to the sequence of frames captured by the image-capturing device to determine motion data for each frame further comprises integrating rotational velocity data received from the motion-estimating device to estimate inter-frame rotation.