Methods and systems for image stabilization

• US 8,648,919 B2
• Filed: 06/06/2011
• Issued: 02/11/2014
• Est. Priority Date: 06/06/2011
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Mantzel, William E., Greenebaum, Kenneth I.
• Primary Examiner(s): CUTLER, ALBERT H
• Application Number:
  US13/154,375
• Publication Number:
  US 20120307085A1
• Time in Patent Office: 981 Days
• Field of Search: None
• US Class Current: 348/208.6
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/689 Motion occurring during a rolling shutter mode

Abstract

Several methods, devices and systems for stabilizing images and correcting rolling shutter effects are described. In one embodiment, a computer implemented method for image stabilization for an image-capturing device with associated calibration data includes determining motion data for the image-capturing device using a motion-estimating device. The method also includes matching motion data to a sequence of frames captured by the image-capturing device to determine motion data for each frame, constructing a target motion path of the image-capturing device based on the motion data for each frame, and computing a desired motion correction from a motion path observed in the motion data to the target motion path.

Claims

1. A computer implemented method for image stabilization for an image-capturing device with associated calibration data, the method comprising:

utilizing camera calibration data of the image-capturing device to map image coordinates of an image plane of an image sensor of the image-capturing device into normalized image coordinates of a three dimensional (3D) coordinate space;

determining motion data including 3D rotation data for the image-capturing device using a motion-estimating device after utilizing camera calibration data of the image-capturing device to map image coordinates of an image plane of an image sensor of the image-capturing device into normalized image coordinates of a 3D coordinate space;

matching motion data to a sequence of frames captured by the image-capturing device to determine motion data for each frame;

constructing a target motion path of the image-capturing device based on the motion data for each frame; and

computing a desired motion correction from a motion path observed in the motion data to the target motion path.

2. The method of claim 1, further comprising:

utilizing camera calibration data of the image-capturing device to resample each frame to generate a corrected sequence of stabilized frames according to the desired motion correction; and

cropping and filling unknown regions of the stabilized frames.

3. The method of claim 1, wherein the image stabilization to correct for rotational motion and vibration of the image-capturing device.

4. The method of claim 3, wherein the calibration data comprises radial distortion, field of view of the image, and center point of the capturing device, or any combination thereof.

5. The method of claim 1, wherein the motion-estimating device comprises a gyroscope, an accelerometer, or any combination thereof.

6. The method of claim 1, wherein determining the motion data for each frame comprises determining rotational velocity vectors in three dimensions for each frame.

7. The method of claim 1, wherein matching motion data to the sequence of frames captured by the image-capturing device to determine motion data for each frame comprises translating time stamps of the motion-estimating device into video time of the frames.

8. The method of claim 7, wherein matching motion data to the sequence of frames captured by the image-capturing device to determine motion data for each frame further comprises integrating rotational velocity data received from the motion-estimating device to estimate inter-frame rotation.

9. A computer readable non-transitory medium containing executable computer program instructions which when executed by a data processing system cause said system to perform a method, the method comprising:

utilizing calibration data of an image-capturing device to map image coordinates of an image plane of an image sensor of the system into normalized image coordinates of a three dimensional (3D) coordinate space;

determining motion data including 3D rotation data for the system using a motion-estimating device after utilizing calibration data of the image-capturing device to map image coordinates of an image plane of an image sensor of the system into normalized image coordinates of a three dimensional coordinate space;

matching motion data to a sequence of frames captured by the system to determine motion data for each frame; and

constructing a target motion path of the system based on the motion data for each frame; and

computing a desired motion correction from a motion path observed in the motion data to the target motion path.

10. The computer readable non-transitory medium of claim 9, further comprising:

utilizing calibration data of the system to correct image coordinates of the image plane based on the desired motion correction; and

generating stabilized frames based on the corrected image coordinates.

11. The computer readable non-transitory medium of claim 10, wherein utilizing calibration data comprises mapping the image coordinates, which represent two dimensional pixels, into the normalized image coordinates of the three dimensional coordinate space, which represent light rays.

12. The computer readable non-transitory medium of claim 10, further comprising:

adaptively cropping and filling any unknown region of the stabilized frames.

13. The computer readable non-transitory medium of claim 9, wherein constructing the target motion path of the system comprises applying a lowpass filter in each dimension to the estimated motion path.

14. The computer readable non-transitory medium of claim 9, wherein determining the motion data for each frame comprises determining rotational velocity vectors in three dimensions for each frame.

15. The computer readable non-transitory medium of claim 9, wherein matching motion data to a sequence of frames captured by the system to determine motion data for each frame comprises translating time stamps of the motion-estimating device into video time of the frames.

16. An image-capturing device, comprising:

an image sensor with associated calibration data to sense images;

a storage device coupled to the image sensor, the storage device to store captured images;

a motion-estimating device to detect motion data including three dimensional (3D) rotation data; and

a processing system coupled to the storage device and the motion-estimating device, the processing system is configured to perform an automatic image stabilization mechanism by utilizing calibration data to map image coordinates, which represent two dimensional pixels, into normalized image coordinates of a three dimensional coordinate space, which represent light rays, subsequently determining motion data for the image-capturing device using the motion-estimating device, matching motion data to a sequence of frames captured by the image-capturing device to determine motion data for each frame, and estimating a motion path of the image-capturing device based on the motion data for each frame.

17. The image-capturing device of claim 16, wherein the processing system is further configured to:

construct a target motion path of the system based on the estimated motion path of the system;

compute a desired motion correction from the estimated motion path to the target motion path;

correct image coordinates of the image plane based on the calibration data and the desired motion correction;

apply resampling for each frame from the original image coordinates to the corrected image coordinates to generate stabilized frames; and

crop and fill an unknown region of the image plane.

18. The image-capturing device of claim 17, wherein correcting image coordinates of the image plane based on the calibration data and the desired motion correction comprises:

utilizing the calibration data to map the image coordinates, which represent two dimensional pixels, into the normalized image coordinates of the coordinate space, which represent light rays;

applying a desired rotation to all light rays, according to the desired motion correction; and

utilizing the calibration data to map these rotated light rays back to the image plane.

19. The image-capturing device of claim 17, wherein constructing the target motion path comprises filtering the estimated motion path.

20. The image-capturing device of claim 16, wherein determining the motion data for each frame comprises determining rotational velocity vectors in three dimensions for each frame.

21. The image-capturing device of claim 16, wherein matching motion data to a sequence of frames captured by the image-capturing device to determine motion data for each frame comprises translating time stamps of the motion-estimating device into video time of the frames.

22. The image-capturing device of claim 21, wherein matching motion data to the sequence of frames captured by the image-capturing device to determine motion data for each frame further comprises integrating rotational velocity data received from the motion-estimating device to estimate inter-frame rotation.