Method for dynamically calibrating rotation offset in a camera system

• US 8,860,818 B1
• Filed: 07/31/2013
• Issued: 10/14/2014
• Est. Priority Date: 07/31/2013
• Status: Active Grant

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: Hunter, Kevin L., Zuliani, Marco, Sachs, Todd S., Molgaard, Claus
• Primary Examiner(s): HSIA, SHERRIE Y
• Application Number:
  US13/955,467
• Time in Patent Office: 440 Days
• Field of Search: 348/180, 348/187, 348/188, 348/583, 702/85, 702/94, 702/95, 702150-153
• US Class Current: 348/187
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04N 23/62 Control of parameters via user interfaces
• CPC Class Codes: H04N 23/631 Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
• CPC Class Codes: H04N 23/633 for displaying additional information relating to control or operation of the camera

Abstract

A method for dynamically calibrating rotational offset in a device includes obtaining an image captured by a camera of the device. Orientation information of the device at the time of image capture may be associated with the image. Pixel data of the image may be analyzed to determine an image orientation angle for the image. A device orientation angle may be determined from the orientation information. A rotational offset, based on the image orientation angle and the device orientation angle, may be determined. The rotational offset is relative to the camera or orientation sensor. A rotational bias may be determined from statistical analysis of numerous rotational offsets from numerous respective images. In some embodiments, various thresholds and predetermined ranges may be used to exclude some rotational offsets from the statistical analysis or to discontinue processing for that image.

Claims

1. A device comprising:

a camera configured to capture images;

an orientation sensor configured to provide orientation information for the device;

one or more processors; and

memory storing program instructions executable by the one or more processors to;

obtain an image captured by the camera, wherein orientation information from the orientation sensor is associated with the image, wherein the orientation information is from the time of capture of the image;

analyze pixel data of the image to determine an image orientation angle for the image;

obtain the orientation sensor information for the image;

determine a device orientation angle from the orientation sensor information; and

based on the image orientation angle and the device orientation angle, determine a rotational offset, wherein the rotational offset is relative to the camera or orientation sensor.

2. The device of claim 1, wherein to analyze pixel data of the image to determine an image orientation angle for the image, the program instructions are further executable to:

determine a horizon line for the image from application of a frequency domain technique to the captured image; and

calculate an angle for the image as the image orientation angle for the image, wherein the angle is between the horizon line for the image and an x-axis of the image.

3. The device of claim 1, wherein the program instructions are further executable by the one or more processors to:

output a rotational bias based on the determined rotational offset; and

use the rotational bias for compensation in a horizon adjustment associated with a cropping or panorama function of the device.

4. The device of claim 3, wherein the rotational bias indicates a rotational angle of the orientation sensor or a rotational angle of the camera with respect to the housing of the device.

5. The device of claim 1, wherein the program instructions are further executable by the one or more processors to:

for a plurality of images, repeat said obtain an image, said analyze pixel data, said obtain the orientation sensor information, said determine a device orientation angle, and said based on the image orientation angle and the device orientation angle, determine a rotational offset, wherein the rotational offset is relative to the camera or orientation sensor;

wherein said repeat comprises;

determine whether the image orientation angle is within a predetermined range and not determine a rotational offset for the image for the image when the image orientation angle is outside the predetermined range;

output a plurality of respective rotational offsets for said plurality of images; and

statistically analyze at least a portion of the plurality of rotational offsets for a respective portion of the plurality of images to determine the rotational bias of the device for use in compensation in subsequent image processing, wherein the determined rotational bias is based on the statistical analysis.

6. The device of claim 5, wherein to statistically analyze at least a portion of the plurality of rotational offsets, the program instructions are further executable by the one or more processors to:

determine whether one of the plurality of rotational offsets exceeds a threshold; and

discard a particular one of the plurality of rotational offsets from the statistical analysis for a particular one of the plurality of rotational offsets that exceeds a threshold.

7. The device of claim 5, wherein said repeat further comprises discontinue computing a rotational offset for the image associated with a camera angle that exceeds a device orientation threshold.

8. A method, comprising:

performing by one or more computing devices having one or more processors;

obtaining an image captured by a camera, wherein orientation information from an orientation sensor is associated with the image, wherein the orientation information is from the time of capture of the image;

analyzing pixel data of the image to determine an image orientation angle for the image;

obtaining the orientation sensor information for the image;

determining a device orientation angle from the orientation sensor information; and

based on the image orientation angle and the device orientation angle, determining a rotational offset, wherein the rotational offset is relative to the camera or orientation sensor.

9. The method of claim 8, further comprising;

determining a horizon line for the image from application of a frequency domain technique to the captured image; and

calculating an angle for the image as the image orientation angle for the image, wherein the angle is between the horizon line for the image and an x-axis of the image.

10. The method of claim 8, further comprising:

outputting a rotational bias based on the determined rotational offset; and

using the rotational bias for compensation in a horizon adjustment associated with a cropping or sweep panorama function of the device.

11. The method of claim 10, wherein the rotational bias indicates a rotational angle of the orientation sensor or a rotational angle of the camera with respect to the housing of the device.

12. The method of claim 8, further comprising:

for a plurality of images, repeating said obtaining an image, said analyzing pixel data, said obtaining the orientation sensor information, said determining a device orientation angle, and said based on the image orientation angle and the device orientation angle, determining a rotational offset, wherein the rotational offset is relative to the camera or orientation sensor;

wherein said repeat comprises;

determining whether the image orientation angle is within a predetermined range and not determining a rotational offset for the image for the image when the image orientation angle is outside the predetermined range;

outputting a plurality of respective rotational offsets for said plurality of images; and

statistically analyzing at least a portion of the plurality of rotational offsets for a respective portion of the plurality of images to determine the rotational bias of the device for use in compensation in subsequent image processing, wherein the determined rotational bias is based on the statistical analysis.

13. The method of claim 12, further comprising:

determining whether one of the plurality of rotational offsets exceeds a threshold; and

discarding a particular one of the plurality of rotational offsets from the statistical analysis for a particular one of the plurality of rotational offsets that exceeds a threshold.

14. The method of claim 12, wherein said repeat further comprises discontinue computing a rotational offset for the image associated with a camera angle that exceeds a device orientation threshold.

15. A non-transitory computer readable storage medium storing program instructions executable to perform:

obtaining an image captured by a camera, wherein orientation information from an orientation sensor is associated with the image, wherein the orientation information is from the time of capture of the image;

analyzing pixel data of the image to determine an image orientation angle for the image;

obtaining the orientation sensor information for the image;

determining a device orientation angle from the orientation sensor information; and

based on the image orientation angle and the device orientation angle, determining a rotational offset, wherein the rotational offset is relative to the camera or orientation sensor.

16. The non-transitory computer readable storage medium of claim 15, further comprising;

determining a horizon line for the image from application of a frequency domain technique to the captured image; and

calculating an angle for the image as the image orientation angle for the image, wherein the angle is between the horizon line for the image and an x-axis of the image.

17. The non-transitory computer readable storage medium of claim 15, further comprising:

outputting a rotational bias based on the determined rotational offset; and

using the rotational bias for compensation in a horizon adjustment associated with a cropping or sweep panorama function of the device.

18. The non-transitory computer readable storage medium of claim 17, wherein the rotational bias indicates a rotational angle of the orientation sensor or a rotational angle of the camera with respect to the housing of the device.

19. The non-transitory computer readable storage medium of claim 18, further comprising:

determining whether one of the plurality of rotational offsets exceeds a threshold; and

discarding a particular one of the plurality of rotational offsets from the statistical analysis for a particular one of the plurality of rotational offsets that exceeds a threshold.

20. The non-transitory computer readable storage medium of claim 17, further comprising:

for a plurality of images, repeating said obtaining an image, said analyzing pixel data, said obtaining the orientation sensor information, said determining a device orientation angle, and said based on the image orientation angle and the device orientation angle, determining a rotational offset, wherein the rotational offset is relative to the camera or orientation sensor;

wherein said repeat comprises;

determining whether the image orientation angle is within a predetermined range and not determining a rotational offset for the image for the image when the image orientation angle is outside the predetermined range;

outputting a plurality of respective rotational offsets for said plurality of images; and

statistically analyzing at least a portion of the plurality of rotational offsets for a respective portion of the plurality of images to determine the rotational bias of the device for use in compensation in subsequent image processing, wherein the determined rotational bias is based on the statistical analysis.