Sensor calibration and position estimation based on vanishing point determination

• US 9,135,705 B2
• Filed: 11/07/2012
• Issued: 09/15/2015
• Est. Priority Date: 10/16/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Chao, Hui, Poduri, Sameera, Das, Saumitra Mohan, Naguib, Ayman Fawzy, Mohammad Mirzaei, Faraz
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): CHIU, WESLEY JASON
• Application Number:
  US13/671,390
• Publication Number:
  US 20140104437A1
• Time in Patent Office: 1,042 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01C 17/38 Testing, calibrating, or compensating of compasses
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04W 4/027 using movement velocity, acceleration information

Abstract

Disclosed are systems, apparatus, devices, method, computer program products, and other implementations, including a method that includes capturing an image of a scene by an image capturing unit of a device that includes at least one sensor, determining relative device orientation of the device based, at least in part, on determined location of at least one vanishing point in the captured image of the scene, and performing one or more calibration operations for the at least one sensor based, at least in part, on the determined relative device orientation.

Claims

1. A method comprising:

capturing an image of a scene by an image capturing unit of a device that includes at least one inertial sensor to measure orientation of the device;

determining relative device orientation, representative of a relationship between a real-world coordinate system and a coordinate system of the image capturing unit of the device, based, at least in part, on determined location of at least one vanishing point in the captured image of the scene;

computing a difference between a measured orientation determined by the at least one inertial sensor and the relative device orientation determined based, at least in part, on the determined location of the at least one vanishing point in the captured image of the scene; and

in response to a determination that the computed difference between the measured orientation and the relative device orientation exceeds a predetermined threshold value, performing one or more calibration operations for the at least one inertial sensor based, at least in part, on the relative device orientation determined based, at least in part, on the determined location of the at least one vanishing point in the captured image of the scene.

2. The method of claim 1, wherein determining the relative device orientation comprises:

determining a rotation matrix relating location coordinates in the image capturing unit'"'"'s frame of reference and the scene'"'"'s frame of reference.

3. The method of claim 1, wherein determining the relative device orientation based, at least in part, on the determined location of the at least one vanishing point comprises:

determining the relative device orientation based on determined locations of two or more vanishing points in the captured image of the scene.

4. The method of claim 1, further comprising:

determining actual device orientation based, at least in part, on the relative device orientation determined from the location of the at least one vanishing point in the captured image of the scene, and further based on a scene orientation value associated with the scene.

5. The method of claim 4, further comprising:

retrieving the scene orientation value from map data for a region including an area corresponding to the scene in the image captured by the image capturing unit.

6. The method of claim 1, further comprising:

determining actual device orientation based, at least in part, on the determined relative device orientation, and further based on a previously determined orientation value.

7. The method of claim 1, further comprising:

determining a change of relative device orientation based, at least in part, on relative device orientation values determined based, at least in part, on locations of vanishing points determined from a plurality of images captured by the image capturing unit over a period of time; and

calibrating the at least one inertial sensor based on an orientation change measured by the at least one inertial sensor during the period of time and based on the determined change of relative device orientation.

8. The method of claim 1, wherein the at least one inertial sensor comprises one or more of:

a gyroscope, or a magnetometer.

9. The method of claim 1, wherein performing the one or more calibration operations for the at least one inertial sensor comprises one of:

calibrating the at least one inertial sensor in response to comparison results, produced from a comparison of the measured orientation determined by the at least one inertial sensor and the determined relative device orientation, indicating that the at least one inertial sensor requires calibration;

determining that the at least one inertial sensor is calibrated in response to the comparison results indicating that the at least one inertial sensor is calibrated;

calibrating a magnetometer, in response to the comparison results indicating that the magnetometer requires calibration, based on an indication of a presence of magnetic interference causing temporary erroneous measurements of the measured orientation;

or

determining that the magnetometer is calibrated based on the comparison results and further based on the indication of the presence of magnetic interference causing the temporary erroneous measurements of the measured orientation.

10. The method of claim 9, wherein calibrating the at least one inertial sensor comprises:

determining for the at least one inertial sensor, based on the comparison results, one or more of;

sensor bias, sensor scaling, or sensor misalignment.

11. The method of claim 1, wherein determining the relative device orientation comprises:

determining deviation of the at least one vanishing point from a center of the captured image for the scene.

33. The method of claim 1, wherein performing the one or more calibration operations comprises:

i) calibrating the at least one inertial sensor in response to comparison results, produced from a comparison of the measured orientation determined by the at least one inertial sensor and the determined relative device orientation, indicating that the at least one inertial sensor requires calibration;

ii) determining that the at least one inertial sensor is calibrated in response to the comparison results indicating that the at least one inertial sensor is calibrated;

iii) calibrating a magnetometer, in response to the comparison results indicating that the magnetometer requires calibration, based on an indication of a presence of magnetic interference causing temporary erroneous measurements of the measured orientation;

or

iv) determining that the magnetometer is calibrated based on the comparison results and further based on the indication of the presence of magnetic interference causing the temporary erroneous measurements of the measured orientation;

or

v) any combination thereof.

34. The method of claim 1, wherein determining the relative device orientation based, at least in part, on the determined location of at least one vanishing point in the captured image of the scene comprises determining the location of the at least one vanishing point in the captured image, including:

extracting edge lines in the captured image;

selecting a pre-determined number of pairs of the extracted edge lines;

determining for each of the selected pairs of the extracted edge lines a vanishing point hypothesis corresponding to an intersection point of the respective extracted edge lines;

clustering the each of the determined vanishing point hypothesis into vanishing point clusters; and

setting a center point in at least one of the identified vanishing point clusters as the at least one vanishing point.

12. A device comprising:

an image capturing unit;

at least one inertial sensor to measure orientation of the device; and

a controller configured to;

determine relative device orientation, representative of a relationship between a real-world coordinate system and a coordinate system of the image capturing unit of the device, based, at least in part, on determined location of at least one vanishing point in an image of a scene captured by the image capturing unit;

compute a difference between a measured orientation determined by the at least one inertial sensor and the relative device orientation determined based, at least in part, on the determined location of the at least one vanishing point in the captured image of the scene; and

in response to a determination that the computed difference between the measured orientation and the relative device orientation exceeds a predetermined threshold value, perform one or more calibration operations for the at least one inertial sensor based on the relative device orientation determined based, at least in part, on the determined location of the at least one vanishing point in the captured image of the scene.

13. The device of claim 12, wherein the controller configured to determine the relative device orientation is configured to:

determine a rotation matrix relating location coordinates in the image capturing unit'"'"'s frame of reference and the scene'"'"'s frame of reference.

14. The device of claim 12, wherein the controller configured to determine the relative device orientation based, at least in part, on the determined location of the at least one vanishing point is configured to:

determine the relative device orientation based on determined locations of two or more vanishing points in the captured image of the scene.

15. The device of claim 12, wherein the controller is further configured to:

determine actual device orientation based, at least in part, on the relative device orientation determined from the location of the at least one vanishing point in the captured image of the scene, and further based on a scene orientation value associated with the scene.

16. The device of claim 12, wherein the controller is further configured to:

determine a change of relative device orientation based, at least in part, on relative device orientation values determined based, at least in part, on locations of vanishing points determined from a plurality of images captured by the image capturing unit over a period of time; and

calibrate the at least one inertial sensor based on an orientation change measured by the at least one inertial sensor during the period of time and based on the determined change of relative device orientation.

17. The device of claim 12, wherein the at least one inertial sensor comprises one or more of:

a gyroscope, or a magnetometer.

18. The device of claim 12, wherein the controller configured to perform the one or more calibration operations for the at least one inertial sensor is configured to perform one of:

calibrate the at least one inertial sensor in response to comparison results, produced from a comparison of the measured orientation determined by the at least one inertial sensor and the determined relative device orientation, indicating that the at least one inertial sensor requires calibration;

determine that the at least one inertial sensor is calibrated in response to the comparison results indicating that the at least one inertial sensor is calibrated;

calibrate a magnetometer, in response to the comparison results indicating that the magnetometer requires calibration, based on an indication of a presence of magnetic interference causing temporary erroneous measurements of the measured orientation;

or

determine that the magnetometer is calibrated based on the comparison results and further based on the indication of the presence of magnetic interference causing the temporary erroneous measurements of the measured orientation.

19. An apparatus comprising:

means for capturing an image of a scene for a device that includes at least one inertial sensor to measure orientation of the device;

means for determining relative device orientation, representative of a relationship between a real-world coordinate system and a coordinate system of the means for capturing the image of the device, based, at least in part, on determined location of at least one vanishing point in the captured image of the scene;

means for computing a difference between a measured orientation determined by the at least one inertial sensor and the relative device orientation determined based, at least in part, on the determined location of the at least one vanishing point in the captured image of the scene; and

means for performing one or more calibration operations for the at least one inertial sensor, in response to a determination that the computed difference between the measured orientation and the relative device orientation exceeds a predetermined threshold value, based, at least in part, on the relative device orientation determined based, at least in part, on the determined location of the at least one vanishing point in the captured image of the scene.

20. The apparatus of claim 19, wherein the means for determining the relative device orientation comprise:

means for determining a rotation matrix relating location coordinates in a frame of reference of the means for capturing the image and the scene'"'"'s frame of reference.

21. The apparatus of claim 19, wherein the means for determining the relative device orientation based, at least in part, on the determined location of the at least one vanishing point comprise:

means for determining the relative device orientation based on determined locations of two or more vanishing points in the captured image of the scene.

22. The apparatus of claim 19, further comprising:

means for determining actual device orientation based, at least in part, on the relative device orientation determined from the location of the at least one vanishing point in the captured image of the scene, and further based on a scene orientation value associated with the scene.

23. The apparatus of claim 19, further comprising:

means for determining a change of relative device orientation based, at least in part, on relative device orientation values determined based, at least in part, on locations of vanishing points determined from a plurality of images captured by the means for capturing the image over a period of time; and

means for calibrating the at least one inertial sensor based on an orientation change measured by the at least one inertial sensor during the period of time and based on the determined change of relative device orientation.

24. The apparatus of claim 19, wherein the at least one inertial sensor comprises one or more of:

a gyroscope, or a magnetometer.

25. The apparatus of claim 19, wherein the means for performing the one or more calibration operations for the at least one inertial sensor comprise means for performing one of:

calibrating the at least one inertial sensor in response to comparison results, produced from a comparison of the measured orientation determined by the at least one inertial sensor and the determined relative device orientation, indicating that the at least one inertial sensor requires calibration;

determining that the at least one inertial sensor is calibrated in response to the comparison results indicating that the at least one inertial sensor is calibrated;

calibrating a magnetometer, in response to the comparison results indicating that the magnetometer requires calibration, based on an indication of a presence of magnetic interference causing temporary erroneous measurements of the measured orientation;

or

determining that the magnetometer is calibrated based on the comparison results and further based on the indication of the presence of magnetic interference causing the temporary erroneous measurements of the measured orientation.

26. A non-transitory processor readable media programmed with a set of instructions executable on a processor that, when executed, cause operations comprising:

capturing an image of a scene by an image capturing unit of a device that includes at least one inertial sensor to measure orientation of the device;

determining relative device orientation, representative of a relationship between a real-world coordinate system and a coordinate system of the image capturing unit of the device, based, at least in part, on determined location of at least one vanishing point in the captured image of the scene;

computing a difference between a measured orientation determined by the at least one inertial sensor and the relative device orientation determined based, at least in part, on the determined location of the at least one vanishing point in the captured image of the scene;

in response to a determination that the computed difference between the measured orientation and the relative device orientation exceeds a predetermined threshold value, performing one or more calibration operations for the at least one inertial sensor based, at least in part, on the relative device orientation determined based, at least in part, on the determined location of the at least one vanishing point in the captured image of the scene.

27. The processor readable media of claim 26, wherein determining the relative device orientation comprises:

determining a rotation matrix relating location coordinates in the image capturing unit'"'"'s frame of reference and the scene'"'"'s frame of reference.

28. The processor readable media of claim 26, wherein determining the relative device orientation based, at least in part, on the determined location of the at least one vanishing point comprises:

determining the relative device orientation based on determined locations of two or more vanishing points in the captured image of the scene.

29. The processor readable media of claim 26, further comprising instructions to cause further operations comprising:

determining actual device orientation based, at least in part, on the relative device orientation determined from the location of the at least one vanishing point in the captured image of the scene, and further based on a scene orientation value associated with the scene.

30. The processor readable media of claim 26, further comprising instructions to cause further operations comprising:

determining a change of relative device orientation based, at least in part, on relative device orientation values determined based, at least in part, on locations of vanishing points determined from a plurality of images captured by the image capturing unit over a period of time; and

calibrating the at least one inertial sensor based on an orientation change measured by the at least one inertial sensor during the period of time and based on the determined change of relative device orientation.

31. The processor readable media of claim 26, wherein the at least one inertial sensor comprises one or more of:

a gyroscope, or a magnetometer.

32. The processor readable media of claim 26, wherein performing the one or more calibration operations for the at least one inertial sensor comprises one of:

calibrating the at least one inertial sensor in response to comparison results, produced from a comparison of the measured orientation determined by the at least one inertial sensor and the determined relative device orientation, indicating that the at least one inertial sensor requires calibration;

determining that the at least one inertial sensor is calibrated in response to the comparison results indicating that the at least one inertial sensor is calibrated;

calibrating a magnetometer, in response to the comparison results indicating that the magnetometer requires calibration, based on an indication of a presence of magnetic interference causing temporary erroneous measurements of the measured orientation;

or

determining that the magnetometer is calibrated based on the comparison results and further based on the indication of the presence of magnetic interference causing the temporary erroneous measurements of the measured orientation.