Sensor calibration and position estimation based on vanishing point determination

• US 9,361,688 B2
• Filed: 03/04/2015
• Issued: 06/07/2016
• Est. Priority Date: 10/16/2012
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Chao, Hui, Poduri, Sameera, Das, Saumitra Mohan, Naguib, Ayman Fawzy, Mohammad Mirzaei, Faraz
• Primary Examiner(s): Haskins, Twyler
• Assistant Examiner(s): CHIU, WESLEY JASON
• Application Number:
  US14/638,943
• Publication Number:
  US 20150178924A1
• Time in Patent Office: 461 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G01C 17/38 Testing, calibrating, or compensating of compasses
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04W 4/027 using movement velocity, acceleration information

Abstract

Disclosed are systems, apparatus, devices, method, computer program products, and other implementations, including a method that includes capturing an image of a scene by an image capturing unit of a device that includes at least one sensor, determining relative device orientation of the device based, at least in part, on determined location of at least one vanishing point in the captured image of the scene, and performing one or more calibration operations for the at least one sensor based, at least in part, on the determined relative device orientation.

Claims

1. A method comprising:

capturing an image of a scene, by an image capturing unit of a device, the device further comprising at least one sensor;

determining a location of at least one vanishing point in the captured image of the scene;

determining a measured device orientation of the device based on one or more measurements performed by the at least one sensor of the device; and

determining candidate positions of the device in a region including an area corresponding to the scene based, at least in part, on the location of the at least one vanishing point in the captured image of the scene, the measured device orientation, and orientation data associated with multiple areas within the region, wherein determining the candidate positions comprises determining candidate areas from the multiple areas within the region based on relative orientation of the device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation, and wherein determining the candidate areas comprises determining candidate hallways in the area, with the area including multiple hallways associated with respective orientation values, based on the relative orientation of the device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation.

2. The method of claim 1, wherein determining the candidate positions of the device in the region comprises:

setting the measured device orientation, determined based on the one or more measurements performed by the at least one sensor of the device, as an orientation estimate of the device in response to a determination that the at least one vanishing point is substantially at a center of the image; and

identifying candidate areas, from the multiple areas within the region, with associated orientation values substantially matching the orientation estimate of the device.

3. The method of claim 1, further comprising:

in response to determining that the determined location of the at least one vanishing point in the captured image is at an off-center location, generating an indication to prompt a user to move the device so that a subsequent vanishing point in a subsequent captured image will be located substantially at a subsequent image center of the subsequent captured image.

4. The method of claim 1, wherein determining the relative orientation of the device comprises:

determining a rotation matrix relating location coordinates in the image capturing unit'"'"'s frame of reference and the scene'"'"'s frame of reference.

5. The method of claim 1, wherein determining the relative orientation of the device based, at least in part, on the determined location of the at least one vanishing point comprises:

determining the relative device orientation based on determined locations of two or more vanishing points in the captured image of the scene.

6. The method of claim 1, further comprising:

computing the candidate hallways'"'"' orientation by factoring in the relative orientation of the device, determined from the location of the at least one vanishing point in the captured image, into the measured device orientation determined based on the one or more measurements performed by the at least one sensor.

7. The method of claim 1, further comprising:

retrieving the orientation data associated with the multiple areas within the region from map data for the region.

8. The method of claim 1, wherein the at least one sensor comprises one or more of:

a gyroscope, a magnetometer, or an accelerometer, or any combination thereof.

9. The method of claim 1, wherein determining the location of the at least one vanishing point in the captured image comprises:

extracting edge lines in the captured image;

selecting a pre-determined number of pairs of the extracted edge lines;

determining for each of the selected pairs of the extracted edge lines a vanishing point hypothesis corresponding to an intersection point of the respective extracted edge lines;

clustering the each of the determined vanishing point hypothesis into vanishing point clusters; and

setting a center point in at least one of the identified vanishing point clusters as the at least one vanishing point.

10. A mobile device comprising:

an image capturing unit to capture an image of a scene;

at least one sensor to measure orientation of the mobile device; and

a controller configured to;

determine a location of at least one vanishing point in the image of the scene captured by the image capturing unit;

determine a measured device orientation of the mobile device based on one or more measurements performed by the at least one sensor of the mobile device; and

determine candidate positions of the mobile device in a region including an area corresponding to the scene based, at least in part, on the location of the at least one vanishing point in the captured image of the scene, the measured device orientation, and orientation data associated with multiple areas within the region, wherein the controller configured to determine the candidate positions of the mobile device in the region is configured to determine candidate areas from the multiple areas within the region based on relative orientation of the mobile device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation, and wherein the controller configured to determine the candidate areas is configured to determine candidate hallways in the area, with the area including multiple hallways associated with respective orientation values, based on the relative orientation of the mobile device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation.

11. The mobile device of claim 10, wherein the controller configured to determine the candidate positions of the mobile device in the region is configured to:

set the measured device orientation, determined based on the one or more measurements performed by the at least one sensor of the mobile device, as an orientation estimate of the mobile device in response to a determination that the at least one vanishing point is substantially at a center of the image; and

identify candidate areas, from the multiple areas within the region, with associated orientation values substantially matching the orientation estimate of the mobile device.

12. The mobile device of claim 10, wherein the controller is further configured to:

in response to determining that the determined location of the at least one vanishing point in the captured image is at an off-center location, generate an indication to prompt a user to move the mobile device so that a subsequent vanishing point in a subsequent captured image will be located substantially at subsequent image center of the subsequent captured image.

13. The mobile device of claim 10, wherein the controller configured to determine the relative orientation of the mobile device is configured to:

determine a rotation matrix relating location coordinates in the image capturing unit'"'"'s frame of reference and the scene'"'"'s frame of reference.

14. The mobile device of claim 10, wherein the controller is further configured to:

retrieve the orientation data associated with the multiple areas within the region from map data for the region.

15. The mobile device of claim 10, wherein the at least one sensor comprises one or more of:

a gyroscope, a magnetometer, or an accelerometer, or any combination thereof.

16. An apparatus comprising:

means for capturing an image of a scene for a device that includes at least one sensor;

means for determining a location of at least one vanishing point in the captured image of the scene;

means for determining a measured device orientation of the device based on one or more measurements performed by the at least one sensor of the device; and

means for determining candidate positions of the device in a region including an area corresponding to the scene based, at least in part, on the location of the at least one vanishing point in the captured image of the scene, the measured device orientation, and orientation data associated with multiple areas within the region, wherein the means for determining the candidate positions comprises means for determining candidate areas from the multiple areas within the region based on relative orientation of the device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation, and wherein the means for determining the candidate areas comprises means for determining candidate hallways in the area, with the area including multiple hallways associated with respective orientation values, based on the relative orientation of the device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation.

17. The apparatus of claim 16, wherein the means for determining the candidate positions of the device in the region comprises:

means for setting the measured device orientation, determined based on the one or more measurements performed by the at least one sensor of the device, as an orientation estimate of the device in response to a determination that the at least one vanishing point is substantially at a center of the image; and

means for identifying candidate areas, from the multiple areas within the region, with associated orientation values substantially matching the orientation estimate of the device.

18. The apparatus of claim 16, wherein the at least one sensor comprises one or more of:

a gyroscope, a magnetometer, or an accelerometer, or any combination thereof.

19. A non-transitory computer readable media programmed with a set of instructions, executable on a processor, to:

capture an image of a scene, by an image capturing unit of a device, the device further comprising at least one sensor;

determine a location of at least one vanishing point in an image of a scene captured by the image capturing unit;

determine a measured device orientation of the device based on one or more measurements performed by the at least one sensor of the device; and

determine candidate positions of the device in a region including an area corresponding to the scene based, at least in part, on the location of the at least one vanishing point in the captured image of the scene, the measured device orientation, and orientation data associated with multiple areas within the region, wherein the instructions to determine the candidate positions of the mobile device in the region comprise one or more instructions to determine candidate areas from the multiple areas within the region based on relative orientation of the device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation, and wherein the one or more instructions to determine the candidate areas comprise further instructions to determine candidate hallways in the area, with the area including multiple hallways associated with respective orientation values, based on the relative orientation of the device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation.

20. The computer readable media of claim 19, wherein the instructions to determine the candidate positions of the device in the region comprise instructions to:

set the measured device orientation, determined based on the one or more measurements performed by the at least one sensor of the device, as an orientation estimate of the device in response to a determination that the at least one vanishing point is substantially at a center of the image; and

identify candidate areas, from the multiple areas within the region, with associated orientation values substantially matching the orientation estimate of the device.

21. The computer readable media of claim 19, wherein the set of instructions further comprises instructions to:

retrieve the orientation data associated with the multiple areas within the region from map data for the region.

22. The computer readable media of claim 19, wherein the at least one sensor comprises one or more of:

a gyroscope, a magnetometer, or an accelerometer, or any combination thereof.