SENSOR CALIBRATION AND POSITION ESTIMATION BASED ON VANISHING POINT DETERMINATION

• US 20140104437A1
• Filed: 11/07/2012
• Published: 04/17/2014
• Est. Priority Date: 10/16/2012
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: CHAO, Hui, PODURI, Sameera, DAS, Saumitra Mohan, NAGUIB, Ayman Fawzy, MIRZAEI, Faraz Mohammad
• Granted Patent: US 9,135,705 B2
• Time in Patent Office: Days
• US Class Current: 348/187
• CPC Class Codes: G01C 17/38 Testing, calibrating, or compensating of compasses
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 17/002 for television cameras
• CPC Class Codes: H04W 4/027 using movement velocity, acceleration information

Abstract

Disclosed are systems, apparatus, devices, method, computer program products, and other implementations, including a method that includes capturing an image of a scene by an image capturing unit of a device that includes at least one sensor, determining relative device orientation of the device based, at least in part, on determined location of at least one vanishing point in the captured image of the scene, and performing one or more calibration operations for the at least one sensor based, at least in part, on the determined relative device orientation.

Claims

1. A method comprising:

capturing an image of a scene by an image capturing unit of a device that includes at least one sensor;

determining relative device orientation of the device based, at least in part, on determined location of at least one vanishing point in the captured image of the scene; and

performing one or more calibration operations for the at least one sensor based, at least in part, on the determined relative device orientation.

2. The method of claim 1, wherein determining the relative device orientation comprises:

determining a rotation matrix relating location coordinates in the image capturing unit'"'"'s frame of reference and the scene'"'"'s frame of reference.

3. The method of claim 1, wherein determining the relative device orientation based, at least in part, on the determined location of the at least one vanishing point comprises:

determining the relative device orientation based on determined locations of two or more vanishing points in the captured image of the scene.

4. The method of claim 1, further comprising:

determining actual device orientation based, at least in part, on the relative device orientation determined from the location of the at least one vanishing point in the captured image of the scene, and further based on a scene orientation value associated with the scene.

5. The method of claim 4, further comprising:

retrieving the scene orientation value from map data for a region including an area corresponding to the scene in the image captured by the image capturing unit.

6. The method of claim 1, further comprising:

determining actual device orientation based, at least in part, on the determined relative device orientation, and further based on a previously determined orientation value.

7. The method of claim 1, further comprising:

determining a change of device orientation based, at least in part, on relative device orientation values determined based, at least in part, on locations of vanishing points determined from a plurality of images captured by the image capturing unit over a period of time.

8. The method of claim 1, wherein the at least one sensor comprises one or more of:

a gyroscope, or a magnetometer.

9. The method of claim 1, wherein performing the one or more calibration operations for the at least one sensor comprises one of:

calibrating the at least one sensor in response to comparison results, produced from a comparison of a measured orientation determined by the at least one sensor and the determined relative device orientation, indicating that the at least one sensor requires calibration;

determining that the at least one sensor is calibrated in response to the comparison results indicating that the at least one sensor is calibrated;

calibrating a magnetometer, in response to the comparison results indicating that the magnetometer requires calibration, based on an indication of a presence of magnetic interference causing temporary erroneous measurements of the measured orientation;

or

determining that the magnetometer is calibrated based on the comparison results and further based on the indication of the presence of magnetic interference causing the temporary erroneous measurements of the measured orientation.

10. The method of claim 9, wherein calibrating the at least one sensor comprises:

determining for the at least one sensor, based on the comparison results, one or more of;

sensor bias, sensor scaling, or sensor misalignment.

11. The method of claim 1, wherein determining the relative device orientation comprises:

determining deviation of the at least one vanishing point from a center of the captured image for the scene.

12. A device comprising:

an image capturing unit;

at least one sensor to measure orientation of the device; and

a controller configured to;

determine relative device orientation of the device based, at least in part, on determined location of at least one vanishing point in an image of a scene captured by the image capturing unit; and

perform one or more calibration operations for the at least one sensor based on the determined relative device orientation.

13. The device of claim 12, wherein the controller configured to determine the relative device orientation is configured to:

determine a rotation matrix relating location coordinates in the image capturing unit'"'"'s frame of reference and the scene'"'"'s frame of reference.

14. The device of claim 12, wherein the controller configured to determine the relative device orientation based, at least in part, on the determined location of the at least one vanishing point is configured to:

determine the relative device orientation based on determined locations of two or more vanishing points in the captured image of the scene.

15. The device of claim 12, wherein the controller is further configured to:

determine actual device orientation based, at least in part, on the relative device orientation determined from the location of the at least one vanishing point in the captured image of the scene, and further based on a scene orientation value associated with the scene.

16. The device of claim 12, wherein the controller is further configured to:

determine a change of device orientation based, at least in part, on relative device orientation values determined based, at least in part, on locations of vanishing points determined from a plurality of images captured by the image capturing unit over a period of time.

17. The device of claim 12, wherein the at least one sensor comprises one or more of:

a gyroscope, or a magnetometer.

18. The device of claim 12, wherein the controller configured to perform the one or more calibration operations for the at least one sensor is configured to perform one of:

calibrate the at least one sensor in response to comparison results, produced from a comparison of a measured orientation determined by the at least one sensor and the determined relative device orientation, indicating that the at least one sensor requires calibration;

determine that the at least one sensor is calibrated in response to the comparison results indicating that the at least one sensor is calibrated;

calibrate a magnetometer, in response to the comparison results indicating that the magnetometer requires calibration, based on an indication of a presence of magnetic interference causing temporary erroneous measurements of the measured orientation;

or

determine that the magnetometer is calibrated based on the comparison results and further based on the indication of the presence of magnetic interference causing the temporary erroneous measurements of the measured orientation.

19. An apparatus comprising:

means for capturing an image of a scene for a device that includes at least one sensor;

means for determining relative device orientation of the device based, at least in part, on determined location of at least one vanishing point in the captured image of the scene; and

means for performing one or more calibration operations for the at least one sensor based, at least in part, on the determined relative device orientation.

20. The apparatus of claim 19, wherein the means for determining the relative device orientation comprise:

means for determining a rotation matrix relating location coordinates in a frame of reference of the means for capturing the image and the scene'"'"'s frame of reference.

21. The apparatus of claim 19, wherein the means for determining the relative device orientation based, at least in part, on the determined location of the at least one vanishing point comprise:

means for determining the relative device orientation based on determined locations of two or more vanishing points in the captured image of the scene.

22. The apparatus of claim 19, further comprising:

means for determining actual device orientation based, at least in part, on the relative device orientation determined from the location of the at least one vanishing point in the captured image of the scene, and further based on a scene orientation value associated with the scene.

23. The apparatus of claim 19, further comprising:

means for determining a change of device orientation based, at least in part, on relative device orientation values determined based, at least in part, on locations of vanishing points determined from a plurality of images captured by the means for capturing the image over a period of time.

24. The apparatus of claim 19, wherein the at least one sensor comprises one or more of:

a gyroscope, or a magnetometer.

25. The apparatus of claim 19, wherein the means for performing the one or more calibration operations for the at least one sensor comprise means for performing one of:

calibrating the at least one sensor in response to comparison results, produced from a comparison of a measured orientation determined by the at least one sensor and the determined relative device orientation, indicating that the at least one sensor requires calibration;

determining that the at least one sensor is calibrated in response to the comparison results indicating that the at least one sensor is calibrated;

calibrating a magnetometer, in response to the comparison results indicating that the magnetometer requires calibration, based on an indication of a presence of magnetic interference causing temporary erroneous measurements of the measured orientation;

or

determining that the magnetometer is calibrated based on the comparison results and further based on the indication of the presence of magnetic interference causing the temporary erroneous measurements of the measured orientation.

26. A non-transitory processor readable media programmed with a set of instructions executable on a processor that, when executed, cause operations comprising:

capturing an image of a scene by an image capturing unit of a device that includes at least one sensor;

determining relative device orientation of the device based, at least in part, on determined location of at least one vanishing point in the captured image of the scene; and

performing one or more calibration operations for the at least one sensor based, at least in part, on the determined relative device orientation.

27. The processor readable media of claim 26, wherein determining the relative device orientation comprises:

determining a rotation matrix relating location coordinates in the image capturing unit'"'"'s frame of reference and the scene'"'"'s frame of reference.

28. The processor readable media of claim 26, wherein determining the relative device orientation based, at least in part, on the determined location of the at least one vanishing point comprises:

determining the relative device orientation based on determined locations of two or more vanishing points in the captured image of the scene.

29. The processor readable media of claim 26, further comprising instructions to cause further operations comprising:

determining actual device orientation based, at least in part, on the relative device orientation determined from the location of the at least one vanishing point in the captured image of the scene, and further based on a scene orientation value associated with the scene.

30. The processor readable media of claim 26, further comprising instructions to cause further operations comprising:

determining a change of device orientation based, at least in part, on relative device orientation values determined based, at least in part, on locations of vanishing points determined from a plurality of images captured by the image capturing unit over a period of time.

31. The processor readable media of claim 26, wherein the at least one sensor comprises one or more of:

a gyroscope, or a magnetometer.

32. The processor readable media of claim 26, wherein performing the one or more calibration operations for the at least one sensor comprises one of:

calibrating the at least one sensor in response to comparison results, produced from a comparison of a measured orientation determined by the at least one sensor and the determined relative device orientation, indicating that the at least one sensor requires calibration;

determining that the at least one sensor is calibrated in response to the comparison results indicating that the at least one sensor is calibrated;

calibrating a magnetometer, in response to the comparison results indicating that the magnetometer requires calibration, based on an indication of a presence of magnetic interference causing temporary erroneous measurements of the measured orientation;

or

determining that the magnetometer is calibrated based on the comparison results and further based on the indication of the presence of magnetic interference causing the temporary erroneous measurements of the measured orientation.

33. A method comprising:

capturing an image, by an image capturing unit of a device, of a scene associated with a scene orientation value, the device further comprising at least one sensor;

determining location of at least one vanishing point in the captured image of the scene;

determining a measured device orientation of the device based on one or more measurements performed by the at least one sensor of the device; and

determining candidate positions of the device in a region including an area corresponding to the scene based, at least in part, on the location of the at least one vanishing point in the captured image of the scene, the measured device orientation, and orientation data associated with multiple areas within the region.

34. The method of claim 33, further comprising:

in response to determining that the determined location of the at least one vanishing point in the captured image is at an off-center location, generating an indication to cause the device to be moved such that a subsequent vanishing point in a subsequent captured image will be located substantially at the center of the subsequent captured image.

35. The method of claim 33, wherein determining the candidate positions comprises:

determining candidate hallways, in an area including multiple hallways associated with respective orientation values, based on relative orientation of the device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation.

36. The method of claim 33, further comprising:

retrieving the orientation data associated with the multiple areas within the region from map data for the region.

37. A device comprising:

an image capturing unit;

at least one sensor to measure orientation of the device; and

a controller configured to;

determine location of at least one vanishing point in an image of a scene captured by the image capturing unit;

determine a measured device orientation of the device based on one or more measurements performed by the at least one sensor of the device; and

determine candidate positions of the device in a region including an area corresponding to the scene based, at least in part, on the location of the at least one vanishing point in the captured image of the scene, the measured device orientation, and orientation data associated with multiple areas within the region.

38. The device of claim 37, wherein the controller is further configured to:

in response to determining that the determined location of the at least one vanishing point in the captured image is at an off-center location, generate an indication to cause the device to be moved such that a subsequent vanishing point in a subsequent captured image will be located substantially at the center of the subsequent captured image.

39. The device of claim 37, wherein the controller configured to determine the candidate positions is configured to:

determine candidate hallways, in an area including multiple hallways associated with respective orientation values, based on relative orientation of the device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation.

40. The device of claim 37, wherein the controller is further configured to:

retrieve the orientation data associated with the multiple areas within the region from map data for the region.

41. An apparatus comprising:

means for capturing an image of a scene for a device that includes at least one sensor;

means for determining location of at least one vanishing point in the captured image of the scene;

means for determining a measured device orientation of the device based on one or more measurements performed by the at least one sensor of the device; and

means for determining candidate positions of the device in a region including an area corresponding to the scene based, at least in part, on the location of the at least one vanishing point in the captured image of the scene, the measured device orientation, and orientation data associated with multiple areas within the region.

42. The apparatus of claim 41, further comprising:

means for generating an indication to cause the device to be moved, in response to determining that the determined location of the at least one vanishing point in the captured image is at an off-center location, such that a subsequent vanishing point in a subsequent captured image will be located substantially at the center of the subsequent captured image.

43. The apparatus of claim 41, wherein the means for determining the candidate positions comprise:

means for determining candidate hallways, in an area including multiple hallways associated with respective orientation values, based on relative orientation of the device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation.

44. The apparatus of claim 41, further comprising:

means for retrieving the orientation data associated with the multiple areas within the region from map data for the region.

45. A non-transitory processor readable media programmed with a set of instructions executable on a processor that, when executed, cause operations comprising:

capturing an image, by an image capturing unit of a device, of a scene associated with a scene orientation value, the device further comprising at least one sensor;

determining location of at least one vanishing point in the captured image of the scene;

determining a measured device orientation of the device based on one or more measurements performed by the at least one sensor of the device; and

determining candidate positions of the device in a region including an area corresponding to the scene based, at least in part, on the location of the at least one vanishing point in the captured image of the scene, the measured device orientation, and orientation data associated with multiple areas within the region.

46. The processor readable media of claim 45, further comprising instructions to cause further operations comprising:

generating an indication to cause the device to be moved, in response to determining that the determined location of the at least one vanishing point in the captured image is at an off-center location, such that a subsequent vanishing point in a subsequent captured image will be located substantially at the center of the subsequent captured image.

47. The processor readable media of claim 45, wherein determining the candidate positions comprises:

determining candidate hallways, in an area including multiple hallways associated with respective orientation values, based on relative orientation of the device determined from the location of the at least one vanishing point in the captured image, and based on the measured device orientation.

48. The processor readable media of claim 45, further comprising instructions to cause further operations comprising:

retrieving the orientation data associated with the multiple areas within the region from map data for the region.