Orientation determination based on vanishing point computation

• US 9,113,077 B2
• Filed: 01/17/2013
• Issued: 08/18/2015
• Est. Priority Date: 01/17/2013
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Mirzaei, Faraz Mohammad, Chao, Hui, Das, Saumitra Mohan
• Primary Examiner(s): Giles, Nicholas
• Assistant Examiner(s): Tissire, Abdelaaziz
• Application Number:
  US13/744,217
• Publication Number:
  US 20140198227A1
• Time in Patent Office: 943 Days
• Field of Search: 348/208.2, 348/187, 348/36, 348/42, 348/46
• US Class Current: 1/1
• CPC Class Codes: G01C 21/12 executed aboard the object being navigated; Dead reckoning
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/242 by image rotation, e.g. by 90 degrees
• CPC Class Codes: H04N 23/68 for stable pick-up of the scene, e.g. compensating for camera body vibrations
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors

Abstract

Disclosed are systems, apparatus, devices, methods, computer program products, and other implementations, including a method that includes capturing an image of a scene by an image capturing unit of a device that includes an accelerometer, and determining orientation of the device based, at least in part, on determined location of a vanishing point (e.g., a single vanishing point) in the captured image of the scene and at least one measurement obtained from the accelerometer.

Claims

1. A method comprising:

capturing an image of a scene by an image capturing unit of a device that includes an accelerometer;

deriving a gravity vector from at least one measurement obtained from the accelerometer;

determining a location of a vanishing point in the captured image of the scene;

computing a directional vector based on the determined location of the vanishing point in the captured image of the scene;

in response to a determination that the computed directional vector is substantially parallel to the gravity vector, re-computing the directional vector based on another location of another vanishing point determined to be present in the captured image of the scene; and

determining an orientation of the device based, at least in part, on the computed directional vector and the gravity vector derived from the at least one measurement obtained from the accelerometer.

2. The method of claim 1, wherein determining the orientation of the device based on the computed directional vector and the gravity vector comprises:

determining the orientation based, at least in part, on a vector determined from location of a single vanishing point in the captured image and the gravity vector.

3. The method of claim 1, wherein determining the orientation of the device comprises:

determining the orientation of the device relative to a frame of reference of an indoor area in which the device is located.

4. The method of claim 3, further comprising:

normalizing the directional vector corresponding to the determined location of the vanishing point based on an intrinsic calibration matrix of the image capturing unit.

5. The method of claim 4, further comprising:

aligning an axis of the frame of reference of the indoor area to the normalized vanishing point vector; and

determining the orientation of the device based on the aligned axis of the frame of reference of the indoor area.

6. The method of claim 3, wherein determining the orientation of the device comprises:

deriving a rotation matrix to transform location coordinates in the frame of reference of the indoor area to a frame of reference of the image capturing unit of the device.

7. The method of claim 6, further comprising:

determining based on the rotation matrix one or more of;

roll of the device, pitch of the device, or yaw of the device.

8. The method of claim 6, further comprising:

performing QR decomposition on the rotation matrix to obtain a resultant orthonormal rotation matrix.

9. The method of claim 1, further comprising:

performing one or more calibration operations for at least one of a gyroscope or a magnetometer of the device based, at least in part, on the device orientation determined based, at least in part, on the computed directional vector and the gravity vector.

10. The method of claim 9, wherein performing the one or more calibration operations for at least one of the gyroscope or the magnetometer comprises one or more of:

comparing gyroscope-measured orientation with the device orientation determined based, at least in part, on the computed directional vector and the gravity vector to obtain gyroscope-comparison results, and computing one or more of gyroscope bias, gyroscope scaling, or gyroscope misalignment based on the gyroscope-comparison results;

or

comparing magnetometer-measured orientation with the device orientation determined based, at least in part, on the computed directional vector and the gravity vector to obtain magnetometer-comparison results, and computing one or more of magnetometer bias, magnetometer scaling, or magnetometer misalignment based on the magnetometer-comparison results.

11. The method of claim 1, wherein determining the orientation of the device comprises:

identifying line segments in the captured image of the scene; and

determining the location of the vanishing point in the captured image based on the identified line segments.

12. The method of claim 1, further comprising:

computing the directional vector based on a further vanishing point in a subsequently captured image of the scene when the computed directional vector computed based on the vanishing point in the captured image of the scene is determined to be substantially parallel to the gravity vector, and in response to a determination that an additional vanishing point in the captured image of the scene is not available.

13. A device comprising:

an image capturing unit;

an accelerometer; and

a controller configured to;

determine a location of a vanishing point in an image of a scene captured by the image capturing unit;

obtain at least one measurement performed by the accelerometer;

derive a gravity vector from the at least one measurement obtained from the accelerometer;

compute a directional vector based on the determined location of the vanishing point in the captured image of the scene

re-compute the directional vector based on another location of another vanishing point determined to be present in the captured image of the scene in response to a determination that the computed directional vector is substantially parallel to the gravity vector; and

determine an orientation of the device based, at least in part, on the computed directional vector and the gravity vector derived from the at least one measurement obtained from the accelerometer.

14. The device of claim 13, wherein the controller configured to determine the orientation of the device based on the computed directional vector and the gravity vector is configured to:

determine the orientation based, at least in part, on a vector determined from location of a single vanishing point in the captured image and the gravity vector.

15. The device of claim 13, wherein the controller configured to determine the orientation of the device is configured to:

determine the orientation of the device relative to a frame of reference of an indoor area in which the device is located.

16. The device of claim 15, wherein the controller configured to determine the orientation of the device is configured to:

derive a rotation matrix to transform location coordinates in the frame of reference of the indoor area to a frame of reference of the image capturing unit of the device.

17. The device of claim 16, wherein the controller is further configured to:

determine based on the rotation matrix one or more of;

roll of the device, pitch of the device, or yaw of the device.

18. The device of claim 13, wherein the controller is further configured to:

normalize the directional vector corresponding to the determined location of the vanishing point based on an intrinsic calibration matrix of the image capturing unit.

19. The device of claim 13, wherein the controller is further configured to:

perform one or more calibration operations for at least one of a gyroscope or a magnetometer of the device based, at least in part, on the device orientation determined based, at least in part, on the computed directional vector and the gravity vector.

20. The device of claim 13, wherein the controller configured to determine the orientation of the device is configured to:

identify line segments in the captured image of the scene; and

determine the location of the vanishing point in the captured image based on the identified line segments.

21. An apparatus comprising:

means for capturing an image of a scene;

means for deriving a gravity vector from at least one measurement obtained from an accelerometer of a device;

means for determining a location of a vanishing point in the captured image of the scene;

means for computing a directional vector based on the determined location of the vanishing point in the captured image of the scene

means for re-computing the directional vector based on another location of another vanishing point determined to be present in the captured image of the scene in response to a determination that the computed directional vector is substantially parallel to the gravity vector; and

means for determining an orientation of the device based, at least in part, on the computed directional vector and the gravity vector derived from the at least one measurement obtained from the accelerometer.

22. The apparatus of claim 21, wherein the means for determining the orientation of the device based on the computed directional vector and the gravity vector comprise:

means for determining the orientation based, at least in part, on a vector determined from location of a single vanishing point in the captured image and the gravity vector.

23. The apparatus of claim 21, wherein the means for determining the orientation of the device comprise:

means for determining the orientation of the device relative to a frame of reference of an indoor area in which the device is located.

24. The apparatus of claim 23, wherein the means for determining the orientation of the device comprise:

means for deriving a rotation matrix to transform location coordinates in the frame of reference of the indoor area to a frame of reference of the means for capturing the image.

25. The apparatus of claim 24, further comprising:

means for determining based on the rotation matrix one or more of;

roll of the device, pitch of the device, or yaw of the device.

26. The apparatus of claim 21, further comprising:

means for normalizing the directional vector corresponding to the determined location of the vanishing point based on an intrinsic calibration matrix of the means for capturing the image.

27. The apparatus of claim 21, further comprising:

means for performing one or more calibration operations for at least one of a gyroscope or a magnetometer of the device based, at least in part, on the device orientation determined based, at least in part, on the computed directional vector and the gravity vector.

28. The apparatus of claim 21, wherein the means for determining the orientation of the device comprise:

means for identifying line segments in the captured image of the scene; and

means for determining the location of the vanishing point in the captured image based on the identified line segments.

29. A non-transitory processor readable media programmed with a set of instructions executable on a processor that, when executed, cause operations comprising:

capturing an image of a scene by an image capturing unit of a device that includes an accelerometer;

deriving a gravity vector from at least one measurement obtained from the accelerometer;

determining a location of a vanishing point in the captured image of the scene;

computing a directional vector based on a determined location of a vanishing point in the captured image of the scene;

in response to a determination that the computed directional vector is substantially parallel to the gravity vector, re-computing the directional vector based on another location of another vanishing point determined to be present in the captured image of the scene; and

determining an orientation of the device based, at least in part, on the computed directional vector and the gravity vector derived from the at least one measurement obtained from the accelerometer.

30. The processor readable media of claim 29, wherein determining the orientation of the device based on the computed directional vector and the gravity vector comprises:

determining the orientation based, at least in part, on a vector determined from location of a single vanishing point in the captured image and the gravity vector.

31. The processor readable media of claim 29, wherein determining the orientation of the device comprises:

determining the orientation of the device relative to a frame of reference of an indoor area in which the device is located.

32. The processor readable media of claim 31, wherein determining the orientation of the device comprises:

deriving a rotation matrix to transform location coordinates in the frame of reference of the indoor area to a frame of reference of the image capturing unit of the device.

33. The processor readable media of claim 32, further comprising instructions to cause further operations comprising:

determining based on the rotation matrix one or more of;

roll of the device, pitch of the device, or yaw of the device.

34. The processor readable media of claim 29, further comprising instructions to cause further operations comprising:

normalizing the directional vector corresponding to the determined location of the vanishing point based on an intrinsic calibration matrix of the image capturing unit.

35. The processor readable media of claim 29, further comprising instructions to cause further operations comprising:

performing one or more calibration operations for at least one of a gyroscope or a magnetometer of the device based, at least in part, on the device orientation determined based, at least in part, on the computed directional vector and the gravity vector.

36. The processor readable media of claim 29, wherein determining the orientation of the device comprises:

identifying line segments in the captured image of the scene; and

determining the location of the vanishing point in the captured image based on the identified line segments.