Camera aided motion direction and speed estimation

• US 9,330,471 B2
• Filed: 02/14/2013
• Issued: 05/03/2016
• Est. Priority Date: 02/14/2013
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Venkatraman, Subramaniam, Kulik, Victor
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Gilliard, Delomia L
• Application Number:
  US13/767,755
• Publication Number:
  US 20140226864A1
• Time in Patent Office: 1,174 Days
• Field of Search: 382/107
• US Class Current: 1/1
• CPC Class Codes: G01C 21/12 executed aboard the object being navigated; Dead reckoning
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51

Abstract

Methods and devices for camera aided motion direction and speed estimation are disclosed. The method of determining position characteristics of a mobile device comprises capturing a plurality of images that represent views from the mobile device, adjusting perspectives of the plurality of images based at least in part on an orientation of the mobile device, determining a misalignment angle with respect to a direction of motion of the mobile device using the plurality of images, and storing the misalignment angle and the direction of motion in a storage device.

Claims

1. A method of determining position characteristics of a mobile device, comprising:

capturing a plurality of images that represent views from the mobile device;

tracking features in the plurality of images;

adjusting asymmetric distortion of the features in the plurality of images based at least in part on an orientation of the mobile device, wherein adjusting asymmetric distortion of the features in the plurality of images comprises compensating for asymmetric distortion of the features in the plurality of images using an area based on centers of the plurality of images;

determining a misalignment angle with respect to a direction of motion of the mobile device using the adjusted plurality of images; and

storing the misalignment angle and the direction of motion in a storage device.

2. The method of claim 1, wherein adjusting asymmetric distortion of the features in the plurality of images further comprises:

compensating for asymmetric distortion of the features in the plurality of images based at least in part on a weighted average of locations of features in the plurality of images, wherein features around the centers of the plurality of images are given more weights and features away from the centers of the plurality of images are given less weights.

3. The method of claim 1, wherein determining the misalignment angle comprises:

estimating direction of motion of the mobile device;

estimating the orientation of the mobile device using sensor data; and

determining the misalignment angle based at least in part on the direction of motion and the orientation of the mobile device.

4. The method of claim 3, wherein the tracking features from the plurality of images comprises:

rejecting outliers in features of the plurality of images to eliminate at least one moving object in the plurality of images.

5. The method of claim 1, further comprising at least one of:

determining a confidence of the misalignment angle with respect to the direction of motion of the mobile device using information provided by a gyroscope of the mobile device;

determining the confidence of the misalignment angle with respect to the direction of motion of the mobile device using information provided by a magnetometer of the mobile device;

determining the confidence of the misalignment angle with respect to the direction of motion of the mobile device using features of the plurality of images;

or some combination thereof.

6. The method of claim 1 further comprises:

determining a speed estimation of the mobile device;

determining a confidence of the speed estimation of the mobile device; and

applying the speed estimation and the confidence of the speed estimation to navigate a user of the mobile device.

7. The method of claim 6, wherein determining a speed estimate comprises:

extracting features from the plurality of images;

computing an average displacement of the mobile device using the features from the plurality of images; and

computing the speed estimate based at least in part on the average displacement of the mobile device.

8. The method of claim 7, wherein computing the speed estimate comprises:

comparing the features from the plurality of images;

determining a separation of pixels between two consecutive images in the plurality of images;

determining a time interval between the two consecutive images; and

calculating the speed estimate of the mobile device in accordance with the separation of pixels and the time interval between the two consecutive images.

9. The method of claim 8 further comprises:

calibrating a height of the mobile device according to at least one of GNSS location information and WIFI location information of the mobile device.

10. The method of claim 1 further comprises:

applying the misalignment angle and a confidence of the misalignment angle to navigate a user of the mobile device.

11. An apparatus, comprising:

a control unit including processing logic, the processing logic comprising;

logic configured to capture a plurality of images that represent views from a mobile device;

logic configured to track features in the plurality of images;

logic configured to adjust asymmetric distortion of the features in the plurality of images based at least in part on an orientation of the mobile device, wherein adjusting asymmetric distortion of the features in the plurality of images comprises compensating for asymmetric distortion of the features in the plurality of images using an area based on centers of the plurality of images;

logic configured to determine a misalignment angle with respect to a direction of motion of the mobile device using the adjusted plurality of images; and

logic configured to store the misalignment angle and the direction of motion in a storage device.

12. The apparatus of claim 11, wherein the logic configured to adjust asymmetric distortion of the features in the plurality of images further comprises:

logic configured to compensate for asymmetric distortion of the features in the plurality of images based at least in part on a weighted average of locations of features in the plurality of images, wherein features around the centers of the plurality of images are given more weights and features away from the centers of the plurality of images are given less weights.

13. The apparatus of claim 11, wherein the logic configured to determine the misalignment angle comprises:

logic configured to estimate direction of motion of the mobile device;

logic configured to estimate the orientation of the mobile device using sensor data; and

logic configured to determine the misalignment angle based at least in part on the direction of motion and the orientation of the mobile device.

14. The apparatus of claim 13, wherein the logic configured to track features from the plurality of images comprises:

logic configured to reject outliers in features of the plurality of images to eliminate at least one moving object in the plurality of images.

15. The apparatus of claim 11, further comprising at least one of:

logic configured to determine a confidence of the misalignment angle with respect to the direction of motion of the mobile device using information provided by a gyroscope of the mobile device;

logic configured to determine the confidence of the misalignment angle with respect to the direction of motion of the mobile device using information provided by a magnetometer of the mobile device;

logic configured to determine the confidence of the misalignment angle with respect to the direction of motion of the mobile device using features of the plurality of images;

or some combination thereof.

16. The apparatus of claim 11 further comprises:

logic configured to determine a speed estimation of the mobile device;

logic configured to determine a confidence of the speed estimation of the mobile device; and

logic configured to apply the speed estimation and the confidence of the speed estimation to navigate a user of the mobile device.

17. The apparatus of claim 16, wherein the logic configured to determine a speed estimate comprises:

logic configured to extract features from the plurality of images;

logic configured to compute an average displacement of the mobile device using the features from the plurality of images; and

logic configured to compute the speed estimate based at least in part on the average displacement of the mobile device.

18. The apparatus of claim 17, wherein the logic configured to compute the speed estimate comprises:

logic configured to compare the features from the plurality of images;

logic configured to determine a separation of pixels between two consecutive images in the plurality of images;

logic configured to determine a time interval between the two consecutive images; and

logic configured to calculate the speed estimate of the mobile device in accordance with the separation of pixels and the time interval between the two consecutive images.

19. The apparatus of claim 18 further comprises:

logic configured to calibrate a height of the mobile device according to at least one of GNSS location information and WIFI location information of the mobile device.

20. The apparatus of claim 11 further comprises:

logic configured to apply the misalignment angle and a confidence of the misalignment angle to navigate a user of the mobile device.

21. A computer program product comprising non-transitory medium storing instructions for execution by one or more computer systems, the instructions comprising:

instructions for capturing a plurality of images that represent views from a mobile device;

instructions for tracking features in the plurality of images;

instructions for adjusting asymmetric distortion of the features in the plurality of images based at least in part on an orientation of the mobile device, wherein adjusting asymmetric distortion of the features in the plurality of images comprises compensating for asymmetric distortion of the features in the plurality of images using an area based on centers of the plurality of images;

instructions for determining a misalignment angle with respect to a direction of motion of the mobile device using the adjusted plurality of images; and

instructions for storing the misalignment angle and the direction of motion in a storage device.

22. The computer program product of claim 21, wherein the instructions for adjusting asymmetric distortion of the features in the plurality of images further comprises:

instructions for compensating for asymmetric distortion of the features in the plurality of images based at least in part on a weighted average of locations of features in the plurality of images, wherein features around the centers of the plurality of images are given more weights and features away from the centers of the plurality of images are given less weights.

23. The computer program product of claim 21, wherein the instructions for determining the misalignment angle comprises:

instructions for estimating direction of motion of the mobile device;

instructions for estimating the orientation of the mobile device using sensor data; and

instructions for determining the misalignment angle based at least in part on the direction of motion and the orientation of the mobile device.

24. The computer program product of claim 23, wherein the instructions for tracking features from the plurality of images comprises:

instructions for rejecting outliers in features of the plurality of images to eliminate at least one moving object in the plurality of images.

25. The computer program product of claim 21, further comprising at least one of:

instructions for determining a confidence of the misalignment angle with respect to the direction of motion of the mobile device using information provided by a gyroscope of the mobile device;

instructions for determining the confidence of the misalignment angle with respect to the direction of motion of the mobile device using information provided by a magnetometer of the mobile device;

instructions for determining the confidence of the misalignment angle with respect to the direction of motion of the mobile device using features of the plurality of images;

or some combination thereof.

26. The computer program product of claim 21 further comprises:

instructions for determining a speed estimation of the mobile device;

instructions for determining a confidence of the speed estimation of the mobile device; and

instructions for applying the speed estimation and the confidence of the speed estimation to navigate a user of the mobile device.

27. The computer program product of claim 26, wherein the instructions for determining a speed estimate comprises:

instructions for extracting features from the plurality of images;

instructions for computing an average displacement of the mobile device using the features from the plurality of images; and

instructions for computing the speed estimate based at least in part on the average displacement of the mobile device.

28. The computer program product of claim 27, wherein the instructions for computing the speed estimate comprises:

instructions for comparing the features from the plurality of images;

instructions for determining a separation of pixels between two consecutive images in the plurality of images;

instructions for determining a time interval between the two consecutive images; and

instructions for calculating the speed estimate of the mobile device in accordance with the separation of pixels and the time interval between the two consecutive images.

29. The computer program product of claim 28 further comprises:

instructions for calibrating a height of the mobile device according to at least one of GNSS location information and WIFI location information of the mobile device.

30. The computer program product of claim 21 further comprises:

instructions for applying the misalignment angle and a confidence of the misalignment angle to navigate a user of the mobile device.

31. A system, comprising:

means for capturing a plurality of images that represent views from a mobile device;

means for tracking features in the plurality of images;

means for adjusting asymmetric distortion of the features in the plurality of images based at least in part on an orientation of the mobile device, wherein adjusting asymmetric distortion of the features in the plurality of images comprises compensating for asymmetric distortion of the features in the plurality of images using an area based on centers of the plurality of images;

means for determining a misalignment angle with respect to a direction of motion of the mobile device using the adjusted plurality of images; and

means for storing the misalignment angle and the direction of motion in a storage device.

32. The system of claim 31, wherein the means for adjusting asymmetric distortion of the features in the plurality of images further comprises:

means for compensating for asymmetric distortion of the features in the plurality of images based at least in part on a weighted average of locations of features in the plurality of images, wherein features around the centers of the plurality of images are given more weights and features away from the centers of the plurality of images are given less weights.

33. The system of claim 31, wherein the means for determining the misalignment angle comprises:

means for estimating direction of motion of the mobile device;

means for estimating the orientation of the mobile device using sensor data; and

means for determining the misalignment angle based at least in part on the direction of motion and the orientation of the mobile device.

34. The system of claim 31, further comprising at least one of:

means for determining a confidence of the misalignment angle with respect to the direction of motion of the mobile device using information provided by a gyroscope of the mobile device;

means for determining the confidence of the misalignment angle with respect to the direction of motion of the mobile device using information provided by a magnetometer of the mobile device;

means for determining the confidence of the misalignment angle with respect to the direction of motion of the mobile device using features of the plurality of images;

or some combination thereof.

35. The system of claim 31 further comprises:

means for determining a speed estimation of the mobile device;

means for determining a confidence of the speed estimation of the mobile device; and

means for applying the speed estimation and the confidence of the speed estimation to navigate a user of the mobile device.