Camera-based inertial sensor alignment for PND

• US 9,160,980 B2
• Filed: 06/30/2011
• Issued: 10/13/2015
• Est. Priority Date: 01/11/2011
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Garin, Lionel J.
• Primary Examiner(s): WONG, ALLEN C
• Application Number:
  US13/173,261
• Publication Number:
  US 20120176492A1
• Time in Patent Office: 1,566 Days
• Field of Search: 348/116
• US Class Current: 1/1
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 25/005 initial alignment, calibration or starting-up of inertial devices
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

An apparatus and method to enhance dead-reckoning navigation using inertial sensor measurements based on images from a camera are disclosed. A camera build into a mobile device is used to calibrate inertial sensors and rotation matrices. Images from a camera may be used (1) to remove a gravitational element from accelerometer measurements; (2) to set a scaling factor and an offset for a gyrometer; and (3) to set initial and updated values for rotation matrices.

Claims

1. A mobile device using images for navigation, the mobile device comprising:

a camera, wherein the camera acquires a sequence of images;

a memory to hold three rotation matrices comprising;

a first rotation matrix, wherein the first rotation matrix is for converting between a body reference system of the mobile device and a platform reference system of a vehicle;

a second rotation matrix, wherein the second rotation matrix is for converting between the platform reference system of the vehicle and a local reference system; and

a third rotation matrix, wherein the third rotation matrix is for converting between the body reference system and the local reference system; and

a processor coupled to the camera and the memory, wherein the processor is configured to;

set an initial value of the first rotation matrix based on the sequence of images; and

set an initial value of the third rotation matrix based on the sequence of images.

2. The mobile device of claim 1, further comprising:

an accelerometer to provide accelerometer measurements;

wherein the processor is further coupled to the accelerometer; and

wherein the processor is further configured to set an initial value of a gravitational vector based on the sequence of images, wherein the gravitational vector is used to null an effect of gravity in the accelerometer measurements.

3. The mobile device of claim 2, the accelerometer comprises a three-dimensional accelerometer.

4. The mobile device of claim 1, further comprising:

a gyrometer functionally coupled to the camera;

wherein the processor is further coupled to the gyrometer.

5. The mobile device of claim 4, wherein the processor is further configured to update calibration of the gyrometer based on the sequence of images.

6. The mobile device of claim 1, further comprising a position location module.

7. The mobile device of claim 6, wherein the position location module comprises a global navigation satellite system receiver (GNSS receiver).

8. The mobile device of claim 7, wherein the GNSS receiver comprises a global positioning satellite (GPS) receiver.

9. The mobile device of claim 1, wherein the mobile device is mechanically coupled to the vehicle.

10. The mobile device of claim 1, wherein the processor is further configured to perform dead reckoning based on the sequence of images.

11. The mobile device of claim 1, further comprising:

an accelerometer to provide accelerometer measurements; and

a gyrometer functionally coupled to the camera;

wherein the processor is further coupled to the accelerometer and the gyrometer; and

wherein the processor is further configured to;

set an initial value of a gravitational vector based on the sequence of images, wherein the gravitational vector is used to null an effect of gravity in acceleration measurements; and

update calibration of the gyrometer based on the sequence of images.

12. A method in a mobile device of using images for navigation, the method comprising:

acquiring a sequence of images from a camera;

setting an initial value of a first rotation matrix based on the sequence of images, wherein the first rotation matrix is for converting between a body reference system of the mobile device and a platform reference system of a vehicle;

setting an initial value of a second rotation matrix based on the sequence of images, wherein the second rotation matrix is for converting between the platform reference system of the vehicle and a local reference system;

setting an initial value of a third rotation matrix based on the sequence of images, wherein the third rotation matrix is for converting between the body reference system and the local reference system;

writing the second rotation matrix to memory; and

converting measurements, between a body reference system and a local reference system, using the third rotation matrix.

13. The method of claim 12, further comprising

setting an initial value of a gravitational vector based on the sequence of images;

wherein the gravitational vector is used to null an effect of gravity in accelerometer measurements.

14. The method of claim 12, further comprising updating a calibration of a gyrometer based on the sequence of images.

15. The method of claim 12, further comprising mechanically coupling the mobile device to a vehicle.

16. The method of claim 12, wherein acquiring the sequence of images comprises resampling the sequence of images, having a first resolution, to a second resolution, wherein the second resolution is lower than the first resolution.

17. The method of claim 12, wherein acquiring the sequence of images comprises extracting intensity values from the sequence of images.

18. The method of claim 12, wherein setting the initial value of the second rotation matrix based on the sequence of images comprises determining a Focus of Expansion point (FOE point) derived from at least two of the sequence of images to set a direction of movement of the mobile device in a body reference system, and further comprising:

determining a gravitational vector in the body reference system; and

computing the first rotation matrix based on;

(1) the third rotation matrix; and

(2) an inverse of the second rotation matrix.

19. The method of claim 18, wherein the FOE point indicates a height angle (α

) and a width angle (β

) relative to the body reference system.

20. The method of claim 18, wherein the FOE point indicates a point in an X-Y image plane relative to the body reference system.

21. The method of claim 18, wherein determining the FOE point comprises:

extracting a plurality of features from a first image of the sequence of images thereby defining a corresponding first plurality of points;

tracking the plurality of features on a second image of the sequence of images thereby defining a corresponding second plurality of points and further defining a corresponding plurality of lines between the corresponding first plurality of points and the corresponding second plurality of points; and

setting the FOE point at a center of the corresponding plurality of lines.

22. The method of claim 21, wherein the center of the corresponding plurality of lines comprises a least-squares error point of the corresponding plurality of lines.

23. The method of claim 21, wherein the center of the corresponding plurality of lines is based on Kalman filter estimate.

24. The method of claim 12, wherein acquiring the sequence of images comprises correcting the sequence of images to remove a lens aberration.

25. A mobile device using images for navigation, the mobile device comprising:

means for acquiring a sequence of images from a camera;

means for setting an initial value of a first rotation matrix based on the sequence of images used to convert from a body reference system of the mobile device and a platform reference system of a vehicle;

means for setting an initial value of a second rotation matrix based on the sequence of images, wherein the second rotation matrix is for converting between the platform reference system of the vehicle and a local reference system;

means for setting an initial value of a third rotation matrix based on the sequence of images, wherein the third rotation matrix is for converting between the body reference system and the local reference system;

means for writing the third rotation matrix to memory; and

means for converting measurements, between the body reference system of the mobile device and a local reference system, using the third rotation matrix.

26. A non-transitory computer-readable medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform a method for using images for navigation in a mobile device having a camera, memory and a processor, said method comprising operations of:

acquiring a sequence of images from a camera;

setting an initial value of a first rotation matrix based on the sequence of images, wherein the first rotation matrix is for converting between a body reference system of the mobile device and a platform reference system of a vehicle;

setting an initial value of a second rotation matrix based on the sequence of images, wherein the second rotation matrix is for converting between the platform reference system of the vehicle and a local reference system;

setting an initial value of a third rotation matrix based on the sequence of images, wherein the third rotation matrix is for converting between the body reference system and the local reference system;

writing the third rotation matrix to memory; and

converting measurements, between the body reference system of the mobile device and a local reference system, using the third rotation matrix.