CAMERA AND INERTIAL MEASUREMENT UNIT INTEGRATION WITH NAVIGATION DATA FEEDBACK FOR FEATURE TRACKING

• US 20120078510A1
• Filed: 09/24/2010
• Published: 03/29/2012
• Est. Priority Date: 09/24/2010
• Status: Active Grant

• Current Assignee: Honeywell International Inc.
• Original Assignee: Honeywell International Inc.
• Inventors: Ma, Yunqian, Hawkinson, Wesley J.
• Granted Patent: US 8,676,498 B2
• Time in Patent Office: Days
• US Class Current: 701/426
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165

Abstract

A navigation device is provided herein comprising an inertial measurement unit (IMU), a camera, and a processor. The IMU provides an inertial measurement to the processor and the camera provides at least one image frame to the processor. The processor is configured to determine navigation data based on the inertial measurement and the at least one image frame, wherein at least one feature is extracted from the at least one image frame based on the navigation data.

Claims

1. A navigation device, comprising:

an inertial measurement unit (IMU) that provides inertial measurement to a processor; and

a camera that provides at least one image frame to the processor;

wherein the processor is configured to determine navigation data based on the inertial measurement and the at least one image frame, wherein at least one feature is extracted from the at least one image frame based on the navigation data.

2. The navigation device of claim 1, wherein the processor determines navigation data is further based on tracking the at least one feature from the at least one image frame to a subsequent image frame.

3. The navigation device of claim 1, wherein the processor is further configured to compensate drift errors in the IMU based on the position of the at least one feature in the at least one image frame.

4. The navigation device of claim 1, wherein the processor is configured to execute software, wherein the software comprises a feature extraction and data association functionality that narrows a search of the at least one feature in a subsequent image frame based on a predicted location of the at least one feature in the subsequent image frame due to a position of the camera determined from the navigation data.

5. The navigation device of claim 1, wherein the navigation data is at least one of position, attitude, and velocity of the navigation device.

6. The navigation device of claim 1, further comprising:

a display device that displays at least a portion of the navigation data.

7. The navigation device of claim 1, wherein the camera is an omnidirectional camera.

8. The navigation device of claim 1, wherein the camera comprises an array of cameras that each generate a sub-image, wherein the at least one feature is tracked between at least two of the sub-images.

9. A camera aided inertial navigation system, comprising:

at least one monocular camera that captures at least a first image frame;

an inertial measurement unit integrated with the at least one monocular camera, wherein the inertial measurement unit takes inertial measurements; and

a processor coupled to the at least one monocular camera and the inertial measurement unit, wherein the processor determines a position of at least one feature in the first image frame based on an output from processing the inertial measurements with a previous position of the at least one feature.

10. The system of claim 9, wherein the processor further compensates the inertial measurements based on the position of the at least one feature.

11. The system of claim 10, wherein the processor further resets at least one state of a filter based at least on the at least one feature using a weighted average of the inertial measurements and the at least one feature.

12. The system of claim 9, wherein:

the inertial measurement unit takes inertial measurements at a first frequency; and

the at least one monocular camera captures images at a second frequency, wherein the second frequency is lower than the first frequency.

13. The system of claim 12, further comprising a hybrid extended Kalman filter software functionality executed by the processor, wherein the hybrid extended Kalman filter software filters the inertial measurements and the previous position of the at least one feature.

14. The system of claim 13, wherein:

the hybrid extended Kalman filter determines a state estimate and covariance estimate based on the inertial measurements, wherein the state estimate and covariance estimate are determined at the second frequency; and

the hybrid extended Kalman filter determines a gain computation, a state estimate update, and a covariance estimate update at discrete times based on the at least one feature.

15. The system of claim 9, wherein the processor is configured to execute software, wherein the software comprises a feature extraction and data association functionality that narrows a search of the at least one feature in a second image frame based on a location of the camera determined from the navigation data.

16. The system of claim 9, further comprising:

a storage medium containing a landmark database, wherein feature data relating to the at least one feature and other extracted features are stored.

17. A navigation device, comprising:

a monocular camera configured to capture image frames at discrete time intervals;

an inertial measurement unit configured to measure angular rate and acceleration; and

a processor coupled to the monocular camera and the inertial measurement unit, wherein the processor is configured to execute software, wherein the software comprises a feature extraction and data association functionality and a filter functionality that;

tracks one or more features from a first image frames captured at a first location and from a second image frame captured at a second location;

determines a physical displacement of the navigation device based on the measured angular rate and acceleration;

determines a position correction based on the relative movement of the one or more features between the first image frame and the second image frame; and

outputs navigation data based on the physical displacement and the position correction; and

wherein feedback from the navigation data is used for tracking the one or more features.

18. The navigation device of claim 17, further comprising:

a display device that displays the navigation data; and

a storage medium containing a landmark database, wherein feature data relating to the at least one feature and other extracted features are stored when detected in the first image frame and deleted when no longer detected in the second image frame;

wherein the monocular camera comprises a plurality of monocular cameras arrayed together such that the plurality of monocular cameras output a combined field of view that is greater than the field of view of any of the individual monocular cameras.

19. The navigation device of claim 17, wherein the measured angular rate and acceleration are compensated for based on the position correction.

20. The navigation device of claim 17, wherein the feature extraction and data association functionality executed by the processor determines an approximate location of the one or more features within the second image frame based on the angular rate and acceleration.