Constrained key frame localization and mapping for vision-aided inertial navigation

• US 9,607,401 B2
• Filed: 05/07/2014
• Issued: 03/28/2017
• Est. Priority Date: 05/08/2013
• Status: Active Grant

• Current Assignee: Regents of The University of Minnesota (University of Minnesota)
• Original Assignee: Regents of The University of Minnesota (University of Minnesota)
• Inventors: Nerurkar, Esha D., Roumeliotis, Stergios I.
• Primary Examiner(s): Chio, Tat
• Application Number:
  US14/271,971
• Publication Number:
  US 20140333741A1
• Time in Patent Office: 1,056 Days
• US Class Current: 1/1
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

Estimation techniques for vision-aided inertial navigation are described. In one example, a vision-aided inertial navigation system (VINS) comprises an image source to produce image data for a keyframe and one or more non-keyframes along a trajectory, the one or more non-keyframes preceding the keyframe along the trajectory. The VINS comprises an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes, and a processing unit comprising an estimator that processes the IMU data and the image data to compute state estimates of the VINS. The estimator computes the state estimates of the VINS for the keyframe by constraining the state estimates based on the IMU data and the image data for the one or more non-keyframes of the VINS without computing state estimates of the VINS for the one or more non-keyframes.

Claims

1. A vision-aided inertial navigation system comprising:

an image source to produce image data for a first keyframe, one or more non-keyframes and a second keyframe along a trajectory of the vision-aided inertial navigation system (VINS), the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;

an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes; and

a processing unit comprising an estimator that processes the IMU data and the image data to compute respective state estimates for a position and orientation of the VINS for the first keyframe and for the second keyframe,

wherein, when computing the state estimates, the estimator constrains the state estimates for the second keyframe relative to the state estimates for the first keyframe based on the IMU data and the image data from the one or more non-keyframes, and

wherein, when constraining the state estimates, the estimator treats a landmark observed within the image data for the first keyframe or the second keyframe as different from the same landmark observed within the image data for the non-keyframes.

2. The vision-aided inertial navigation system of claim 1,

wherein the estimator processes the IMU data and the image data associated with the non-keyframes to compute one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe, and

where in the estimator applies the one or more constraints when computing the state estimates of the VINS for the second keyframe.

3. The vision-aided inertial navigation system of claim 2, wherein the estimator computes each of the constraints as an estimate of motion from the first keyframe to the second keyframe, and a covariance indicating an uncertainty of the estimated motion.

4. The vision-aided inertial navigation system of claim 1, wherein the estimator computes the respective state estimates for the first keyframe and the second keyframe without computing state estimates for the position and the orientation of the VINS for each of the non-keyframes.

5. The vision-aided inertial navigation system of claim 1,

wherein the estimator computes the state estimates to include a respective estimated position of each landmark observed for the first keyframe and the second keyframe, and

wherein, when computing the state estimates for the second keyframe, the estimator constrains the position and orientation of the VINS within the second keyframe based on the IMU data and the image data from the one or more non-keyframes while disregarding dependencies for each of the landmarks with respect to landmarks observed within the image data for the one or more non-keyframes.

6. The vision-aided inertial navigation system of claim 1, wherein the processing unit computes estimates for positions of landmarks observed from the first keyframe and the second keyframe without computing estimates for positions of landmarks observed only from the non-keyframes for which a respective pose of the VINS is not computed.

7. The vision-aided inertial navigation system of claim 1,

wherein the processing unit maintains a state vector that specifies, for computation, state estimates the position and the orientation of the VINS for the first keyframe and one or more landmarks observed from the first keyframe or the second keyframe without including variables for non-keyframe poses or landmarks observed only from non-keyframes,

wherein the estimator unit iteratively processes the state vector to compute the state estimates, and

wherein, for each iteration, the estimator constrains updates for the state estimates for the position and orientation of the VINS and the landmarks observed from the keyframes for the VINS based on the IMU data and the image data for the one or more non-keyframes of the VINS.

8. The vision-aided inertial navigation system of claim 7,

wherein the estimator selects key frames along the trajectory for which respective state estimates are to be computed within the state vector, and

wherein the estimator selects the key frames based on a set of criteria comprising one or more of a distance traveled between two consecutive key poses and poses at which points of interest were detected within the image data.

9. The vision-aided inertial navigation system of claim 1, wherein the estimator builds a map of the environment to include the state estimates of the VINS.

10. The vision-aided inertial navigation system of claim 1, wherein the vision-aided inertial navigation system is integrated within a mobile phone or a robot.

11. A method for computing state estimates for a vision-aided inertial navigation system (VINS) comprising:

receiving image data produced by an image source of the vision-aided inertial navigation system for a first keyframe, one or more non-keyframes and a second keyframe along a trajectory of the vision-aided inertial navigation system (VINS), the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;

receiving, from an inertial measurement unit (IMU), IMU data indicative of motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes; and

processing the IMU data and the image data to compute respective state estimates for a position and orientation of the VINS for the first keyframe and for the second keyframe,

wherein computing the state estimates comprises constraining the state estimates for the second keyframe relative to the state estimates for the first keyframe based on the IMU data and the image data from the one or more non-keyframes, and

wherein constraining the state estimates comprises treating a landmark observed within the image data for the first keyframe and the second keyframe as different from the same landmark observed within the image data for the non-keyframes.

12. The method of claim 11, wherein computing the state estimates for the keyframe comprises:

processing the IMU data and the image data associated with the non-keyframes to compute one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe; and

applying the one or more constraints when computing the state estimates of the VINS for the keyframe.

13. The method of claim 12, wherein processing the IMU data and the image data associated with the non-keyframes to compute one or more constraints comprises computing each of the constraints as (i) an estimate of motion from the first keyframe to the second keyframe, and (ii) a covariance indicating an uncertainty of the estimated motion.

14. The method of claim 11, wherein computing the state estimates for the keyframe comprises computing the respective state estimates for the first keyframe and the second keyframe without computing state estimates for the position and the orientation of the VINS for each of the non-keyframes.

15. The method of claim 11, wherein computing the state estimates for the keyframe comprises:

computing the state estimates to include a respective estimated position of each landmark observed for the first keyframe and the second keyframe, and

computing the state estimates for the second keyframe by constraining the position and orientation of the VINS within the second keyframe based on the IMU data and the image data from the one or more non-keyframes while disregarding dependencies for each of the landmarks with respect to landmarks observed within the image data for the one or more non-keyframes.

16. The method of claim 11, wherein computing the state estimates for the keyframe comprises:

maintaining a state vector that specifies, for computation, state estimates for the position and the orientation of the VINS for the first keyframe and one or more landmarks observed from the first keyframe or the second keyframe without including variables for non-keyframe poses or landmarks observed only from non-keyframes; and

iteratively processing the state vector to compute the state estimates.

17. The method of claim 16, wherein iteratively processing the state vector comprises, for each iteration, constraining computation of the state estimates for the position and orientation of the VINS and the landmarks observed from the first keyframe or the second keyframe based on the IMU data and the image data for the one or more non-keyframes of the VINS.

18. The method of claim 11, further comprising selecting key frames along the trajectory for which respective state estimates are to be computed based on a set of criteria comprising one or more of a distance traveled between two consecutive key poses and poses at which points of interest were detected within the image data.

19. The method of claim 11, further comprising constructing a map of the environment to include the state estimates of the VINS.

20. A mobile device comprising:

an image source to produce image data for a first keyframe, one or more non-keyframes and a second keyframe along a trajectory of the vision-aided inertial navigation system (VINS), the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;

an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes; and

a processor configured to compute respective state estimates for a respective position and orientation of the VINS for the first keyframe and for the second keyframe,

wherein, when computing the state estimates, the processor constrains the state estimates for the second keyframe relative to the state estimates for the first keyframe based on the IMU data and the image data from the one or more non-keyframes.

21. The mobile device of claim 20, wherein, when constraining the state estimates, the estimator treats a landmark observed within the image data for the first keyframe and the second keyframe as different from the same landmark observed within the image data for the non-keyframes.

22. A non-transitory computer-readable storage device comprising program code to cause a processor to:

receive image data produced by an image source of the vision-aided inertial navigation system for a first keyframe, one or more non-keyframes and a second keyframe along a trajectory of the vision-aided inertial navigation system (VINS), the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;

receive, from an inertial measurement unit (IMU), IMU data indicative of motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes; and

process the IMU data and the image data to compute state estimates for a position and orientation of the VINS within the second keyframe by constraining the state estimates relative to state estimates for the position and orientation of the VINS within the first keyframe based on the IMU data and the image data of the one or more non-keyframes of the VINS without computing state estimates for the position and orientation of the VINS for the one or more non-keyframes.

23. The non-transitory computer-readable storage device of claim 22, wherein, when constraining the state estimates, the processor treats a landmark observed within the image data for the first keyframe and the second keyframe as different from the same landmark observed within the image data for the non-keyframes by disregarding dependencies for landmarks observed within the first keyframe or the second keyframe with respect to landmarks observed within the image data for the one or more non-keyframes.

24. A vision-aided inertial navigation system comprising:

an image source to produce image data for a first keyframe, one or more non-keyframes and a second keyframe along a trajectory of the vision-aided inertial navigation system (VINS), the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;

an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes; and

a processing unit comprising an estimator that processes the IMU data and the image data to compute respective state estimates for a position and orientation of the VINS for the first keyframe and for the second keyframe,

wherein, when computing the state estimates, the estimator constrains the state estimates for the second keyframe relative to the state estimates for the first keyframe based on the IMU data and the image data from the one or more non-keyframes.

25. The vision-aided inertial navigation system of claim 24,

wherein, when constraining the state estimates, the estimator computes a plurality of constraints based on the image data and the IMU data for the non-keyframes and applies the constraints to the state estimates of the VINS for the second keyframe.

26. The vision-aided inertial navigation system of claim 25, wherein, when computing each of the constraints, the estimator treats a landmark observed within the image data for the first keyframe or the second keyframe as different from the same landmark observed within the image data for the non-keyframes.

27. The vision-aided inertial navigation system of claim 25, wherein, when computing each of the constraints, the estimator treats at least one keyframe as different keyframes for each of the constraints.