Constrained key frame localization and mapping for vision-aided inertial navigation

• US 9,996,941 B2
• Filed: 03/27/2017
• Issued: 06/12/2018
• Est. Priority Date: 05/08/2013
• Status: Active Grant

• Current Assignee: Regents of The University of Minnesota (University of Minnesota)
• Original Assignee: Regents of The University of Minnesota (University of Minnesota)
• Inventors: Roumeliotis, Stergios I., Nerurkar, Esha D.
• Primary Examiner(s): Chio, Tat C
• Application Number:
  US15/470,595
• Publication Number:
  US 20170294023A1
• Time in Patent Office: 442 Days
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/248 involving reference images or patches
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

Estimation techniques for vision-aided inertial navigation are described. In one example, a vision-aided inertial navigation system (VINS) comprises an image source to produce image data for a keyframe and one or more non-keyframes along a trajectory, the one or more non-keyframes preceding the keyframe along the trajectory. The VINS comprises an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the VINS along the trajectory for the keyframe and the one or more non-keyframes, and a processing unit comprising an estimator that processes the IMU data and the image data to compute state estimates of the VINS. The estimator computes the state estimates of the VINS for the keyframe by constraining the state estimates based on the IMU data and the image data for the one or more non-keyframes of the VINS without computing state estimates of the VINS for the one or more non-keyframes.

Claims

1. A device comprising:

an image source configured to produce image data for a first keyframe, one or more nonkeyframes, and a second keyframe along a trajectory of the device, the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;

an inertial measurement unit (IMU) configured to produce IMU data indicative of a motion of the device along the trajectory; and

a processor configured to;

process the IMU data and the image data to compute respective state estimates for a position and orientation of the device for the first keyframe and for the second keyframe;

process the image data associated with the non-keyframes to compute one or more constraints to the position and the orientation of the device for the second keyframe relative to the position and the orientation of the device for the first keyframe, wherein when computing the one or more constraints the processor is configured to treat a landmark observed within the image data for the first keyframe or the second keyframe as different from the same landmark observed within the image data for the non-keyframes; and

apply the one or more constraints when computing the state estimates of the device for the second keyframe.

2. The device of claim 1, wherein, to process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the device for the second keyframe relative to the position and the orientation of the device for the first keyframe, the processor is configured to process both the IMU data and the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the device for the second keyframe relative to the position and the orientation of the device for the first keyframe.

3. The device of claim 1, wherein, to process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the device for the second keyframe relative to the position and the orientation of the device for the first keyframe, the processor is configured to:

process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the device for the second keyframe relative to the position and the orientation of the device for the first keyframe; and

discard IMU data associated with the non-keyframes.

4. The device of claim 1, wherein the processor is further configured to compute each of the constraints as an estimate of motion from the first keyframe to the second keyframe, and a covariance indicating an uncertainty of the estimated motion.

5. The device of claim 1, wherein the processor is further configured to compute the respective state estimates for the first keyframe and the second keyframe without computing state estimates for the position and the orientation of the device for each of the non-keyframes.

6. The device of claim 1,

wherein the processor is further configured to compute the state estimates to include a respective estimated position of each landmark observed for the first keyframe and the second keyframe, and

wherein, to compute the state estimates for the second keyframe, the processor is further configured to constrain the position and orientation of the device within the second keyframe based on the image data from the one or more non-keyframes while disregarding dependencies for each of the landmarks with respect to landmarks observed within the image data for the one or more non-keyframes.

7. The device of claim 1, wherein the processor is configured to compute estimates for positions of landmarks observed from the first keyframe and the second keyframe without computing estimates for positions of landmarks observed only from the non-keyframes for which a respective pose of the device is not computed.

8. The device of claim 1,

wherein the processor is configured to maintain a state vector that specifies, for computation, state estimates of the position and the orientation of the device for the first keyframe and one or more landmarks observed from the first keyframe or the second keyframe without including variables for non-keyframe poses or landmarks observed only from non-keyframes,

wherein the processor is configured to iteratively process the state vector to compute the state estimates, and

wherein, for each iteration, the processor is configured to constrain updates for the state estimates for the position and orientation of the device and the landmarks observed from the keyframes for the device based on the image data for the one or more non-keyframes of the device.

9. The device of claim 8, wherein the processor is further configured to:

select key frames along the trajectory for which respective state estimates are to be computed within the state vector, and

select the key frames based on a set of criteria comprising one or more of a distance traveled between two consecutive key poses and poses at which points of interest were detected within the image data.

10. The device of claim 1, wherein the processor is configured to build a map of the environment to include the state estimates of the device.

11. The device of claim 1, wherein the device is integrated within a mobile phone or a robot.

12. The device of claim 1, wherein, to process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the device, the processor is further configured to process the image data associated with the non-keyframes to compute the one or more constraints without processing the IMU data to compute the one or more constraints.

13. The device of claim 1, wherein the processor is further configured to compute, based on the computed state estimates of the device, a speed of the device.

14. The device of claim 1, wherein the processor is further configured to compute, based on the computed state estimates of the device, an odometry of the device.

15. A system comprising:

an image source to produce image data for a first keyframe, one or more non-keyframes, and a second keyframe along a trajectory of a vision-aided inertial navigation system (VINS), the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;

an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the VINS along the trajectory; and

a processing unit comprising an estimator that processes the IMU data and the image data to compute respective state estimates for a position and orientation of the VINS for the first keyframe and for the second keyframe,

wherein the estimator processes the image data associated with the non-keyframes to compute one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe,

wherein, when computing the one or more constraints, the processor is configured to treat a landmark observed within the image data for the first keyframe or the second keyframe as different from the same landmark observed within the image data for the non-keyframes, and

wherein the estimator applies the one or more constraints when computing the state estimates of the VINS for the second keyframe.

16. The system of claim 15, wherein, to process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe, the estimator is configured to process both the IMU data and the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe.

17. The system of claim 15, wherein, to process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe, the estimator is configured to:

process the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe; and

discard IMU data associated with the non-keyframes.

18. A method comprising:

producing, by an image source, image data for a first keyframe, one or more nonkeyframes, and a second keyframe along a trajectory of a vision-aided inertial navigation system (VINS), the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;

producing, by an inertial measurement unit (IMU), IMU data indicative of a motion of the VINS along the trajectory;

processing, by one or more processors, the IMU data and the image data to compute respective state estimates for a position and orientation of the VINS for the first keyframe and for the second keyframe;

processing, by the one or more processors, the image data associated with the non-keyframes to compute one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe, wherein, when computing the one or more constraints the processor is configured to treat a landmark observed within the image data for the first keyframe or the second keyframe as different from the same landmark observed within the image data for the non-keyframes; and

applying, by one or more processors, the one or more constraints when computing the state estimates of the VINS for the second keyframe.

19. The method of claim 18, wherein processing the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe comprises processing both the IMU data and the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe.

20. The method of claim 19, wherein processing the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe comprises:

processing the image data associated with the non-keyframes to compute the one or more constraints to the position and the orientation of the VINS for the second keyframe relative to the position and the orientation of the VINS for the first keyframe; and

discarding IMU data associated with the non-keyframes.

21. A device comprising:

an image source configured to produce image data for a first keyframe, one or more nonkeyframes, and a second keyframe along a trajectory of the device, the one or more non-keyframes located between the first keyframe and second keyframe along the trajectory;

an inertial measurement unit (IMU) configured to produce IMU data indicative of a motion of the device along the trajectory; and

a processor configured to;

process the image data associated with the non-keyframes to compute one or more constraints to a position of the device for the second keyframe relative to the position of the device for the first keyframe, wherein, when computing the one or more constraints, the processor is configured to treat a landmark observed within the image data for the first keyframe or the second keyframe as different from the same landmark observed within the image data for the non-keyframes; and

determine, based on the IMU data, the image data, and the one or more constraints, the position of the device for one or more of the first keyframe and the second keyframe.

22. The device of claim 21, wherein the processor is further configured to generate a map based on the determined position of the device.

23. The device of claim 21, wherein the processor is further configured to update a display based on the determined position of the device.

24. The device of claim 21, wherein the device comprises one of a mobile device, a mobile phone, a computer, a tablet, a robot, or a vehicle.

25. A device comprising:

an image source configured to produce image data for a first keyframe, one or more nonkeyframes, and a second keyframe along a trajectory of the device, wherein the one or more nonkeyframes are located between the first keyframe and second keyframe along the trajectory; and

a processor configured to;

compute, based on the image data associated with the nonkeyframes, one or more constraints to an estimate of a position of the device for the second keyframe relative to a position of the device for the first keyframe without computing estimates for positions of one or more landmarks that were observed only from the nonkeyframes; and

compute, based on the image data associated with the first keyframe and the second keyframe and the one or more constraints, the estimate of the position of the device for the second keyframe relative to the position of the device for the first keyframe.

26. The device of claim 25, wherein, to compute the one or more constraints to the estimate of the position of the device for the second keyframe relative to the position of the device for the first keyframe without computing estimates for the positions of the one or more landmarks that were observed only from the nonkeyframes, the processor is further configured to compute the one or more constraints to the estimate of the position of the device for the second keyframe relative to the position of the device for the first keyframe while disregarding dependencies for one or more landmarks that were observed from the first keyframe or the second keyframe with respect to one or more landmarks observed within the image data for the one or more non-keyframes.

27. The device of claim 25, wherein the image source comprises at least one of:

one or more cameras that capture two-dimensional images;

one or more cameras that capture three-dimensional images;

a red-green-blue (RGB) camera;

a black-and-white camera;

a sonar imaging device;

a laser scanner;

or

a depth sensor that produces image data indicative of ranges for features within the environment.

28. The device of claim 25, wherein the processor is further configured to generate a map based on the estimate of the position of the device.

29. The device of claim 25, wherein the processor is further configured to update a display based on the estimate of the position of the device.

30. The device of claim 25, wherein the device comprises one of a mobile device, a mobile phone, a computer, a tablet, a robot, or a vehicle.