Extrinsic parameter calibration of a vision-aided inertial navigation system

• US 10,254,118 B2
• Filed: 02/21/2014
• Issued: 04/09/2019
• Est. Priority Date: 02/21/2013
• Status: Active Grant

• Current Assignee: Regents of The University of Minnesota (University of Minnesota)
• Original Assignee: Regents of The University of Minnesota (University of Minnesota)
• Inventors: Roumeliotis, Stergios I., Kottas, Dimitrios G., Wu, Kejian J.
• Primary Examiner(s): Nasri, Maryam A
• Application Number:
  US14/768,733
• Publication Number:
  US 20160005164A1
• Time in Patent Office: 1,873 Days
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: B25J 9/1664 characterised by motion, path, trajectory planning
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G01C 21/165 combined with non-inertial navigation instruments
• CPC Class Codes: G01C 21/1652 with ranging devices, e.g. LIDAR or RADAR
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 21/185 for gravity
• CPC Class Codes: G01C 21/188 for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/277 involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/09 Closed loop, sensor feedback controls arm movement

Abstract

This disclosure describes various techniques for use within a vision-aided inertial navigation system (VINS). A VINS comprises an image source to produce image data comprising a plurality of images, and an inertial measurement unit (IMU) to produce IMU data indicative of a motion of the vision-aided inertial navigation system while producing the image data, wherein the image data captures features of an external calibration target that is not aligned with gravity. The VINS further includes a processing unit comprising an estimator that processes the IMU data and the image data to compute calibration parameters for the VINS concurrently with computation of a roll and pitch of the calibration target, wherein the calibration parameters define relative positions and orientations of the IMU and the image source of the vision-aided inertial navigation system.

Claims

1. A method for calibration of a vision-aided inertial navigation system comprising:

receiving image data produced by an image source of the vision-aided inertial navigation system (VINS), wherein the image data captures features of a calibration target;

receiving, from an inertial measurement unit (IMU) of the VINS, IMU data indicative of motion of the VINS; and

computing, based on the image data and the IMU data and using an estimator of the VINS, calibration parameters for the VINS concurrently with computation of a roll and pitch of the calibration target at least by applying a constrained estimation algorithm to compute state estimates based on the image data and the IMU data while preventing projection of information from the image data and the IMU data along at least one unobservable degree of freedom of the VINS,

wherein the calibration parameters define relative positions and orientations of the IMU and the image source of the VINS.

2. The method of claim 1, wherein computing, based on the image data and the IMU data, calibration parameters for the VINS concurrently with computation of the roll and pitch of the calibration target comprises:

constructing an enhanced state vector that specifies a plurality of state estimates to be computed, wherein the state vector specifies the plurality of state estimates to include the roll and the pitch of the calibration target as well as the calibration parameters for the VINS; and

iteratively processing the enhanced state vector with a Kalman Filter to compute the roll and the pitch of the calibration target concurrently with computation of calibration parameters for the VINS.

3. The method of claim 2, wherein the enhanced state vector specifies the calibration parameters to be computed as relative positions, orientations and velocities of the IMU and the image source and one or more signal biases for the IMU.

4. The method of claim 2, wherein the enhanced state vector specifies the calibration parameters to be computed as an orientation of a global frame of reference relative to an IMU frame of reference, an orientation of the image source relative to the IMU frame of reference, a position and a velocity of the IMU within the global frame of reference, a position of the image source within the IMU frame of reference, and a set of bias vectors for biases associated with signals of the IMU.

5. The method of claim 2, wherein iteratively applying the Kalman Filter comprises applying a Multi-state Constraint Kalman Filter (MSC-KF) that applies the constrained estimation algorithm to prevent the projection of information from the image data and IMU data along the at least one unobservable degrees of freedom of the VINS.

6. The method of claim 1, wherein computing the calibration parameters for the VINS concurrently with computation of the roll and pitch of the calibration target comprises applying the constrained estimation algorithm to compute the state estimates by projecting information from the image data and IMU data for at least translations in horizontal and vertical directions while preventing projection of information from the image data and IMU data along the at least one unobservable degree of freedom, the at least one unobservable degree of freedom comprising at least a gravity vector.

7. The method of claim 1, wherein computing the calibration parameters further comprises processing the image data to reject one or more outliers within a set of features identified within the image data by:

receiving a first portion of the image data as a first set of images produced by the image source;

receiving a second portion of the image data as a second set of images produced by the image source;

determining an orientation of the first set of images relative to the second set of images based on the IMU data;

processing the first set of images and the second set of images to identify a set of common features based on the IMU data.

8. The method of claim 7, wherein the first set of images are produced by a first image source of the VINS and the second set of images are produced by a second image source of the VINS.

9. The method of claim 7, wherein the first set of images are produced by a first image source of the VINS at a first orientation and the second set of images are produced by the first image source of the VINS at a second rotation without translation of the first image source.

10. The method of claim 1, wherein the calibration target is not aligned with gravity.

11. The method of claim 1, wherein the VINS comprises a robot or a vehicle.

12. The method of claim 1, wherein the VINS comprises one of a mobile sensing platform, a mobile phone, a workstation, a computing center, or a set of one or more servers.

13. A vision-aided inertial navigation system (VINS) comprising:

an image source to produce image data comprising a plurality of images;

an inertial measurement unit (IMU) comprising at least one of an accelerometer or a gyroscope, the IMU being configured to produce IMU data indicative of a motion of the VINS while producing the image data, wherein the image data captures features of an external calibration target;

one or more processors configured to process the IMU data and the image data to compute calibration parameters for the VINS concurrently with computation of a roll and pitch of the calibration target at least by applying a constrained estimation algorithm to compute state estimates based on the image data and the IMU data while preventing projection of information from the image data and the IMU data along at least one unobservable degree of freedom of the VINS,

wherein the calibration parameters define relative positions and orientations of the IMU and the image source of the VINS.

14. The vision-aided inertial navigation system (VINS) of claim 13,

wherein the one or more processors are further configured to construct an enhanced state vector that specifies a plurality of state estimates to be computed based on the image data and the IMU data, wherein the state vector specifies the plurality of state estimates to include the roll and the pitch of the calibration target as well as the calibration parameters for the VINS, and

wherein the one or more processors are further configured to iteratively process the enhanced state vector with a Kalman Filter to compute the roll and the pitch of the calibration target concurrently with computation of calibration parameters for the VINS.

15. The vision-aided inertial navigation system (VINS) of claim 13, wherein the one or more processors execute within a device that houses the image source and the IMU.

16. The vision-aided inertial navigation system (VINS) of claim 13, wherein the one or more processors are remote from the image source and the IMU unit.

17. The vision-aided inertial navigation system (VINS) of claim 13, wherein the VINS comprises one of a mobile sensing platform, a mobile phone, a workstation, a computing center, or a set of one or more servers.

18. The vision-aided inertial navigation system (VINS) of claim 13, wherein the image source comprises one or more of a cameras that capture 2D or 3D images, a laser scanner that produces a stream of 1D image data, a depth sensor that produces image data indicative of ranges for features within an environment and a stereo vision system having multiple cameras to produce 3D information.

19. The vision-aided inertial navigation system of claim 13, wherein the vision-aided inertial navigation comprises a robot or a vehicle.

20. A method for calibration of a vision-aided inertial navigation system, further comprising:

receiving image data produced by an image source of the vision-aided inertial navigation system (VINS), wherein the image data captures features of a calibration target;

receiving, from an inertial measurement unit (IMU) of the VINS, IMU data indicative of motion of the VINS;

computing, based on the image data and the IMU data and using an estimator of the VINS, calibration parameters for the VINS concurrently with computation of a roll and pitch of the calibration target, wherein the calibration parameters define relative positions and orientations of the IMU and the image source of the VINS;

after computing the calibration parameters for the VINS, receiving a second set of images produced by the image source of the VINS;

receiving, from the inertial measurement unit (IMU) of the vision-aided inertial navigation system, IMU data indicative of motion of the VINS while capturing the second set of images;

processing the second set of images as a sliding window of sequenced images to detect a hovering condition during which a translation of the VINS is below a threshold amount of motion for each of a set of degrees of freedom; and

computing a position and an orientation of the VINS based on the sliding window of the sequenced set of images and inertial measurement data for the device.

21. The method of claim 20, further comprising:

upon detecting the hovering condition, replacing a newest one of the images in the sliding window of sequenced images with a current image produced by the image source; and

upon detecting that the absence of the hovering condition, dropping an oldest one of the images in the sliding window of sequenced images and including the current image within sliding window of sequenced images the current image.

22. The method of claim 20, further comprising computing a map of an area traversed by the vision-aided inertial navigation system based at least in part on the sliding window of sequenced images.

23. The method of claim 20, further comprising computing an odometry traversed by the device based at least in part on the sliding window of sequenced images.

24. The method of claim 20, further comprising computing at least one of a pose of the device, a velocity of the device, a displacement of the device and 3D positions of visual landmarks based at least in part on the sliding window of sequenced images.