Visual-based inertial navigation

• US 10,152,795 B2
• Filed: 08/12/2016
• Issued: 12/11/2018
• Est. Priority Date: 08/12/2013
• Status: Active Grant

• Current Assignee: Apple Inc., Regents of The University of Minnesota (University of Minnesota)
• Original Assignee: Apple Inc., Regents of The University of Minnesota (University of Minnesota)
• Inventors: Flint, Alex, Naroditsky, Oleg, Broaddus, Christopher P., Grygorenko, Andriy, Roumeliotis, Stergios, Bergig, Oriel
• Primary Examiner(s): Shah, Utpal
• Application Number:
  US15/236,008
• Publication Number:
  US 20160350926A1
• Time in Patent Office: 851 Days
• CPC Class Codes: G01C 21/1652 with ranging devices, e.g. LIDAR or RADAR
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01C 21/188 for accumulated errors, e.g. by coupling inertial systems with absolute positioning systems
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes

Abstract

A method includes: receiving sensor measurements from a pre-processing module, in which the sensor measurements include image data and inertial data for a device; transferring, using a processor, information derived from the sensor measurements, from a first set of variables associated with a first window of time to a second set of variables associated with a second window of time, in which the first and second windows consecutively overlap in time; and outputting, to a post-processing module, a state of the device based on the transferred information.

Claims

1. An electronic computing system for performing navigation, the electronic computing system comprising an electronic processor and memory operable to implement a sliding-window inverse filter module, wherein the sliding-window inverse filter module is configured to:

transfer information, from a first set of variables associated with a first window of time to a second set of variables associated with a second window of time, wherein the first window of time and the second window of time consecutively overlap in time; and

generate a path history of a device based on the first set of variables and the second set of variables,

wherein the first window of time is associated with a corresponding first information matrix including a plurality of first variables representing a position and/or orientation of at least one object proximate to the device and a plurality of second variables representing a state of the device at two or more points in time, and

wherein transferring the information comprises;

marginalizing at least a subset of the first variables associated with the first information matrix of the first window of time to obtain a second information matrix, which still maintains associations between the plurality of second variables;

marginalizing a subset of the second variables associated with the second information matrix to obtain a third information matrix;

scaling the third information matrix to obtain a fourth information matrix; and

generating a fifth information matrix associated with the second window of time based on the fourth information matrix.

2. The electronic computing system of claim 1, wherein scaling the third information matrix to obtain the fourth information matrix includes dividing the third information matrix by a number of measurements in the first window of time.

3. The electronic computing system of claim 1, wherein the third information matrix is a summarized representation of the state of the device and of the position and/or orientation of the at least one object proximate to the device at points in time when the first window of time and the second window of time overlap.

4. The electronic computing system of claim 1, wherein the sliding-window inverse filter module is configured to receive sensor measurements from a pre-processing module, wherein the sensor measurements comprise image data and inertial data for the device, and wherein the fifth information matrix is generated based further on a portion of the sensor measurements corresponding to the second window of time.

5. The electronic computing system of claim 4, further comprising:

the pre-processing module;

an image detection unit coupled to the pre-processing module, wherein the pre-processing module is configured to receive a plurality of images captured by the image detection unit and derive the image data from the plurality of images; and

a motion sensing unit coupled to the pre-processing module, wherein the pre-processing module is configured to receive the inertial data from the motion sensing unit.

6. The electronic computing system of claim 1, wherein the sliding-window inverse filter module is further configured to output, to a post-processing module, a state of the device based on the fifth information matrix.

7. The electronic computing system of claim 1, wherein the plurality of first variables represents 3D positions and/or orientations of image features across a plurality of images, and

wherein the plurality of second variables comprises linear acceleration data and angular velocity data.

8. The electronic computing system of claim 7, wherein the state of the device further comprises biases for a motion sensing unit.

9. The electronic computing system of claim 1, wherein the first information matrix comprises:

first information about at least one variable representing an estimate for a position or orientation of an object proximate to the device;

second information about at least one variable representing an estimate of the state of the device; and

shared information between the first information and the second information.

10. A computer-implemented method, comprising:

transferring, using a processor, information from a first set of variables associated with a first window of time to a second set of variables associated with a second window of time, wherein the first window of time and the second window of time consecutively overlap in time; and

generating a path history of a device based on the first set of variables and the second set of variables,

wherein the first window of time is associated with a corresponding first information matrix including a plurality of first variables representing a position and/or orientation of at least one object proximate to the device and a plurality of second variables representing a state of the device at two or more points in time, and

wherein transferring the information comprises;

marginalizing at least a subset of the first variables associated with the first information matrix of the first window of time to obtain a second information matrix, which still maintains associations between the plurality of second variables;

marginalizing a subset of the second variables associated with the second information matrix to obtain a third information matrix;

scaling the third information matrix to obtain a fourth information matrix; and

generating a fifth information matrix associated with the second window of time based on the fourth information matrix.

11. The computer-implemented method of claim 10, wherein the first information matrix comprises:

first information about at least one variable representing an estimate for a position or orientation of an object proximate to the device;

second information about at least one variable representing an estimate of the state of the device; and

shared information between the first information and the second information.

12. The computer-implemented method of claim 11, wherein the shared information represents shared confidence between the estimates of a position or orientation of an object proximate to the device and the estimate of the state of the device.

13. The computer-implemented method of claim 11,

wherein marginalizing at least the subset of the first variables associated with the first information matrix of the first window of time comprises marginalizing the shared information.

14. A non-transitory computer storage medium encoded with computer program instructions that when executed by one or more electronic computing devices cause the one or more electronic computing devices to perform operations comprising:

transferring, using a processor, information from a first set of variables associated with a first window of time to a second set of variables associated with a second window of time, wherein the first window of time and the second window of time consecutively overlap in time; and

generating a path history of a device based on the first set of variables and the second set of variables,

wherein the first window of time is associated with a corresponding first information matrix including a plurality of first variables representing a position and/or orientation of at least one object proximate to the device and a plurality of second variables representing a state of the device at two or more points in time, and

wherein transferring the information comprises;

marginalizing at least a subset of the first variables associated with the first information matrix of the first window of time to obtain a second information matrix, which still maintains associations between the plurality of second variables;

marginalizing a subset of the second variables associated with the second information matrix to obtain a third information matrix;

scaling the third information matrix to obtain a fourth information matrix; and

generating a fifth information matrix associated with the second window of time based on the fourth information matrix.

15. The medium of claim 14, wherein scaling the third information matrix to obtain the fourth information matrix includes dividing the third information matrix by a number of measurements in the first window of time.

16. The medium of claim 14, wherein the third information matrix is a summarized representation of the information about the state of the device and of the position and/or orientation of the at least one object proximate to the device at points in time when the first window of time and the second window of time overlap.

17. The medium of claim 14, wherein the operations further include receiving sensor measurements from a pre-processing module, wherein the sensor measurements comprise image data and inertial data for the device, and wherein fifth information matrix is generated based further on a portion of the sensor measurements corresponding to the second window of time.

18. The medium of claim 14, wherein a subset of the first set of variables associated with the first window of time overlaps in time with a subset of the second set of variables associated with the second window of time.

19. The medium of claim 18, wherein the overlap between the first set of variables and the second set of variables is at the earliest time that is present in both the first window of time and the second window of time.

20. The medium of claim 14, wherein the plurality of first variables represents 3D positions and/or orientations of image features across a plurality of images, and

wherein the plurality of second variables comprises linear acceleration data and angular velocity data.