Visual-inertial positional awareness for autonomous and non-autonomous tracking

• US 10,043,076 B1
• Filed: 08/29/2016
• Issued: 08/07/2018
• Est. Priority Date: 08/29/2016
• Status: Active Grant

• Current Assignee: Trifo Incorporated
• Original Assignee: PerceptIn, Inc.
• Inventors: Zhang, Zhe, Tsai, Grace, Liu, Shaoshan
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US15/250,393
• Time in Patent Office: 708 Days
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06T 17/05 Geographic models
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 10/751 Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/58 Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads

Abstract

The described positional awareness techniques employing visual-inertial sensory data gathering and analysis hardware with reference to specific example implementations implement improvements in the use of sensors, techniques and hardware design that can enable specific embodiments to provide positional awareness to machines with improved speed and accuracy.

Claims

1. A system including:

a mobile platform having disposed thereon;

at least one camera;

a multi-axis inertial measuring unit (IMU); and

an interface to a host including one or more processors coupled to memory, the memory loaded with computer instructions to updating a position of the mobile platform that includes the camera with distance calculation and multi-axis IMU, the instructions, when executed on the processors, implement actions comprising;

receiving a location of the mobile platform and perspective, including view direction, of the camera, referred to collectively as an initial pose;

while waiting for a new frame, between successive camera frames, updating the initial pose using inertial data from the multi-axis IMU, to generate a propagated pose;

correcting drift between the propagated pose, based on the inertial data, and an actual perspective of a new pose, using the new frame captured by the camera, including;

using the propagated pose, estimating an overlap between the successive camera frames to reduce computation requirements, correlating the new frame with a previous frame by 2D comparison of the successive camera frames, beginning with the estimated overlap;

retrieving at least some feature points within a field of view of the propagated pose from a 3D map using the propagated pose;

extracting new features from the new frame;

matching the extracted new features to the retrieved feature points based on (1) reuse of matched features from the previous frame and (2) matching of features in the new frame with reprojected feature positions from the 3D map onto a 2D view from a perspective of the propagated pose, producing a list of matching features; and

calculating a visually corrected pose using positions of the matching features in the list of matching feature to determine a perspective from which the new frame was viewed by the camera; and

responsive to requests for location of the mobile platform and/or the perspective of the camera, providing data based on one or both of the propagated pose, based on the inertial data, and the visually corrected pose.

2. The system of claim 1, further implementing actions comprising retrieving at least some feature points from a point cloud hybrid map.

3. The system of claim 2, further implementing actions comprising using the 2D view from the perspective of the propagated pose to plan a path of travel.

4. The system of claim 3, further implementing actions comprising using the point cloud while travelling the planned path to avoid colliding with obstacles.

5. The system of claim 1, wherein the matching of features includes smartly searching the 3D map using previous time period velocity/acceleration to predict pose, thereby narrowing search region, by:

performing a linear search when a count of unmatched features is below a threshold; and

otherwise performing a log(n) search using a kd-tree or an octree.

6. The system of claim 1, wherein the successive frames are received at a rate of 15-60 frames per second.

7. The system of claim 1, wherein the inertial data from the multi-axis IMU is received at a rate of 100-2000 Hz.

8. A method of updating a position of a mobile unit that includes a camera with distance calculation and multi-axis inertial measuring unit (IMU), the method including:

receiving a location of the mobile unit and perspective, including view direction, of the camera, referred to collectively as an initial pose;

while waiting for a new frame, between successive camera frames, updating the initial pose using inertial data from the multi-axis IMU, to generate a propagated pose;

correcting drift between the propagated pose, based on the inertial data, and an actual perspective of a new pose, using the new frame captured by the camera, including;

using the propagated pose, estimating an overlap between the successive camera frames to reduce computation requirements, correlating the new frame with a previous frame by 2D comparison of the successive camera frames, beginning with the estimated overlap;

retrieving at least some feature points within a field of view of the propagated pose from a 3D map using the propagated pose;

extracting new features from the new frame;

matching the extracted new features to the retrieved feature points based on (1) reuse of matched features from the previous frame and (2) matching of features in the new frame with reprojected feature positions from the 3D map onto a 2D view from a perspective of the propagated pose, producing a list of matching features; and

calculating a visually corrected pose using positions of the matching features in the list of matching feature to determine a perspective from which the new frame was viewed by the camera; and

responsive to requests for location of the mobile unit and/or the perspective of the camera, providing data based on one or both of the propagated pose, based on the inertial data, and the visually corrected pose.

9. The method of claim 8, further including retrieving at least some feature points from a point cloud hybrid map.

10. The method of claim 9, further including using the 2D view from the perspective of the propagated pose to plan a path of travel.

11. The method of claim 10, further including using the point cloud while travelling the planned path to avoid colliding with obstacles.

12. The method of claim 8, wherein the matching of features further includes smartly searching the 3D map using previous time period velocity/acceleration to predict pose, thereby narrowing search region, by:

performing a linear search when a count of unmatched features is below a threshold; and

otherwise performing a log(n) search using a kd-tree or an octree.

13. The method of claim 8, wherein the successive frames are received at a rate of 15-60 frames per second.

14. The method of claim 8, wherein the inertial data from the multi-axis IMU is received at a rate of 100-2000 Hz.

15. A non-transitory computer readable storage medium impressed with computer program instructions to update a position of a mobile unit that includes a camera with distance calculation and multi-axis inertial measuring unit (IMU), the instructions, when executed on a processor, implement a method comprising:

receiving a location of a mobile unit and perspective, including view direction, of the camera, referred to collectively as an initial pose;

while waiting for a new frame, between successive camera frames, updating the initial pose using inertial data from the multi-axis IMU, to generate a propagated pose;

correcting drift between the propagated pose, based on the inertial data, and an actual perspective of a new pose, using the new frame captured by the camera, including;

using the propagated pose, estimating an overlap between the successive camera frames to reduce computation requirements, correlating the new frame with a previous frame by 2D comparison of the successive camera frames, beginning with the estimated overlap;

retrieving at least some feature points within a field of view of the propagated pose from a 3D map using the propagated pose;

extracting new features from the new frame;

matching the extracted new features to the retrieved feature points based on (1) reuse of matched features from the previous frame and (2) matching of features in the new frame with reprojected feature positions from the 3D map onto a 2D view from a perspective of the propagated pose, producing a list of matching features; and

calculating a visually corrected pose using positions of the matching features in the list of matching feature to determine a perspective from which the new frame was viewed by the camera; and

responsive to requests for location of the mobile platform and/or the perspective of the camera, providing data based on one or both of the propagated pose, based on the inertial data, and the visually corrected pose.

16. The non-transitory computer readable storage medium of claim 15, implementing the method further comprising retrieving at least some feature points from a point cloud hybrid map.

17. The non-transitory computer readable storage medium of claim 16, implementing the method further comprising using the 2D view from the perspective of the propagated pose to plan a path of travel.

18. The non-transitory computer readable storage medium of claim 17, implementing the method further comprising using the point cloud while travelling the planned path to avoid colliding with obstacles.

19. The non-transitory computer readable storage medium of claim 15, wherein the matching of features further includes smartly searching the 3D map using previous time period velocity/acceleration to predict pose, thereby narrowing search region, by:

performing a linear search when a count of unmatched features is below a threshold; and

otherwise performing a log(n) search using a kd-tree or an octree.

20. The non-transitory computer readable storage medium of claim 15, wherein the successive frames are received at a rate of 15-60 frames per second.

21. The non-transitory computer readable storage medium of claim 15, wherein the inertial data from the multi-axis IMU is received at a rate of 100-2000 Hz.

22. A virtual reality headset including:

at least one camera;

a multi-axis inertial measuring unit (IMU); and

an interface to a host including one or more processors coupled to memory, the memory loaded with computer instructions to updating a position of the virtual reality headset that includes the camera with distance calculation and multi-axis IMU, the instructions, when executed on the processors, implement actions comprising;

receiving a location of the virtual reality headset and perspective, including view direction, of the camera, referred to collectively as an initial pose;

while waiting for a new frame, between successive camera frames, updating the initial pose using inertial data from the multi-axis IMU, to generate a propagated pose;

correcting drift between the propagated pose, based on the inertial data, and an actual perspective of a new pose, using the new frame captured by the camera, including;

using the propagated pose, estimating an overlap between the successive camera frames to reduce computation requirements, correlating the new frame with a previous frame by 2D comparison of the successive camera frames, beginning with the estimated overlap;

retrieving at least some feature points within a field of view of the propagated pose from a 3D map using the propagated pose;

extracting new features from the new frame;

matching the extracted new features to the retrieved feature points based on (1) reuse of matched features from the previous frame and (2) matching of features in the new frame with reprojected feature positions from the 3D map onto a 2D view from a perspective of the propagated pose, producing a list of matching features; and

calculating a visually corrected pose using positions of the matching features in the list of matching feature to determine a perspective from which the new frame was viewed by the camera; and

responsive to requests for location of the virtual reality headset and/or the perspective of the camera, providing data based on one or both of the propagated pose, based on the inertial data, and the visually corrected pose.