Mapping optimization in autonomous and non-autonomous platforms

• US 10,453,213 B2
• Filed: 07/12/2017
• Issued: 10/22/2019
• Est. Priority Date: 08/29/2016
• Status: Active Grant

• Current Assignee: Trifo Incorporated
• Original Assignee: Trifo Incorporated
• Inventors: Tsai, Grace, Zhang, Zhe, Liu, Shaoshan
• Primary Examiner(s): Owens, Tsion B
• Application Number:
  US15/648,372
• Publication Number:
  US 20180286072A1
• Time in Patent Office: 832 Days
• CPC Class Codes: G01C 21/20 Instruments for performing navigational calculations G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G01P 15/18 in two or more dimensions
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 10/44 Local feature extraction by analysis of parts of the pattern, e.g. by detecting edges, contours, loops, corners, strokes or intersections; Connectivity analysis, e.g. of connected components

Abstract

The described positional awareness techniques employing visual-inertial sensory data gathering and analysis hardware with reference to specific example implementations implement improvements in the use of sensors, techniques and hardware design that can enable specific embodiments to provide positional awareness to machines with improved speed and accuracy.

Claims

1. A system, including:

a mobile platform having disposed thereon;

at least one camera;

a multi-axis inertial measuring unit (IMU); and

an interface to a host including one or more processors coupled to memory storing computer instructions to correct at least some estimated poses and locations of at least some 3D points that define a map, the map used to provide guidance to the mobile platform that includes the camera with distance calculation and the multi-axis inertial measuring unit (IMU), which computer instructions, when executed on the processors, implement actions comprising;

receiving sets of image data including 2d feature points and pose information, the pose information including a location of the mobile platform and view of the camera that captured the sets of image data, wherein sets comprised of image data and pose information are referred to collectively as keyrigs;

reviewing keyrigs to select keyrig content to include in a map defined from 3D points, based upon comparisons of keyrig content with content of other selected keyrigs, by;

apportioning freedom among keyrigs by selecting, from among keyrigs, a set of free keyrigs available to be refined including a first subset of keyrigs chosen at random and a second subset of keyrigs that (i) are within a radius of at least one of the first subset of keyrigs and (ii) share a field of view with the at least one of the first subset of keyrigs;

wherein keyrigs not selected comprise a set of fixed keyrigs that are not available to be refined; and

determining for select 3D points in the map and within a field of view of at least one keyrig, a group identity including one selected from a set of;

(i) a free point that is available to be refined;

wherein a 3D point that is within a viewpoint of more than two keyrigs, including at least one keyrig that is free is a free point;

(ii) a semi-free point that is not available to be refined but can be changed in location during course of refining a free point;

wherein a 3D point that is within a viewpoint of at most one keyrig that is free is a semi-free point; and

a (iii) fixed point that is not available to be changed;

wherein a 3D point that is within a viewpoint of a keyrig that is fixed is a fixed point; and

refining the map to correct at least some estimated poses and locations of at least some of the free points, and using corrected estimated poses and locations to provide guidance to the mobile platform.

2. The system of claim 1, further implementing actions comprising refining the map defined from 3D points corresponding to 2d features by:

selecting sets of at least some of image data and pose information of the map defined from 3D points;

reprojecting 3D points from the map into reprojected points on a 2D image plane using the pose information;

calculating image coordinates for the reprojected points on the 2D image plane and comparing the image coordinates calculated for the reprojected points on the 2D image plane to actual coordinates of corresponding observed points obtained directly from image data selected to compute a plurality of re-projection errors; and

minimizing a weighted sum of the re-projection errors for the sets of at least some of image data and pose information selected.

3. The system of claim 1, wherein the set of fixed keyrigs requires at least one fixed keyrig that maintains a global coordinate system for the map.

4. The system of claim 1, further comprising apportioning freedom among keyrigs into “

fixed” and

“

free”

subsets whenever a new refinement process is applied to adjust at least some portion of the map.

5. The system of claim 1, further comprising triggering refining the map at intervals of N seconds.

6. The system of claim 5, wherein N includes 1 second.

7. The system of claim 1, further comprising triggering refining the map when a current location or orientation and a last triggering pose is larger than a threshold.

8. The system of claim 7, wherein location or orientation and a last triggering pose is larger than a threshold includes device has moved more than X meter or has changed orientation direction more than Y degree.

9. The system of claim 8, wherein X is 0.5 meter.

10. The system of claim 1, further comprising triggering refining the map when a current pose is sufficiently distinct enough in comparison to a plurality of keyrigs in the map.

11. The system of claim 10, wherein sufficiently distinct includes greater than 0.5 meter.

12. The system of claim 10, wherein sufficiently distinct includes greater than 40 degrees.

13. A non-transitory computer readable storage medium storing instructions to correct at least some estimated poses and locations of at least some 3D points that define a map, the map used to provide guidance to a mobile platform that includes a camera with distance calculation and a multi-axis inertial measuring unit (IMU), which instructions, when executed on one or more processors, implement actions comprising:

receiving sets of image data including 2d feature points and pose information, the pose information including a location of the mobile platform and view of the camera that captured the sets of image data, wherein sets comprised of image data and pose information are referred to collectively as keyrigs;

reviewing keyrigs to select keyrig content to include in a map defined from 3D points, based upon comparisons of keyrig content with content of other selected keyrigs, by;

apportioning freedom among keyrigs by selecting, from among keyrigs, a set of free keyrigs available to be refined including a first subset of keyrigs chosen at random and a second subset of keyrigs that (i) are within a radius of at least one of the first subset of keyrigs and (ii) share a field of view with the at least one of the first subset of keyrigs;

wherein keyrigs not selected comprise a set of fixed keyrigs that are not available to be refined; and

determining for select 3D points in the map and within a field of view of at least one keyrig, a group identity including one selected from a set of;

(i) a free point that is available to be refined;

wherein a 3D point that is within a viewpoint of more than two keyrigs, including at least one keyrig that is free is a free point;

(ii) a semi-free point that is not available to be refined but can be changed in location during course of refining a free point;

wherein a 3D point that is within a viewpoint of at most one keyrig that is free is a semi-free point; and

a (iii) fixed point that is not available to be changed;

wherein a 3D point that is within a viewpoint of a keyrig that is fixed is a fixed point; and

refining the map to correct at least some estimated poses and locations of at least some of the free points, and using corrected estimated poses and locations to provide guidance to the mobile platform.

14. The non-transitory computer readable storage medium of claim 13, further implementing actions comprising refining the map defined from 3D points corresponding to 2d features by:

selecting sets of at least some of image data and pose information of the map defined from 3D points;

reprojecting 3D points from the map into reprojected points on a 2D image plane using the pose information;

calculating image coordinates for the reprojected points on the 2D image plane and comparing the image coordinates calculated for the reprojected points on the 2D image plane to actual coordinates of corresponding observed points obtained directly from image data selected to compute a plurality of re-projection errors; and

minimizing a weighted sum of the re-projection errors for the sets of at least some of image data and pose information selected.

15. The non-transitory computer readable storage medium of claim 13, wherein the set of fixed keyrigs requires at least one fixed keyrig that maintains a global coordinate system for the map.

16. The non-transitory computer readable storage medium of claim 13, further implementing actions comprising apportioning freedom among keyrigs into “

fixed” and

“

free”

subsets whenever a new refinement process is applied to adjust at least some portion of the map.

17. The non-transitory computer readable storage medium of claim 13, further implementing actions comprising triggering refining the map at intervals of N seconds.

18. The non-transitory computer readable storage medium of claim 17, wherein N includes 1 second.

19. The non-transitory computer readable storage medium of claim 13, further implementing actions comprising triggering refining the map when a current location or orientation and a last triggering pose is larger than a threshold.

20. The non-transitory computer readable storage medium of claim 19, wherein location or orientation and a last triggering pose is larger than a threshold includes device has moved more than X meter or has changed orientation direction more than Y degree.

21. The non-transitory computer readable storage medium of claim 13, further implementing actions comprising triggering refining the map when a current pose is at least one of greater than 0.5 meter and greater than 40 degrees in comparison to a plurality of keyrigs in the map.

22. A method for correcting at least some estimated poses and locations of at least some 3D points that define a map, the map used to provide guidance to a mobile platform that includes a camera with distance calculation and a multi-axis inertial measuring unit (IMU), the method comprising:

receiving sets of image data including 2d feature points and pose information, the pose information including a location of the mobile platform and view of the camera that captured the sets of image data, wherein sets comprised of image data and pose information are referred to collectively as keyrigs;

reviewing keyrigs to select keyrig content to include in a map defined from 3D points, based upon comparisons of keyrig content with content of other selected keyrigs, by;

apportioning freedom among keyrigs by selecting, from among keyrigs, a set of free keyrigs available to be refined including a first subset of keyrigs chosen at random and a second subset of keyrigs that (i) are within a radius of at least one of the first subset of keyrigs and (ii) share a field of view with the at least one of the first subset of keyrigs;

wherein keyrigs not selected comprise a set of fixed keyrigs that are not available to be refined; and

determining for select 3D points in the map and within a field of view of at least one keyrig, a group identity including one selected from a set of;

(i) a free point that is available to be refined;

wherein a 3D point that is within a viewpoint of more than two keyrigs, including at least one keyrig that is free is a free point;

(ii) a semi-free point that is not available to be refined but can be changed in location during course of refining a free point;

wherein a 3D point that is within a viewpoint of at most one keyrig that is free is a semi-free point; and

a (iii) fixed point that is not available to be changed;

wherein a 3D point that is within a viewpoint of a keyrig that is fixed is a fixed point; and

refining the map to correct at least some estimated poses and locations of at least some of the free points, and using corrected estimated poses and locations to provide guidance to the mobile platform.

23. The method of claim 22, further comprising refining the map defined from 3D points corresponding to 2d features by:

selecting sets of at least some of image data and pose information of the map defined from 3D points;

reprojecting 3D points from the map into reprojected points on a 2D image plane using the pose information;

calculating image coordinates for the reprojected points on the 2D image plane and comparing the image coordinates calculated for the reprojected points on the 2D image plane to actual coordinates of corresponding observed points obtained directly from image data selected to compute a plurality of re-projection errors; and

minimizing a weighted sum of the re-projection errors for the sets of at least some of image data and pose information selected.