Fault tolerance to provide robust tracking for autonomous positional awareness

• US 10,162,362 B2
• Filed: 07/24/2017
• Issued: 12/25/2018
• Est. Priority Date: 08/29/2016
• Status: Active Grant

• Current Assignee: Trifo Incorporated
• Original Assignee: PerceptIn, Inc.
• Inventors: Tsai, Grace, Zhang, Zhe, Liu, Shaoshan
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US15/658,279
• Publication Number:
  US 20180284802A1
• Time in Patent Office: 519 Days
• CPC Class Codes: G01C 21/005 with correlation of navigation data from several sources, e.g. map or contour matching G01C21/30 takes precedence
• CPC Class Codes: G01C 21/10 by using measurements of speed or acceleration G01C21/24, G01C21/26 take precedence
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06V 10/464 using a plurality of salient features, e.g. bag-of-words [BoW] representations
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/56 : exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 13/261 : with monoscopic-to-stereoscopic image conversion
• CPC Class Codes: H04N 23/60 : Control of cameras or camera modules
• CPC Class Codes: H04N 23/73 : by influencing the exposure time
• CPC Class Codes: Y10S 901/01 : Mobile robot

Abstract

The described positional awareness techniques employing visual-inertial sensory data gathering and analysis hardware with reference to specific example implementations implement improvements in the use of sensors, techniques and hardware design that can enable specific embodiments to provide positional awareness to machines with improved speed and accuracy.

Claims

1. A system of guiding a mobile platform in an unmapped area and without producing a map of the unmapped area, the system including:

a mobile platform;

a sensory interface coupling to one or more sensors including at least some visual sensors and at least some inertial sensors, wherein the one or more sensors are configured to sense one or more of position, motion or environment of the mobile platform;

a processor coupled to the sensory interface and the mobile platform to provide guidance and control, and further coupled to a computer readable storage medium storing computer instructions configured for performing;

maintaining a set of time dependent tracking states that include;

(i) a pose and (ii) one or more frames of sensor readings including at least some frames from visual sensors wherein at least some frames include sets of 2D feature points located using image information from the visual sensors;

wherein each frame of sensor readings can include sensory information received from any one of a plurality of sensors available to the mobile platform;

selecting two or more tracking states from the set of time dependent tracking states;

creating a 2D feature correspondences set comprising of common 2D feature points among the visual sensor frames from the selected tracking states;

triangulating 2D feature points from the 2D feature correspondences set to form a 3D point set;

selecting a subset of the 3D point set that includes 3D points having re-projection error within a threshold;

updating current poses for at least two of the tracking states within the set of time dependent tracking states subject to a criterion including reduction of a total of re-projection errors of the selected 3D points; and

guiding the mobile platform using the updated current pose determined for that at least two tracking states.

2. The system of claim 1, wherein triangulating further includes selecting 3D points having a triangulation error less than a determined number of pixels.

3. The system of claim 1, further including adjusting a threshold for selecting 3d points to control propagation based upon a quantity of 3D points available to update poses in the set of time dependent tracking states.

4. The system of claim 1, further configured for performing dynamically determining a window size including a number of tracking states in the set of time dependent tracking states by maintaining a fixed manageable window size depending upon a parameter of at least one sensor.

5. The system of claim 4, further configured for performing keeping the window size to be 30 frames for a 30 Hz camera.

6. The system of claim 1, further configured for performing dynamically determining a window size including a number of tracking states in the set of time dependent tracking states by maintaining a range of time spanning using the window size determined.

7. The system of claim 6, further configured for performing, keeping frames for 1 sec.

8. The system of claim 1, further configured for performing dynamically determining a window size including a number of tracking states in the set of time dependent tracking states by maintaining a manageable number of 3D points.

9. The system of claim 8, further configured for performing maintaining a number of 3D points in tracking states using a window size of less than 200.

10. The system of claim 1, further configured for performing refining 3D points using a least squares function.

11. The system of claim 1, further configured for performing synchronizing time associated with at least one of the a set of time dependent tracking states with a time maintained by a controller unit controlling the mobile platform, by updating the time associated with the time associated with at least one of the a set of time dependent tracking states by (i) a constant offset and (ii) a transfer delay.

12. The system of claim 1, further including a motion detector, wherein the processor is further configured for performing:

tracking a set of inertial readings from the motion detector;

computing a variance for each dimension of a total of 3 dimensions of the set of inertial readings;

determining whether the mobile platform is moving or at rest based upon the variance computed; and

whenever the mobile platform is determined to be moving, performing extended Kalman filtering on the set of time dependent tracking states to determine pose information from the set of time dependent tracking states and whenever the mobile platform is determined to be static providing a computed pose whenever the mobile platform is determined to be at rest;

wherein the computed pose is determined by selectively performing at least one of;

(i) propagating a zero velocity applied to location information of the pose and change orientation information of the mobile platform using information obtained from a gyroscopic sensor; and

(ii) adding a noise factor to a previous pose.

13. The system of claim 12, further configured for performing:

detecting a number of IMU updates occurring after a successful image update; and

switching to gyroscopic sensor propagation after N consecutive IMU updates are detected.

14. The system of claim 12, further configured for performing:

detecting when one or more visual sensors has failed or is not updating or is insufficiently updating and whenever one or more visual sensors has failed or is not updating or is insufficiently updating, using the motion detector to control pose estimation.

15. The system of claim 14, wherein a variance above a threshold indicates a moving platform;

and a variance below a threshold indicates a static platform.

16. The system of claim 1, further configured for performing:

detecting a number of features in an image; and

dynamically adjusting an exposure value of at least one visual sensor whenever a lack of features the image is detected.

17. The system of claim 1, further configured for performing:

calculating a location of at least one 2D feature in an image using optical flow;

reducing features to features existing along an epipolar line based upon a location calculated and a prediction of a camera pose;

checking for similarity between descriptors of the feature and descriptors for a corresponding feature from a previous frame;

determining whether number of features through optical flow is less than a number of features detected in a current frame, and whenever the number of features through optical flow is less than a number of features, detecting 2D corner features directly from an image at image regions without features;

bin the image spatially, and adjust a number of features in each bin so that the number of features in the bins is balanced.

18. The system of claim 17, further configured for performing:

linking features in different images captured at different times by a same camera.

19. The system of claim 17, further configured for performing:

correlating unmatched features for a camera, by;

collecting a set of features that are not matched to any other features by existing links;

collecting a set of available links that are do not match to a feature from the camera in a current time frame;

calculating a similarity score between each member in the set of features and each member in the set of available links;

wherein similarity between a feature and a link is a best similarity score between a descriptor of the feature and descriptors of features in the link; and

when the similarity score is greater than a threshold, adding the feature to the link.

20. A non-transitory computer readable medium having instructions stored thereon for performing a method of guiding a mobile platform in an unmapped area and without producing a map of the unmapped area, including:

maintaining a set of time dependent tracking states that include;

(i) a pose and (ii) one or more frames of sensor readings including at least some frames from visual sensors wherein at least some frames include sets of 2D feature points located using image information from the visual sensors;

wherein each frame of sensor readings can include sensory information received from any one of a plurality of sensors available to the mobile platform;

selecting two or more tracking states from the set of time dependent tracking states;

creating a 2D feature correspondences set comprising of common 2D feature points among the visual sensor frames from the selected tracking states;

triangulating 2D feature points from the 2D feature correspondences set to form a 3D point set;

selecting a subset of the 3D point set that includes 3D points having re-projection error within a threshold;

updating current poses for at least two of the tracking states within the set of time dependent tracking states subject to a criterion including reduction of a total of re-projection errors of the selected 3D points; and

guiding the mobile platform using the updated current pose determined for that at least two tracking states.

21. A method of guiding a mobile platform in an unmapped area and without producing a map of the unmapped area, including:

maintaining a set of time dependent tracking states that include;

(i) a pose and (ii) one or more frames of sensor readings including at least some frames from visual sensors wherein at least some frames include sets of 2D feature points located using image information from the visual sensors;

wherein each frame of sensor readings can include sensory information received from any one of a plurality of sensors available to the mobile platform;

selecting two or more tracking states from the set of time dependent tracking states;

creating a 2D feature correspondences set comprising of common 2D feature points among the visual sensor frames from the selected tracking states;

triangulating 2D feature points from the 2D feature correspondences set to form a 3D point set;

selecting a subset of the 3D point set that includes 3D points having re-projection error within a threshold;

updating current poses for at least two of the tracking states within the set of time dependent tracking states subject to a criterion including reduction of a total of re-projection errors of the selected 3D points; and

guiding the mobile platform using the updated current pose determined for that at least two tracking states.