Positional awareness with quadocular sensor in autonomous platforms

• US 10,496,104 B1
• Filed: 07/05/2017
• Issued: 12/03/2019
• Est. Priority Date: 07/05/2017
• Status: Expired due to Fees

• Current Assignee: Perceptln Shenzhen Limited
• Original Assignee: Perceptln Shenzhen Limited
• Inventors: Liu, Shaoshan, Zhang, Zhe, Tsai, Grace
• Primary Examiner(s): Marc, Mcdieunel
• Application Number:
  US15/642,258
• Time in Patent Office: 881 Days
• CPC Class Codes: G01C 21/1656 with passive imaging devices, e.g. cameras
• CPC Class Codes: G01S 19/14 specially adapted for specific applications
• CPC Class Codes: G01S 19/23 Testing, monitoring, correcting or calibrating of receiver elements
• CPC Class Codes: G01S 19/485 whereby the further system is an optical system or imaging system
• CPC Class Codes: G01S 19/49 whereby the further system is an inertial position system, e.g. loosely-coupled
• CPC Class Codes: G01S 5/16 using electromagnetic waves other than radio waves
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/20072 Graph-based image processing
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30241 Trajectory
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30248 Vehicle exterior or interior
• CPC Class Codes: G06T 2207/30256 Lane; Road marking
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 : Deblurring; Sharpening
• CPC Class Codes: G06T 5/80 : Geometric correction
• CPC Class Codes: G06T 7/269 : using gradient-based methods
• CPC Class Codes: G06T 7/277 : involving stochastic approaches, e.g. using Kalman filters
• CPC Class Codes: G06T 7/579 : from motion
• CPC Class Codes: G06T 7/593 : from stereo images
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G07C 5/0816 : Indicating performance data, e.g. occurrence of a malfunction

Abstract

The described positional awareness techniques employing sensory data gathering and analysis hardware with reference to specific example implementations implement improvements in the use of sensors, techniques and hardware design that can enable specific embodiments to provide positional awareness to machines with improved speed and accuracy. The sensory data are gathered from multiple operational cameras and one or more auxiliary sensors.

Claims

1. A method for guiding an autonomous mobile platform, including:

receiving images from at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform;

determining, by a processor, a 360-degrees depth map from the frames;

estimating, by a processor, a propagated pose from the 360-degrees depth map;

fusing a first propagated pose from the depth map with a second propagated pose from an IMU to provide an intermediate propagated pose to an Extended Kalman Filter (EKF) propagator;

receiving at the EKF propagator a third propagated pose from a GPS receiver;

preparing, by a processor, an updated propagated pose from the intermediate and third propagated poses; and

providing the updated propagated pose to guide a mobile platform.

2. The method of claim 1, further comprising:

detecting a failure condition of the IMU; and

responsive to the failure condition of the IMU detected, determining the intermediate pose using the first propagated pose from the images.

3. The method of claim 1, further comprising:

detecting a failure condition of at least one of the at least 4 cameras;

when a single camera fails resulting in no 360-degrees view, responsively using image information from remaining cameras to determine the first propagated pose; and

when all cameras fail resulting in no image information, responsively using data from the IMU to determine the intermediate propagated pose.

4. The method of claim 1, further comprising:

detecting a failure condition of the GPS receiver; and

responsive to the failure condition, providing the intermediate propagated pose as the updated propagated pose.

5. The method of claim 1, wherein GPS provides position, velocity and covariance information in North-east down (NED) format and further comprising:

converting the intermediate propagated pose into NED format; and

wherein the EKF propagator produces propagated poses in North-east down (NED) format.

6. The method of claim 1, wherein GPS provides doppler, pseudo ranges and covariance information in Earth-centered, Earth-fixed (ECEF) format and further comprising:

converting the intermediate propagated pose into ECEF format; and

wherein the EKF propagator produces propagated poses in Earth-centered, Earth-fixed (ECEF) format.

7. The method of claim 1, wherein GPS provides NED position and covariance information in North-east down (NED) format and further comprising:

converting the intermediate propagated pose into NED format; and

wherein the EKF propagator produces propagated poses in North-east down (NED) format.

8. The method of claim 1, wherein the IMU propagates a position every 5 ms, and wherein an error is accumulated over time;

and every 100 ms, an updated information is received from the GPS receiver; and

wherein the updated information is used to correct the error.

9. A system including a processor and a computer readable storage unit storing instructions that when executed by the processor perform the method of claim 1.

10. A non-transitory computer readable medium storing instructions that when executed by a processor perform a method of claim 1.

11. A method for guiding an autonomous mobile platform, including:

receiving images from at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform;

determining, by a processor, a 360-degrees depth map from the frames;

estimating, by a processor, a propagated pose from the 360-degrees depth map;

fusing a first propagated pose from the depth map with a second propagated pose from an IMU to provide an intermediate propagated pose to an Extended Kalman Filter (EKF) propagator;

receiving at the EKF propagator a third propagated pose from an odometry sensor coupled with at least one wheel of the autonomous mobile platform;

preparing, by a processor, an updated propagated pose from the intermediate and third propagated poses; and

providing the updated propagated pose to guide a mobile platform.

12. The method of claim 11, further comprising:

detecting a failure condition of the IMU; and

responsive to the failure condition of the IMU detected, determining the intermediate pose using the first propagated pose from the images.

13. The method of claim 11, further comprising:

detecting a failure condition of at least one of the at least 4 cameras;

when a single camera fails resulting in no 360-degrees view, responsively using image information from remaining cameras to determine the first propagated pose; and

when all cameras fail resulting in no image information, responsively using data from the IMU to determine the intermediate propagated pose.

14. The method of claim 11, further comprising:

detecting a failure condition of the odometry sensor; and

responsive to the failure condition, providing the intermediate propagated pose as the updated propagated pose.

15. A system including a processor and a computer readable storage unit storing instructions that when executed by the processor perform the method of claim 11.

16. A non-transitory computer readable medium storing instructions that when executed by a processor perform a method of claim 11.

17. A method for guiding an autonomous mobile platform, including:

receiving images from at least 4 cameras providing at least two frames, each frame providing a 360-degrees view about a centerline of the mobile platform;

determining, by a processor, a 360-degrees depth map from the frames;

estimating, by a processor, a propagated pose from the 360-degrees depth map;

fusing a first propagated pose from the depth map with a second propagated pose from an IMU to provide a first intermediate propagated pose to an Extended Kalman Filter (EKF) propagator;

fusing the first intermediate propagated pose from the images and the IMU with a third propagated pose from an odometry sensor coupled to at least one wheel, to provide a second intermediate propagated pose to an EKF propagator;

receiving at the EKF propagator a fourth propagated pose from a GPS receiver;

preparing, by a processor, an updated propagated pose from the second intermediate and fourth propagated poses; and

providing the updated propagated pose to guide a mobile platform.

18. The method of claim 17, further comprising:

detecting a failure condition of the IMU; and

responsive to the failure condition of the IMU detected, determining the first intermediate pose using the first propagated pose from the images.

19. The method of claim 17, further comprising:

detecting a failure condition of at least one of the at least 4 cameras;

when a single camera fails resulting in no 360-degrees view, responsively using image information from remaining cameras to determine the first propagated pose; and

when all cameras fail resulting in no image information, responsively using data from the IMU to determine the first intermediate propagated pose.

20. The method of claim 17, further comprising:

detecting a failure condition of the odometry sensor; and

responsive to the failure condition, providing the first intermediate propagated pose as the second intermediate propagated pose.

21. The method of claim 17, further comprising:

detecting a failure condition of the GPS receiver; and

responsive to the failure condition, providing the second intermediate propagated pose as the updated propagated pose.

22. A system including a processor and a computer readable storage unit storing instructions that when executed by the processor perform the method of claim 17.

23. A non-transitory computer readable medium storing instructions that when executed by a processor perform a method of claim 17.