Visual-inertial positional awareness for autonomous and non-autonomous device

• US 10,032,276 B1
• Filed: 08/29/2016
• Issued: 07/24/2018
• Est. Priority Date: 08/29/2016
• Status: Active Grant

• Current Assignee: Perceptln Shenzhen Limited
• Original Assignee: PerceptIn, Inc.
• Inventors: Liu, Shaoshan, Zhang, Zhe, Tsai, Grace
• Primary Examiner(s): Johns, Andrew W
• Application Number:
  US15/250,419
• Time in Patent Office: 694 Days
• CPC Class Codes: G01P 1/127 for acceleration values
• CPC Class Codes: G01P 15/18 in two or more dimensions
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/292 Multi-camera tracking
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

The described positional awareness techniques employing visual-inertial sensory data gathering and analysis hardware with reference to specific example implementations implement improvements in the use of sensors, techniques and hardware design that can enable specific embodiments to provide positional awareness to machines with improved speed and accuracy.

Claims

1. A system including:

a mobile platform controllable by a host, the mobile platform having disposed thereon;

a visual sensor comprising at least one RGB sensing capable camera and at least one other grayscale camera disposed at a distance relative to one another to form a region in which the fields of view at least partially overlap, thereby providing stereoscopic imaging capability;

a multi-axis inertial measuring unit (IMU) capable of providing measurement of at least acceleration; and

a visual inertial control unit, including;

a first interface that couples to the visual sensor to receive sets of image data;

a second interface that couples to the multi-axis IMU to receive accelerometer sensor data;

a cache storage that stores the sets of image data;

a single instruction-multiple data processing element having direct memory access to the cache storage;

an inertial measurement engine that performs time stamping of inertial data received via the second interface, corrects the timestamped inertial data for bias, applies a stored scale factor to the corrected inertial data and corrects the scaled inertial data for misalignment in the IMU to form localization data;

an imaging engine that performs imaging undistortion on the sets of image data; and

a communications interface to provide the localization data and the undistorted sets of image data to a host controlling the mobile platform.

2. The system of claim 1, wherein the visual inertial control unit further:

detects features in the undistorted sets of images; and

provides the detected features to the host via the communications interface.

3. The system of claim 2, wherein features are detected using Shi-Tomasi feature detection.

4. The system of claim 2, wherein the imaging engine further:

determines feature descriptors for the detected features; and

provides the feature descriptors to the host via the communications interface.

5. The system of claim 2, wherein the imaging engine further:

determines feature correspondences between features detected from successive images in the sets of images; and

provides the feature correspondences to the host via the communications interface.

6. The system of claim 5, wherein feature correspondences are determined using optical flow.

7. The system of claim 1, wherein the imaging engine further:

generates a bag of words description for an image frame to be used in re-localization.

8. An apparatus for guiding a mobile device using information from one or more cameras with distance calculation and multi-axis inertial measuring unit (IMU), the apparatus including:

a first interface that couples to the one or more cameras to receive sets of image data;

a second interface that couples to a multi-axis IMU to receive accelerometer sensor data;

a cache storage that stores the sets of image data;

a single instruction-multiple data processing element having direct memory access to the cache storage;

an inertial measurement engine that performs time stamping of inertial data received via the second interface, corrects inertial readouts in the timestamped inertial data for bias, applies a stored scale factor to the corrected inertial data and corrects the scaled inertial data for misalignment in the IMU to form localization data;

an imaging engine that performs imaging undistortion on the sets of image data; and

a communications interface to provide the localization data and the undistorted sets of image data to a host controlling the mobile device.

9. The apparatus of claim 8, wherein the apparatus further includes the imaging engine:

detects features in the undistorted sets of images; and

provides the detected features to the host via the communications interface.

10. The apparatus of claim 9, wherein features are detected using Shi-Tomasi feature detection.

11. The apparatus of claim 9, wherein the imaging engine further:

determines feature descriptors for the detected features; and

provides the feature descriptors to the host via the communications interface.

12. The apparatus of claim 9, wherein the imaging engine further:

determines feature correspondences between features detected from successive images in the sets of images; and

provides the feature correspondences to the host via the communications interface.

13. The apparatus of claim 12, wherein feature correspondences are determined using optical flow.

14. The apparatus of claim 8, wherein the imaging engine further:

generates a bag of words description for an image frame to be used in re-localization.

15. A non-transitory computer readable storage medium impressed with computer program instructions to guide a mobile device using information from a camera with distance calculation and multi-axis inertial measuring unit (IMU), the instructions, when executed on a processor, implement a method, the method including:

buffering image sets from a visual sensor comprising at least one RGB sensing capable camera and from at least one other grayscale camera disposed at a distance relative to one another to form a region in which the fields of view at least partially overlap, thereby providing stereoscopic imaging capability;

buffering inertial measurements from a multi-axis inertial measuring unit (IMU) capable of providing measurement of at least acceleration;

receiving at a visual inertial control unit the sets of image data;

receiving at the visual inertial control unit sensor data from the multi-axis IMU;

time stamping by an inertial measurement engine the inertial data received,

correcting inertial readouts in the timestamped inertial data for bias,

scaling the inertial readouts using a stored scale factor the corrected inertial data;

correcting the scaled inertial data for misalignment in the IMU to form localization data;

performing imaging undistortion on the sets of image data; and

providing across a communications interface the localization data and the undistorted sets of image data to a host controlling a mobile platform.

16. The non-transitory computer readable storage medium of claim 15, implementing the method further including:

detecting features in the undistorted sets of images; and

providing the detected features to the host via the communications interface.

17. The non-transitory computer readable storage medium of claim 16, wherein the features are detected using Shi-Tomasi feature detection.

18. The non-transitory computer readable storage medium of claim 16, implementing the method further including:

determining feature descriptors for the detected features; and

providing the feature descriptors to the host via the communications interface.

19. The non-transitory computer readable storage medium of claim 16, implementing the method further including:

determining feature correspondences between features detected from successive images in the sets of images; and

providing the feature correspondences to the host via the communications interface.

20. The non-transitory computer readable storage medium of claim 19, wherein feature correspondences are determined using optical flow.

21. The non-transitory computer readable storage medium of claim 15, implementing the method further including:

generating a bag of words description for an image frame to be used in re-localization.