MULTIMODAL LOCALIZATION AND MAPPING FOR A MOBILE AUTOMATION APPARATUS

• US 20180315007A1
• Filed: 05/01/2017
• Published: 11/01/2018
• Est. Priority Date: 05/01/2017
• Status: Active Grant

• Current Assignee: Symbol Technologies, LLC (Zebra Technologies Corporation)
• Original Assignee: Symbol Technologies, LLC (Zebra Technologies Corporation)
• Inventors: Kingsford, Thomas D., Liu, Chengzhi W., Singh, Harsoveet, Barfoot, Timothy D.
• Granted Patent: US 10,949,798 B2
• Time in Patent Office: Days
• CPC Class Codes: G01C 21/3647 Guidance involving output of stored or live camera images or video streams
• CPC Class Codes: G06Q 10/087 Inventory or stock management, e.g. order filling, procurement or balancing against orders
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06T 7/97 Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85
• CPC Class Codes: G06V 10/40 Extraction of image or video features
• CPC Class Codes: G06V 10/757 Matching configurations of points or features
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

A device and method for multimodal localization and mapping for a mobile automation apparatus is provided. Features are extracted from images acquired by a ceiling-facing image device of the mobile automation apparatus in an environment, and stored in association with estimated positions of the mobile automation apparatus in the environment as determined from one or more proprioceptive sensors, as well as in association with features extracted from depth data acquired from a depth-sensing device, for example as map data. The map data is later used by the mobile automation apparatus to navigate the environment based, at least in part, on further images acquired by the ceiling-facing image device.

Claims

1. A mobile automation apparatus for navigating an environment having a ceiling comprising:

a movement module for moving the mobile automation apparatus in the environment;

an image device configured to acquire images in a ceiling-facing direction from the mobile automation apparatus;

a proprioceptive sensor configured to monitor movement of the mobile automation apparatus; and

a controller configured to;

associate the images received from the image device with proprioceptive data received from the proprioceptive sensor.

2. The mobile automation apparatus of claim 1, wherein the controller is further configured to one or more of;

store, to a memory, the images with associated proprioceptive data; and

transmit, using a communication interface, the images with associated proprioceptive data to a mapping device.

3. The mobile automation apparatus of claim 1, wherein the image device comprises one or more of a monocular camera, a stereo camera, a color camera, a monochrome camera, a depth camera, a structured light camera, a time-of-flight camera, and a LIDAR device.

4. The mobile automation apparatus of claim 1, wherein the proprioceptive data acquired by the proprioceptive sensor is indicative of a distance travelled by the mobile automation apparatus as the movement module moves the mobile automation apparatus through the environment.

5. The mobile automation apparatus of claim 1, wherein the proprioceptive data is further indicative of changes in direction of the mobile automation apparatus as the movement module moves the mobile automation apparatus through the environment.

6. The mobile automation apparatus of claim 1, wherein the proprioceptive sensor comprises one or more of an odometry device, a wheel odometry device, an accelerometer, a gyroscope and an inertial measurement unit.

7. The mobile automation apparatus of claim 1, wherein the proprioceptive data and the images are one more or more of:

time-stamped;

acquired periodically; and

acquired as a function of time.

8. The mobile automation apparatus of claim 1, further comprising a depth-sensing device configured to acquire depth data in one or more directions from the mobile automation apparatus, the controller further configured to:

associate depth data received from the depth-sensing device with the images and the proprioceptive data.

9. The mobile automation apparatus of claim 8, wherein the controller is further configured to one or more of:

store, to a memory, the images with associated proprioceptive data and associated depth data; and

transmit, using a communication interface, the images with the associated proprioceptive data and the associated depth data.

10. The mobile automation apparatus of claim 8, wherein the depth-sensing device comprises a LIDAR device.

11. The mobile automation apparatus of claim 1, wherein the controller is further configured to one or more of navigate the environment and determine an adjusted position in the environment by comparing further features extracted from further images acquired by the image device with extracted features from the images, that are associated with positions in the environment.

12. A method comprising:

at a mobile automation apparatus comprising;

a movement module for moving the mobile automation apparatus in an environment;

an image device configured to acquire images in a ceiling-facing direction from the mobile automation apparatus;

a proprioceptive sensor configured to monitor movement of the mobile automation apparatus; and

a controller, associating, using the controller, the images received from the image device with proprioceptive data received from the proprioceptive sensor.

13. The method of claim 12, further comprising or more of;

storing, to a memory, the images with associated proprioceptive data; and

transmitting, using a communication interface, the images with associated proprioceptive data to a mapping device.

14. A device comprising:

a communication interface;

a memory; and

a mapping controller configured to;

receive, from one or more of the memory and the communication interface, images with associated proprioceptive data, the images representing a ceiling of an environment acquired by a ceiling-facing imaging device at a mobile automation apparatus and the associated proprioceptive data indicative of movement of the mobile automation apparatus through the environment;

extract features from the images;

determine, from the associated proprioceptive data, a respective position of a mobile automation apparatus in the environment where each of the images was acquired; and

,

store, in the memory, extracted features from the images with associated positions.

15. The device of claim 14, wherein the mapping controller is further configured to:

receive, from one or more of the memory and the communication interface, the images with the associated proprioceptive data and associated depth data indicative of objects located in one or more directions from the mobile automation apparatus; and

store, in the memory, the extracted features from the images with the associated positions and further features extracted from the associated depth data.

16. The device of claim 15, wherein the associated positions are stored in association with the extracted features and the further features as map data, and the mapping controller is further configured to refine the map data based on one or more of further images, further depth data, and further proprioceptive data received from the mobile automation apparatus.

17. A method comprising:

at a device comprising;

a communication interface;

a memory; and

a mapping controller, receiving, from one or more of the memory and the communication interface, images with associated proprioceptive data, the images representing a ceiling of an environment acquired by a ceiling-facing imaging device at a mobile automation apparatus and the associated proprioceptive data indicative of movement of the mobile automation apparatus through the environment;

extract, using the mapping controller, features from the images;

determining, using the mapping controller, from the associated proprioceptive data, a respective position of a mobile automation apparatus in the environment where each of the images was acquired; and

,

storing, using the mapping controller, in the memory, extracted features from the images with associated positions.

18. The method of claim 17, further comprising:

receiving, from one or more of the memory and the communication interface, the images with the associated proprioceptive data and associated depth data indicative of objects located in one or more directions from the mobile automation apparatus; and

storing, in the memory, the extracted features from the images with the associated positions and further features extracted from the associated depth data.

19. The method of claim 18, wherein the associated positions are stored in association with the extracted features and the further features as map data, and method further comprises refining the map data based on one or more of further images, further depth data, and further proprioceptive data received from the mobile automation apparatus.