Mobile robot system and method for autonomous localization using straight lines extracted from visual images

• US 10,386,850 B2
• Filed: 05/02/2018
• Issued: 08/20/2019
• Est. Priority Date: 11/02/2015
• Status: Active Grant

• Current Assignee: Starship Technologies Ou (Starship Technologies, Inc.)
• Original Assignee: Starship Technologies Ou (Starship Technologies, Inc.)
• Inventors: Heinla, Ahti, Volkov, Kalle-Rasmus, Roberts, Lindsay, Mandre, Indrek
• Primary Examiner(s): Tissot, Adam D
• Application Number:
  US15/968,838
• Publication Number:
  US 20180253107A1
• Time in Patent Office: 475 Days
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0223 involving speed control of the vehicle vehicle fittings for automatically controlling, i.e. preventing speed from exceeding an arbitrarily established velocity or maintaining speed at a particular velocity, as selected by the vehicle operator B60K31/00
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G06Q 10/0833 Tracking
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G06T 7/97 : Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

A mobile delivery robot has at least one memory component containing at least map data; at least two cameras adapted to take visual images; and at least one processing component. The at least one processing component is adapted to at least extract straight lines from the visual images taken by the at least two cameras and compare them to the map data to at least localize the robot. The mobile robot employs a localization method which involves taking visual images with at least two cameras; extracting straight lines from the individual visual images with at least one processing component; comparing the extracted features with existing map data; and outputting a location hypothesis based on said comparison.

Claims

1. A mobile robot configured to navigate outdoors on a sidewalk to deliver an item, the mobile robot comprising:

a body and a space for holding an item for delivery while in transit,

at least one memory component comprising existing map data, the existing map data comprising information reflective of straight lines extracted from past visual images which were previously taken in the robot'"'"'s current operating area;

at least two cameras mounted on the body and adapted to take new visual images, said at least two cameras pointing in different directions and having non-overlapping fields of view; and

at least one processing component adapted to;

extract new straight lines from the new visual images taken by the at least two cameras; and

compare the extracted new straight lines to the existing map data to obtain different pose possibilities of the mobile robot, and create localization data to localize the robot by executing an iterative algorithm which evaluates the likelihood of each pose possibility being the true one to estimate mobile robot pose, said mobile robot pose comprising a combination of position and orientation.

2. The mobile robot according to claim 1, wherein each camera is adapted to capture 3-5 images per second.

3. The mobile robot according to claim 1, comprising

at least 4 pairs of stereo cameras, members of each pair of stereo cameras located on the mobile robot so as have overlapping fields of view and provide depth information.

4. The mobile robot according to claim 1, wherein the processing component is adapted to localize the robot with an error of at most 10 cm.

5. The mobile robot according to claim 1, adapted to navigate using the localization data from the processing component.

6. The mobile robot according to claim 5, wherein the processing component is adapted to create localization data by:

obtaining an approximate location from at least one or more a GPS component, an accelerometer, a gyroscope, an odometer, a magnetometer, a pressure sensor, an ultrasonic sensor, a time of flight camera sensor, and a Lidar sensor; and

refining the approximate location based on processing features extracted from the new visual images.

7. The mobile robot according to claim 1, further comprising:

a communication component adapted to exchange image data and/or map data with one or more servers, the communication comprising one or more of a slot for a Subscriber Identity Module (SIM card), a modem and a network device.

8. The mobile robot according to claim 7, wherein the robot is further adapted to receive navigation instructions from the server at specific intervals and/or after requesting input.

9. The mobile robot according to claim 1, wherein the robot is adapted to move autonomously and/or semi-autonomously.

10. The mobile robot according to claim 1, wherein the robot is adapted to travel with a speed of no more than 10 km/h.

11. The mobile robot according to claim 1, wherein said iterative algorithm is at least partially based on a particle filter algorithm.

12. The mobile robot according to claim 11, wherein:

the iterative algorithm is adapted to generate a hypothesis on a robot'"'"'s pose by processing data from one or more of a camera, a GPS component, an odometer, a gyroscope, an accelerometer, a Lidar sensor, a time of flight camera, an ultrasonic sensor, a pressure sensor, a dead-reckoning sensor, and a magnetometer.

13. The mobile robot according to claim 12, wherein the processing component is adapted to:

receive input data from at least one camera and at least one other sensor;

weight the input data based on errors associated with the at least one camera and the at least one other sensor; and

generate an estimate of the robot'"'"'s pose based at least in part on the weighted input data.

14. The mobile robot according to claim 1, comprising a first pair of stereo cameras mounted on a front of the body, second and third pairs of stereo cameras mounted on opposite sides of the body and a fourth pair of stereo cameras mounted on a back of the body.

15. The mobile robot according to claim 1, comprising an enclosed space within the body for holding an item to be delivered to a predetermined delivery location.

16. The mobile robot according to claim 15, wherein:

the at least one processing component is adapted to extract new straight lines and compare the extracted new straight lines with the existing map data, while the mobile robot is in transit to said predetermined delivery location.

23. The mobile robot according to claim 1, wherein:

the existing map data comprises straight lines belonging to buildings and/or fences and/or sidewalks.

17. A mobile robot localization method comprising:

(a) taking new visual images with at least two cameras which point in different directions and have non-overlapping fields of view;

(b) extracting new straight lines from the individual new visual images with at least one processing component;

(c) comparing the extracted new straight lines with existing map data comprising information reflective of straight lines extracted from past visual images which were previously taken in the robot'"'"'s current operating area;

(d) outputting a location hypothesis based on the comparison in step (c); and

(e) outputting a mobile robot pose using the outputted location hypothesis of step (d) and an iterative algorithm which evaluates the likelihood of each of a plurality of different pose possibilities being the true one, said mobile robot pose comprising a combination of position and orientation.

18. The mobile robot localization method according to claim 17, wherein the extracting step and/or the comparing step comprises using an iterative probabilistic algorithm to determine a best location hypothesis given the existing map data.

19. The mobile robot localization method according to claim 17, wherein the outputting step and/or the comparing step is performed using a particle filter algorithm.

20. The mobile robot localization method according to claim 17, further comprising in step (d):

extracting second location related input data from one or more of a GPS component, an odometer, a gyroscope, an accelerometer, a Lidar sensor, a time of flight camera, an ultrasonic sensor, a pressure sensor, a dead-reckoning sensor, and a magnetometer; and

combining first location related data obtained from features extracted from the new visual images with the second location related data to form a more precise location hypothesis.

21. The mobile robot localization method according to claim 17, further comprising:

navigating a mobile robot using the outputted mobile robot pose.

22. The mobile robot localization method according to claim 17, wherein:

the existing map data comprises straight lines belonging to buildings and/or fences and/or sidewalks.