Mobile robot system and method for generating map data using straight lines extracted from visual images

• US 10,732,641 B2
• Filed: 05/02/2018
• Issued: 08/04/2020
• Est. Priority Date: 11/02/2015
• Status: Active Grant

• Current Assignee: Starship Technologies Ou (Starship Technologies, Inc.)
• Original Assignee: Starship Technologies Ou (Starship Technologies, Inc.)
• Inventors: Heinla, Ahti, Volkov, Kalle-Rasmus, Roberts, Lindsay, Mandre, Indrek
• Primary Examiner(s): Shafi, Muhammad
• Application Number:
  US15/968,802
• Publication Number:
  US 20180253108A1
• Time in Patent Office: 825 Days
• Field of Search: 701 25
• CPC Class Codes: G01C 21/3848 Data obtained from both position sensors and additional sensors
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0212 with means for defining a desired trajectory involving a plurality of land vehicles G05D1/0287
• CPC Class Codes: G05D 1/0223 involving speed control of the vehicle vehicle fittings for automatically controlling, i.e. preventing speed from exceeding an arbitrarily established velocity or maintaining speed at a particular velocity, as selected by the vehicle operator B60K31/00
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0251 extracting 3D information from a plurality of images taken from different locations, e.g. stereo vision stereoscopic image analysis H04N13/00; depth recovery from images G06T7/593
• CPC Class Codes: G05D 1/0255 using acoustic signals, e.g. ultra-sonic singals sonar systems designed for anti-collision purposes G01S15/93
• CPC Class Codes: G05D 1/027 comprising intertial navigation means, e.g. azimuth detector inertial navigation G01C21/16; inertial navigation combined with non-inertial navigation instruments G01C21/165
• CPC Class Codes: G05D 1/0272 comprising means for registering the travel distance, e.g. revolutions of wheels measuring distance traversed on the ground by vehicles, e.g. using odometers G01C22/00
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0278 using satellite positioning signals, e.g. GPS
• CPC Class Codes: G05D 1/227 Handing over between remote control and on-board control; Handing over between remote control arrangements
• CPC Class Codes: G05D 1/2435 Extracting 3D information
• CPC Class Codes: G05D 1/245 using dead reckoning
• CPC Class Codes: G05D 1/246 using environment maps, e.g. simultaneous localisation and mapping [SLAM]
• CPC Class Codes: G05D 1/247 using signals provided by artificial sources external to the vehicle, e.g. navigation beacons
• CPC Class Codes: G05D 1/248 generated by satellites, e.g. GPS
• CPC Class Codes: G05D 1/249 from positioning sensors located off-board the vehicle, e.g. from cameras
• CPC Class Codes: G05D 1/644 Optimisation of travel parameters, e.g. of energy consumption, journey time or distance
• CPC Class Codes: G05D 1/646 : Following a predefined trajectory, e.g. a line marked on the floor or a flight path
• CPC Class Codes: G05D 1/65 : Following a desired speed profile
• CPC Class Codes: G06Q 10/0833 : Tracking
• CPC Class Codes: G06T 2207/10004 : Still image; Photographic image
• CPC Class Codes: G06T 2207/10016 : Video; Image sequence
• CPC Class Codes: G06T 2207/20212 : Image combination
• CPC Class Codes: G06T 2207/30244 : Camera pose
• CPC Class Codes: G06T 2207/30252 : Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 7/579 : from motion
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06T 7/97 : Determining parameters from multiple pictures depth or shape recovery from multiple images G06T7/55; stereo camera calibration G06T7/85

Abstract

A mobile robot is configured to navigate on a sidewalk and deliver a delivery to a predetermined location. The robot has a body and an enclosed space within the body for storing the delivery during transit. At least two cameras are mounted on the robot body and are adapted to take visual images of an operating area. A processing component is adapted to extract straight lines from the visual images taken by the cameras and generate map data based at least partially on the images. A communication component is adapted to send and receive image and/or map data. A mapping system includes at least two such mobile robots, with the communication component of each robot adapted to send and receive image data and/or map data to the other robot. A method involves operating such a mobile robot in an area of interest in which deliveries are to be made.

Claims

1. A mobile robot mapping method for generating a map, comprising:

(a) providing a mobile robot configured to navigate outdoors on a sidewalk to deliver an item, the mobile robot having a body and a space for holding the item while in transit, the mobile robot further comprising at least two cameras mounted on the body which point in different directions and have non-overlapping fields of view, and at least one processing component;

(b) operating the mobile robot in an outdoor operating area;

(c) taking visual images with the at least two cameras;

(d) extracting at least straight lines from the visual images with at least one processing component, the extracted straight lines corresponding to contours of permanent physical objects in the visual images; and

(e) generating map data using at least the extracted straight lines from the visual images;

wherein;

step (d) comprises;

detecting that a certain number of lines all belong to the same physical object and merging them into one line; and

detecting that a certain line is likely to belong to a transitory physical object and not assigning that line to a physical object.

2. The mobile robot method according to claim 1, further comprising combining the visual images and/or the extracted straight lines in a single reference frame based on the relative placement of the two cameras.

3. The mobile robot method according to claim 1, further comprising:

extracting location-related data from one or more sensors adapted to measure further parameters for building map data, said one or more sensors being from the group consisting of a GPS component, an accelerometer, a gyroscope, an odometer, a magnetometer, a pressure sensor, an ultrasonic sensor, a time of flight camera sensor, and a Lidar sensor; and

combining this location-related data with data from the visual images to build map data.

4. The mobile robot method according to claim 1, wherein, in step (e) generating map data comprises using an iterative algorithm to at least associate the extracted straight lines with physical objects.

5. The mobile robot method according to claim 4, wherein the iterative algorithm comprises an optimization routine optimizing at least one robot pose and the position of the physical objects based on the extracted straight lines.

6. The mobile robot method according to claim 4, wherein the iterative algorithm comprises at least combining lines belonging to the same physical object from images taken by different cameras and/or discarding lines belonging to transient objects and/or light or camera effects.

7. The mobile robot method according to claim 1, further comprising capping errors associated with the robot'"'"'s sensors.

8. The mobile robot method according to claim 7, comprising capping errors associated with systematic sensor errors and/or invalid physical object associations of the extracted straight lines from the visual images.

9. The mobile robot method according to claim 1, wherein:

the mobile robot is an autonomous and/or semi-autonomous mobile robot; and

the method performs simultaneous mapping and localization (SLAM).

10. The mobile robot method of claim 1, comprising:

operating the mobile robot in an outdoor operating area at a speed of no more than 10 km/h.

11. A mobile robot configured to navigate outdoors on a sidewalk and deliver an item, the mobile robot having a body and a space for holding the item while in transit, the robot comprising:

at least two cameras mounted on the body and adapted to take visual images of an operating area, the at least two cameras pointing in different directions and having non-overlapping fields of view;

at least one processing component adapted to extract straight lines from the visual images taken by the at least two cameras and generate map data based at least partially on straight lines corresponding to contours of permanent physical objects in the images; and

a communication component adapted to send and receive image and/or map data;

wherein;

the at least one processing component is configured to;

detect that a certain number of lines all belong to the same physical object and merge them into one line; and

detect that a certain line is likely to belong to a transitory physical object and not assign that line to a physical object.

12. The mobile robot according to claim 11, wherein:

each camera is adapted to capture 3 to 5 images per second.

13. The mobile robot according to claim 11, comprising:

at least 4 pairs of stereo cameras, members of each pair of stereo cameras located on the mobile robot so as have overlapping fields of view and provide depth information.

14. The mobile robot according claim 11, wherein the map data comprises:

one or more of vectors, point features and grid features associated with physical objects, and defined with respect to a coordinate system, and/or

error estimates for one or more of vectors, point features and grid features associated with said physical objects.

15. The mobile robot according to claim 11, wherein the map data comprises visibility information related to locations from which physical objects can and/or cannot be seen.

16. The mobile robot according to claim 11, wherein:

the communication component is adapted to send map data and/or image data to a server which is configured to refine existing map data using the map data and/or image data sent by the robot'"'"'s communication component.

17. The mobile robot according to claim 16, configured to navigate using the refined map data received from the server.

18. The mobile robot according to claim 11, further comprising:

one or more sensors adapted to measure further parameters for building map data, said one or more sensors being from the group consisting of a GPS component, an accelerometer, a gyroscope, an odometer, a magnetometer, a pressure sensor, an ultrasonic sensor, a time of flight camera sensor, and a Lidar sensor.

19. The mobile robot according to claim 11, wherein the robot is autonomous and/or semi-autonomous.

20. The mobile robot according to claim 11, wherein:

the robot is adapted to travel with a speed of no more than 10 km/h.

21. The mobile robot according to claim 11, comprising a first pair of stereo cameras mounted on a front of the body, second and third pairs of stereo cameras mounted on opposite sides of the body and a fourth pair of stereo cameras mounted on a back of the body.

22. The mobile robot according to claim 11, comprising an enclosed space within the body for holding an item to be delivered to a predetermined delivery location.

23. The mobile robot according to claim 22, wherein:

the at least one processing component is adapted to generate said map data, while the mobile robot is in transit to said predetermined delivery location.

24. The mobile robot according to claim 11, wherein:

the at least one processing component is configured to generate map data using an iterative algorithm.

25. A mapping system comprising:

at least two mobile robots, each mobile robot configured to navigate outdoors on a sidewalk and deliver an item, the mobile robot having a body and a space for holding the item while in transit, each mobile robot comprising;

at least two cameras mounted on the body and adapted to take visual images of an operating area, the at least two cameras pointing in different directions and having non-overlapping fields of view;

at least one processing component adapted to extract straight lines from the visual images taken by the at least two cameras and generate map data based at least partially on straight lines corresponding to contours of permanent physical objects in the images; and

a communication component adapted to send and receive image and/or map data;

wherein;

the at least one processing component is configured to;

detect that a certain number of lines all belong to the same physical object and merge them into one line; and

detect that a certain line is likely to belong to a transitory physical object and not assign that line to a physical object wherein;

the communication component of each robot is adapted to send and receive image data and/or map data to the other of the at least two robots.

26. The mapping system according to claim 25, wherein:

the data sent between the robots is combinable into a map of multiple operating areas.

27. The mapping system according to claim 25, further comprising a server adapted to:

communicate with the robots via their communication component;

receive map data and/or image data from a first robot;

process the map data; and

send the processed map data to a second robot, in response to a request from the second robot.

28. The mapping system according to claim 27, wherein the server is adapted to:

receive image and/or map data from the at least two robots; and

further process these data to generate at least further map data.