Systems and methods for performing simultaneous localization and mapping using machine vision systems

• US 9,519,289 B2
• Filed: 09/16/2015
• Issued: 12/13/2016
• Est. Priority Date: 11/26/2014
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Munich, Mario, Romanov, Nikolai, Goel, Dhiraj, Fong, Philip
• Primary Examiner(s): Tarcza, Thomas
• Assistant Examiner(s): Dunn, Alex C
• Application Number:
  US14/856,526
• Publication Number:
  US 20160147230A1
• Time in Patent Office: 454 Days
• US Class Current: 1/1
• CPC Class Codes: B25J 19/005 using batteries, e.g. as a back-up power source
• CPC Class Codes: B25J 5/007 mounted on wheels
• CPC Class Codes: G01C 21/206 specially adapted for indoor navigation
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

The present invention provides a mobile robot configured to navigate an operating environment, that includes a controller circuit that directs a drive of the mobile robot to navigate the mobile robot through an environment using camera-based navigation system and a camera including optics defining a camera field of view and a camera optical axis, where the camera is positioned within the recessed structure and is tilted so that the camera optical axis is aligned at an acute angle of above a horizontal plane in line with the top surface and is aimed in a forward drive direction of the robot body, and the camera is configured to capture images of the operating environment of the mobile robot.

Claims

1. A mobile robot configured to navigate an operating environment, comprising:

a body having a top surface;

a drive mounted to the body;

a recessed structure beneath the plane of the top surface near a geometric center of the body;

a controller circuit in communication with the drive, wherein the controller circuit directs the drive to navigate the mobile robot through an environment using camera-based navigation system; and

a camera including optics defining a camera field of view and a camera optical axis, wherein;

the camera is positioned within the recessed structure and is tilted so that the camera optical axis is aligned at an acute angle of 30-40 degrees above a horizontal plane in line with the top surface and is aimed in a forward drive direction of the robot body,

the field of view of the camera spans a frustum of 45-65 degrees in the vertical direction, and

the camera is configured to capture images of the operating environment of the mobile robot.

2. The mobile robot of claim 1, wherein the camera is protected by a lens cover aligned at an acute angle with respect to the optical axis of the camera.

3. The mobile robot of claim 2, wherein the lens cover is set back relative to an opening of the recessed structure and an acute angle with respect to the optical axis of the camera that is closer to perpendicular than an angle formed between a plane defined by the top surface and the optical axis of the camera.

4. The mobile robot of claim 2, wherein the acute angle is between 15 and 70 degrees.

5. The mobile robot of claim 3, wherein the angle formed between a plane defined by the opening in the recessed structure and the optical axis of the camera ranges between 10 and 60 degrees.

6. The mobile robot of claim 1, wherein the camera field of view is aimed at static features located in a range of 3 feet to 8 feet from a floor surface at a distance of 3 feet to 10 feet from the static features.

7. The mobile robot of claim 6, wherein the camera images contain about 6-12 pixels per inch and features at the top of the image move upward between successive images more quickly than the speed at which the mobile robot moves and features at the bottom of the image move downward between successive images more slowly than the speed at which the mobile robot moves and wherein the controller is configured to determine the speed of the mobile robot and location of features in the image in identifying disparity between successive images.

8. The mobile robot of claim 7 wherein the mobile robot moves at a velocity of 220 mm per second to 450 mm per second and features lower than 45 degrees relative to the horizon will track slower than approximately 306 mm per second and features higher than 45 degrees will track faster than 306 mm per second.

9. The mobile robot of claim 1, wherein the body further contains:

a memory in communication with the controller circuit; and

an odometry sensor system in communication with the drive,

wherein the memory further contains a visual measurement application, a simultaneous location and mapping (SLAM) application, a landmarks database, and a map of landmarks,

wherein the controller circuit directs a processor to;

actuate the drive and capture odometry data using the odometry sensor system;

acquire a visual measurement by providing at least the captured odometry information and the captured image to the visual measurement application;

determine an updated robot pose within an updated map of landmarks by providing at least the odometry information, and the visual measurement as inputs to the SLAM application; and

determine robot behavior based upon inputs including the updated robot pose within the updated map of landmarks.

10. The mobile robot of claim 9, wherein the landmarks database comprises:

descriptions of a plurality of landmarks;

a landmark image of each of the plurality of landmarks and an associated landmark pose from which the landmark image was captured; and

descriptions of a plurality of features associated with a given landmark from the plurality of landmarks including a 3D position for each of the plurality of features associated with the given landmark.

11. The mobile robot of claim 10, wherein the visual measurement application directs the processor to:

identify features within an input image;

identify a landmark from the landmark database in the input image based upon the similarity of the features identified in the input image to matching features associated with a landmark image of the identified landmark in the landmark database; and

estimate a most likely relative pose by determining a rigid transformation of the 3D structure of the matching features associated with the identified landmark that results in the highest degree of similarity with the identified features in the input image, where the rigid transformation is determined based upon an estimate of relative pose and the acute angle at which the optical axis of the camera is aligned above the direction of motion of the mobile robot.

12. The mobile robot of claim 11, wherein identifying a landmark in the input image comprises comparing unrectified image patches from the input image to landmark images within the landmark database.

13. The mobile robot of claim 11, wherein the SLAM application directs the processor to:

estimate the location of the mobile robot within the map of landmarks based upon a previous location estimate, odometry data and at least one visual measurement; and

update the map of landmarks based upon the estimated location of the mobile robot, the odometry data, and the at least one visual measurement.

14. The mobile robot of claim 9, wherein the visual measurement application further directs the processor to generate new landmarks by:

detecting features within images in a sequence of images;

identifying a set of features forming a landmark in multiple images from the sequence of images;

estimating 3D structure of the set of features forming a landmark and relative robot poses at the times each of the multiple images is captured using the identified set of features forming the landmark in each of the multiple images;

recording a new landmark in the landmark database, where recording the new landmark comprises storing;

an image of the new landmark, at least the set of features forming the new landmark, and the 3D structure of the set of features forming the new landmark; and

notify the SLAM application of the creation of a new landmark.

15. The mobile robot of claim 14, wherein the SLAM application directs a processor to:

determine a landmark pose as the pose of the mobile robot at the time the image of the new landmark stored in the landmark database is captured; and

recording the landmark pose for the new landmark in the landmark database.

16. The mobile robot of claim 1, wherein the machine vision sensor system further comprises a plurality of cameras under the top surface of the robot and having focal axes angled upward relative to the horizontal plane of the top surface and wherein at least one camera faces a reverse drive direction opposite to the forward drive direction.

17. The mobile robot of claim 1, wherein the machine vision system comprises a stereo pair of cameras having overlapping fields of view.

18. The mobile robot of claim 1, wherein the camera is positioned within the recessed structure at a distance of at most 6 inches from a floor surface.

19. A mobile robot configured to navigate an operating environment, comprising:

a body that comprises a top surface located at a height not more than 6 inches from a bottom surface, the body comprising a recessed structure under the top surface, the body containing;

at least one processor;

memory containing a behavioral controller application, where the behavioral controller application directs a processor to navigate an environment based on the captured images;

a machine vision sensor system comprising a camera configured to capture images of the operating environment of the mobile robot, the camera including optics defining a camera field of view and a camera optical axis, where the camera is positioned within the recessed structure and is tilted so that the optical axis is aligned at an acute angle between 10 and 60 degrees above a forward drive direction of motion of the mobile robot.

20. The mobile robot of claim 19, the body further containing:

a lens cover protecting the camera, wherein the lens cover is set back relative to an opening of the recessed structure at an acute angle with respect to the optical axis of the camera that is closer to perpendicular than an angle formed between a plane defined by the opening in the recessed structure and the optical axis of the camera.