Systems and methods for performing occlusion detection

• US 10,391,630 B2
• Filed: 07/11/2017
• Issued: 08/27/2019
• Est. Priority Date: 11/26/2014
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Fong, Philip, Meltzer, Jason, Barnada Rius, Marc
• Primary Examiner(s): Mackey, Patrick H
• Application Number:
  US15/646,782
• Publication Number:
  US 20170305010A1
• Time in Patent Office: 777 Days
• CPC Class Codes: B25J 9/1602 characterised by the control system, structure, architecture
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/40572 Camera combined with position sensor
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

The present invention provides a mobile robot configured to navigate an operating environment, that includes a machine vision system comprising a camera that captures images of the operating environment using a machine vision system; detects the presence of an occlusion obstructing a portion of the field of view of a camera based on the captured images, and generate a notification when an occlusion obstructing the portion of the field of view of the camera is detected, and maintain occlusion detection data describing occluded and unobstructed portions of images being used by the SLAM application.

Claims

1. A mobile robot configured to navigate an operating environment, comprising:

a machine vision sensor system comprising a camera;

an indicator light;

at least one processor; and

a memory comprising computer readable program code stored in a non-transitory medium, the computer readable program code comprising a simultaneous localization and mapping (SLAM) application and a behavioral control application, wherein, when executed by the at least one processor, the computer readable program code causes the at least one processor to;

capture images of the operating environment of the mobile robot using the machine vision sensor system;

detect a presence of an occlusion obstructing a portion of a field of view of the camera based on the images captured using the machine vision sensor system; and

illuminate the indicator light responsive to detection of the presence of the occlusion obstructing the portion of the field of view of the camera.

2. The mobile robot of claim 1, wherein the memory further comprises a pre-loaded map of the operating environment stored in the non-transitory medium, and wherein, when executed by the at least one processor, the computer readable program code causes the at least one processor to:

navigate the mobile robot in the operating environment based on the pre-loaded map.

3. The mobile robot of claim 2, wherein the pre-loaded map comprises boundaries of the operating environment, the boundaries comprising at least one of stairs, walls, or doors.

4. The mobile robot of claim 1, further comprising:

an odometry sensor system,

wherein, when executed by the at least one processor, the computer readable program code causes the at least one processor to;

capture odometry data using the odometry sensor system; and

estimate a location of the mobile robot within the operating environment based on the odometry data captured using the odometry sensor system responsive to insufficient feature detection using the machine vision sensor system.

5. The mobile robot of claim 4, wherein the odometry sensor system comprises a wheel odometer configured to capture the odometry data based on rotation of wheels of a drive mechanism and/or an optical flow odometry system configured to obtain the odometry data by capturing images of a tracking surface and determining a distance traveled thereon.

6. The mobile robot of claim 1, wherein the camera comprises an optical axis that is tilted at an angle relative to a top surface of the mobile robot.

7. The mobile robot of claim 6, wherein the camera is housed within a recess in a body of the mobile robot, the body comprising a lens cover aligned at an acute angle with respect to the optical axis of the camera.

8. The mobile robot of claim 1, further comprising:

a cleaning mechanism for the camera,

wherein, when executed by the at least one processor, the computer readable program code causes the at least one processor to operate the cleaning mechanism to attempt to eliminate the occlusion responsive to detection of the presence thereof.

9. The mobile robot of claim 8, wherein the cleaning mechanism comprises at least one of a wiper blade, a fan, or an air knife.

10. The mobile robot of claim 1, wherein, to detect the presence of the occlusion, the computer readable program code, when executed by the at least one processor, causes the at least one processor to:

identify the portion of the field of view of the camera that is not utilized by the SLAM application for a threshold number of images.

11. The mobile robot of claim 10, wherein, to identify the portion of the field of view of the camera, the computer readable program code, when executed by the at least one processor, causes the at least one processor to:

determine respective portions of the field of view of the camera in which features are identified;

update occlusion detection data describing portions of the images that correspond to the respective portions of the field of view of the camera; and

identify the portion of the field of view of the camera that is not utilized by the SLAM application for the threshold number of images based on the occlusion detection data.

12. The mobile robot of claim 1, wherein the computer readable program code, when executed by the at least one processor, further directs the at least one processor to identify a percentage of the field of view of the camera that is occluded prior to illumination of the indicator light.

13. A method of operating a mobile robot, the method comprising:

executing, by at least one processor of the mobile robot, computer readable program code stored in a non-transitory medium, the computer readable program code comprising a simultaneous localization and mapping (SLAM) application and a behavioral control application, wherein the executing of the computer readable program code causes the at least one processor to perform operations comprising;

capturing images of an operating environment of the mobile robot using a machine vision sensor system comprising a camera of the mobile robot;

detecting a presence of an occlusion obstructing a portion of a field of view of the camera based on the images captured using the machine vision sensor system; and

illuminating an indicator light of the mobile robot responsive to detection of the presence of the occlusion obstructing the portion of the field of view of the camera.

14. The method of claim 13, wherein the operations further comprise:

navigating the mobile robot in the operating environment based on a pre-loaded map of the operating environment that is stored in the non-transitory medium.

15. The method of claim 14, wherein the pre-loaded map comprises boundaries of the operating environment, the boundaries comprising at least one of stairs, walls, or doors.

16. The method of claim 13, wherein the operations further comprise:

capturing odometry data using an odometry sensor system; and

estimating a location of the mobile robot within the operating environment based on the odometry data captured using the odometry sensor system responsive to insufficient feature detection using the machine vision sensor system.

17. The method of claim 13, wherein the camera of the mobile robot comprises an optical axis that is tilted at an angle relative to a top surface of the mobile robot.

18. The method of claim 13, wherein the operations further comprise:

operating a cleaning mechanism comprising at least one of a wiper blade, a fan, or an air knife to attempt to eliminate the occlusion responsive to detecting the presence thereof.

19. The method of claim 13, wherein detecting the presence of the occlusion comprises:

identifying the portion of the field of view of the camera that is not utilized by the SLAM application for a threshold number of images.

20. A mobile robot configured to navigate an operating environment, comprising:

a body containing;

a drive mechanism;

an indicator light;

an odometry sensor system configured to capture odometry data;

a machine vision sensor system comprising a camera configured to capture images of the operating environment;

at least one processor; and

a memory comprising computer readable program code stored in a non-transitory medium, the computer readable program code comprising a visual simultaneous localization and mapping (VSLAM) application and a behavioral control application,

wherein, when executed by the at least one processor, the computer readable program code causes the at least one processor to;

detect a presence of an occlusion obstructing a portion of a field of view of the camera based on the images captured by the machine vision sensor system;

flash the indicator light responsive to detection of the presence of the occlusion obstructing the portion of the field of view of the camera; and

localize the mobile robot within the operating environment based on the odometry data captured by the odometry sensor system responsive to insufficient feature detection by the machine vision sensor system.