Systems and methods for performing occlusion detection

• US 10,611,023 B2
• Filed: 07/15/2019
• Issued: 04/07/2020
• Est. Priority Date: 11/26/2014
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Fong, Philip, Meltzer, Jason, Rius, Marc Barnada
• Primary Examiner(s): Mackey, Patrick H
• Application Number:
  US16/511,119
• Publication Number:
  US 20190337151A1
• Time in Patent Office: 267 Days
• CPC Class Codes: B25J 9/1602 characterised by the control system, structure, architecture
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: G05B 2219/40572 Camera combined with position sensor
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

The present invention provides a mobile robot configured to navigate an operating environment, that includes a machine vision system comprising a camera that captures images of the operating environment using a machine vision system; detects the presence of an occlusion obstructing a portion of the field of view of a camera based on the captured images, and generate a notification when an occlusion obstructing the portion of the field of view of the camera is detected, and maintain occlusion detection data describing occluded and unobstructed portions of images being used by the SLAM application.

Claims

1. A mobile robot configured to navigate an operating environment, comprising:

a machine vision sensor system comprising a camera;

at least one processor; and

a memory comprising computer readable program code stored in a non-transitory medium, the computer readable program code comprising a simultaneous localization and mapping application and a behavioral control application, wherein, when executed by the at least one processor, the computer readable program code causes the at least one processor to;

capture images of the operating environment of the mobile robot using the machine vision sensor system;

detect a presence of an occlusion obstructing a portion of a field of view of the camera based on the images captured using the machine vision sensor system; and

generate a notification of the detected occlusion for a user.

2. The mobile robot of claim 1, wherein the notification includes an electronic message.

3. The mobile robot of claim 1, wherein the notification includes an alert indicator.

4. The mobile robot of claim 1, further comprising a communication circuit configured to transmit the generated notification to an external device via a communication link, the external device including a server or a user device.

5. The mobile robot of claim 4, wherein the communication link includes a wireless communication link.

6. The mobile robot of claim 4, wherein, responsive to an execution of the behavioral control application by the at least one processor, the communication circuit is configured to transmit the generated notification on a periodic basis or at a schedule time.

7. The mobile robot of claim 1, wherein the memory further comprises a pre-loaded map of the operating environment stored in the non-transitory medium, and wherein, when executed by the at least one processor, the computer readable program code causes the at least one processor to navigate the mobile robot in the operating environment based on the pre-loaded map.

8. The mobile robot of claim 7, wherein the computer readable program code, when executed by the at least one processor, further directs the at least one processor to update the pre-loaded map based at least on the captured images of the operating environment.

9. The mobile robot of claim 1, further comprising an odometry sensor system, wherein, when executed by the at least one processor, the computer readable program code causes the at least one processor to:

capture odometry data using the odometry sensor system; and

estimate a location of the mobile robot within the operating environment using the captured odometry data.

10. The mobile robot of claim 9, wherein the odometry sensor system comprises an optical dead reckoning sensor aimed at a floor of the operating environment, and wherein, when executed by the at least one processor, the computer readable program code causes the at least one processor to estimate a location of the mobile robot within the operating environment further using data collected by the optical dead reckoning sensor.

11. The mobile robot of claim 1, wherein the computer readable program code, when executed by the at least one processor, further directs the at least one processor to identify a percentage of the field of view of the camera that is occluded prior to the generation of the notification of the detected occlusion.

12. The mobile robot of claim 1, wherein the computer readable program code, when executed by the at least one processor, further directs the at least one processor to identify a landmark using the captured images of the operating environment, and to navigate the mobile robot in the operating environment using the identified landmark.

21. The mobile robot of claim 1, wherein the occlusion includes a semi-transparent occlusion, and the execution of the computer readable program code causes the at least one processor to:

detect an image feature from each of the captured images; and

detect a presence of the semi-transparent occlusion based on a correspondence of the detected image feature among the captured images.

22. The mobile robot of claim 1, wherein the occlusion includes an opaque occlusion, and the execution of the computer readable program code causes the at least one processor to:

detect an image feature from each of the captured images; and

detect a presence of the opaque occlusion based on disparity in dimension or geometry of the detected image feature among the captured images.

23. The mobile robot of claim 1, wherein the occlusion includes an opaque occlusion, and the execution of the computer readable program code causes the at least one processor to:

detect an image feature from each of a plurality of images captured at different rotations or translations of the mobile robot; and

detect a presence of the opaque occlusion based on consistency of spatial location of the detected image feature on the plurality of images.

24. The mobile robot of claim 12, wherein the execution of the computer readable program code causes the at least one processor to:

identify, from a specified number of captured images, one or more occluded images each containing one or more image portions not used by the at least one processor to identify a landmark, the one or more image portions corresponding to respective one or more portions of field of view of the camera; and

detect the presence of an occlusion based on the identification of one or more occluded images.

25. The mobile robot of claim 24, wherein the execution of the computer readable program code causes the at least one processor to:

generate a histogram of the one or more image portions using the captured images; and

detect the presence of an occlusion based on the generated histogram of the one or more image portions.

13. A mobile robot configured to navigate an operating environment, comprising:

a body containing;

a drive mechanism;

an odometry sensor system configured to capture odometry data;

a machine vision sensor system comprising a camera configured to capture images of the operating environment;

at least one processor; and

a memory comprising computer readable program code stored in a non-transitory medium, the computer readable program code comprising a visual simultaneous localization and mapping application and a behavioral control application, wherein, when executed by the at least one processor, the computer readable program code causes the at least one processor to;

detect a presence of an occlusion obstructing a portion of a field of view of the camera based on the images captured by the machine vision sensor system;

generate and a notification of the detected occlusion and deliver the notification to an external device; and

localize the mobile robot within the operating environment using the odometry data captured by the odometry sensor system.

14. A method of operating a mobile robot, the method comprising:

executing, by at least one processor of the mobile robot, computer readable program code stored in a non-transitory medium, the computer readable program code comprising a simultaneous localization and mapping application and a behavioral control application, wherein the executing of the computer readable program code causes the at least one processor to perform operations comprising;

capturing images of an operating environment of the mobile robot using a machine vision sensor system comprising a camera of the mobile robot;

detecting a presence of an occlusion obstructing a portion of a field of view of the camera based on the images captured using the machine vision sensor system; and

generating a notification of the detected occlusion for a user.

15. The method of claim 14, wherein the notification of the detected occlusion includes an electronic message or an alert indicator.

16. The method of claim 14, wherein the operations further comprise transmitting the notification to an external device via a communication link.

17. The method of claim 16, wherein transmitting the notification to the external device is on a periodic basis or at a schedule time.

18. The method of claim 14, wherein the operations further comprise:

capturing odometry data using an odometry sensor system; and

estimating a location of the mobile robot within the operating environment based on the captured odometry data responsive to insufficient feature detection using the machine vision sensor system.

19. The method of claim 14, wherein the operations further comprise navigating the mobile robot in the operating environment based on a pre-loaded map of the operating environment that is stored the non-transitory medium.

20. The method of claim 19, wherein the operations further comprise:

identifying a landmark using the captured images of the operating environment; and

navigating the mobile robot in the operating environment using the identified landmark.