Autonomous coverage robot

• US 8,972,061 B2
• Filed: 03/08/2013
• Issued: 03/03/2015
• Est. Priority Date: 11/02/2012
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Rosenstein, Michael T., Chiappetta, Mark, Schnittman, Mark, Pastore, Andrew
• Primary Examiner(s): HOLLOWAY, JASON R
• Application Number:
  US13/790,867
• Publication Number:
  US 20140124004A1
• Time in Patent Office: 725 Days
• Field of Search: 700/250, 700/253, 700/255, 700/258, 700/259, 901/1, 901/46, 901/47
• US Class Current: 700/259
• CPC Class Codes: A47L 11/4011 Regulation of the cleaning machine by electric means; Control systems and remote control systems therefor
• CPC Class Codes: A47L 11/4061 Steering means; Means for avoiding obstacles; Details related to the place where the driver is accommodated
• CPC Class Codes: A47L 2201/06 Control of the cleaning action for autonomous devices; Automatic detection of the surface condition before, during or after cleaning
• CPC Class Codes: A47L 9/2826 the condition of the floor
• CPC Class Codes: A47L 9/2852 Elements for displacement of the vacuum cleaner or the accessories therefor, e.g. wheels, casters or nozzles
• CPC Class Codes: G05D 1/0219 ensuring the processing of the whole working surface
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: Y10S 901/01 Mobile robot

Abstract

A mobile floor cleaning robot includes a robot body supported by a drive system configured to maneuver the robot over a floor surface. The robot also includes a cleaning system supported by the robot body, an imaging sensor disposed on the robot body, and a controller in communicates with the drive system and the imaging sensor. The controller receives a sequence of images of the floor surface; each image has an array of pixels. For each image, the controller segments the image into color blobs by color quantizing pixels of the image, determines a spatial distribution of each color of the image based on corresponding pixel locations; and for each image color, identifies areas of the image having a threshold spatial distribution for that color. The controller then tracks a location of the color blobs with respect to the imaging sensor across the sequence of images.

Claims

1. A method of operating mobile floor cleaning robot, the method comprising:

receiving, at a controller, a sequence of images of a floor surface from an imaging sensor in communication with the controller, each image having an array of pixels;

for each image, segmenting the image into color blobs, using the controller, by;

color quantizing pixels of the image;

determining a spatial distribution of each color of the image based on corresponding pixel locations; and

for each image color, identifying areas of the image having a threshold spatial distribution for that color;

tracking, using the controller, a location of each color blob with respect to the imaging sensor across the sequence of images;

identifying, using the controller, a location of an object on a floor surface away from the robot;

issuing a first drive command from the controller to a drive system of the robot to drive the robot across the floor surface to clean the floor surface at the identified location of the object;

determining, using the controller, whether the object persists on the floor surface; and

when the object persists, issuing a second drive command from the controller to the drive system to drive the robot across the floor surface to re-clean the floor surface at the identified location of the object.

2. The method of claim 1, further comprising, after cleaning the floor surface at the identified object location, issuing a third drive command from the controller to the drive system to maneuver the robot to determine whether the object persists on the floor surface.

3. The method of claim 1, further comprising identifying the location of the object based on the tracked locations of the color blobs.

4. The method of claim 1, wherein segmenting the image into color blobs further comprises only color quantizing pixels in a lower portion of the image oriented vertically.

5. The method of claim 1, wherein segmenting the image into color blobs further comprises only color quantizing pixels outside of a center portion of the image.

6. The method of claim 1, wherein segmenting the image into color blobs further comprises dividing the image into regions and separately color quantizing the pixels of each region.

7. The method of claim 1, wherein segmenting the image into color blobs further comprises executing a bit shifting operation to convert each pixel from a first color set to second color set smaller than the first color set.

8. The method of claim 7, wherein the bit shifting operation retains the three most significant bits of each of a red, green and blue channel.

9. The method of claim 1, wherein tracking a location of each color blob comprises:

determining a velocity vector of each color blob with respect to the imaging; and

recording determined color blob locations for each image of the image sequence.

10. The method of claim 1, further comprising determining a size of each color blob.

11. The method of claim 1, further comprising issuing a third drive command from the controller to the drive system to maneuver the robot based on the location of one or more color blobs.

12. The method of claim 11, further comprising issuing a fourth drive command from the controller to the drive system to maneuver the robot toward a nearest color blob.

13. The method of claim 12, wherein the nearest color blob is identified in a threshold number of images of the image sequence.

14. The method of claim 1, further comprising:

determining, using the controller, a size of each color blob;

determining, using the controller, a velocity vector of each color blob with respect to the imaging sensor; and

issuing a third drive command from the controller to the drive system to maneuver the robot based on the size and the velocity vector of one or more color blobs.

15. The method of claim 14, further comprising issuing a fourth drive command from the controller to the drive system to maneuver the robot toward a color blob having the largest size and velocity vector toward the robot.

16. The method of claim 14, further comprising executing a heuristic related to color blob size and color blob speed to filter out color blobs non-indicative of debris on the floor surface.

17. The method of claim 1, further comprising assigning a numerical representation for the color of each pixel in a color space.

18. The method of claim 17, further comprising color quantizing the image pixels in a red-green-blue color space, reducing the image to a 9-bit red-green-blue image.

19. The method of claim 1, further comprising executing a control system having a control arbitration system and a behavior system in communication with each other, the behavior system executing a cleaning behavior, the cleaning behavior influencing execution of commands by the control arbitration system based on the image segmentation to identify color blobs corresponding to a dirty floor area and color blob tracking to maneuver over the dirty floor area for cleaning using a cleaning system of the robot.

20. A mobile floor cleaning robot comprising:

a robot body defining a forward drive direction;

a drive system supporting the robot body and configured to maneuver the robot over a floor surface;

a cleaning system supported by the robot body;

an imaging sensor disposed on the robot body; and

a controller in communication with the drive system and the imaging sensor, the controller;

receiving a sequence of images of the floor surface, each image having an array of pixels;

for each image, segmenting the image into color blobs by;

color quantizing pixels of the image;

determining a spatial distribution of each color of the image based on corresponding pixel locations; and

for each image color, identifying areas of the image having a threshold spatial distribution for that color; and

tracking a location of each color blob with respect to the imaging sensor across the sequence of images.

21. The mobile floor cleaning robot of claim 20, wherein segmenting the image into color blobs further comprises only color quantizing pixels in a lower portion of the image oriented vertically.

22. The mobile floor cleaning robot of claim 20, wherein segmenting the image into color blobs further comprises only color quantizing pixels outside of a center portion of the image.

23. The mobile floor cleaning robot of claim 20, wherein segmenting the image into color blobs further comprises dividing the image into regions and separately color quantizing the pixels of each region.

24. The mobile floor cleaning robot of claim 20, wherein segmenting the image into color blobs further comprises executing a bit shifting operation to convert each pixel from a first color set to second color set smaller than the first color set.

25. The mobile floor cleaning robot of claim 24, wherein the bit shifting operation retains the three most significant bits of each of a red, green and blue channel.

26. The mobile floor cleaning robot of claim 20, wherein the image sensor comprises a camera arranged to have a field of view along a forward drive direction of the robot.

27. The mobile floor cleaning robot of claim 26, wherein the camera scans side-to-side or up-and-down with respect to the forward drive direction of the robot.

28. The mobile floor cleaning robot of claim 20, wherein tracking a location of the color blobs comprises:

determining a velocity vector of each color blob with respect to the imaging; and

recording determined color blob locations for each image of the image sequence.

29. The mobile floor cleaning robot of claim 20, wherein the controller determines a size of each color blob.

30. The mobile floor cleaning robot of claim 20, wherein the controller issues a drive command to maneuver the robot based on the location of one or more color blobs.

31. The mobile floor cleaning robot of claim 30, wherein the controller issues a drive command to maneuver the robot toward a nearest color blob.

32. The mobile floor cleaning robot of claim 31, wherein the controller identifies the nearest color blob in a threshold number of images of the image sequence.

33. The mobile floor cleaning robot of claim 20, wherein the controller:

determines a size of each color blob;

determines a velocity vector of each color blob with respect to the imaging sensor; and

issues a drive command to maneuver the robot based on the size and the velocity vector of one or more color blobs.

34. The mobile floor cleaning robot of claim 33, wherein the controller issues a drive command to maneuver the robot toward a color blob having the largest size and velocity vector toward the robot.

35. The mobile floor cleaning robot of claim 33, wherein the controller executes a heuristic related to color blob size and color blob speed to filter out color blobs non-indicative of debris on the floor surface.

36. The mobile floor cleaning robot of claim 20, wherein the controller assigns a numerical representation for the color of each pixel in a color space.

37. The mobile floor cleaning robot of claim 36, wherein the controller color quantizes the image pixels in a red-green-blue color space, reducing the image to a 9-bit red-green-blue image.

38. The mobile floor cleaning robot of claim 36, wherein the controller color quantizes the image pixels in a LAB color space.

39. A mobile floor cleaning robot comprising:

a robot body defining a forward drive direction;

a drive system supporting the robot body and configured to maneuver the robot over a floor surface;

a controller in communication with the drive system and executing a control system;

a cleaning system supported by the robot body and in communication with the controller; and

an imaging sensor in communication with the controller;

wherein the control system comprises a control arbitration system and a behavior system in communication with each other, the behavior system executing a cleaning behavior influencing execution of commands by the control arbitration system based on a sequence of images of the floor surface received from the imaging sensor to identify a dirty floor area and maneuver the cleaning system over the dirty floor area, the cleaning behavior identifying the dirty floor area by;

for each image, segmenting the image into color blobs by;

color quantizing pixels of the image;

determining a spatial distribution of each color of the image based on corresponding pixel locations; and

for each image color, identifying areas of the image having a threshold spatial distribution for that color; and

tracking a location of each color blob with respect to the imaging sensor across the sequence of images.

40. A method of operating a mobile cleaning robot having an imaging sensor, the method comprising:

receiving, at a controller, a sequence of images of a floor surface supporting the robot from an imaging sensor in communication with the controller, each image having an array of pixels;

for each image, segmenting the image into color blobs, using the controller, by;

color quantizing pixels of the image;

determining a spatial distribution of each color of the image based on corresponding pixel locations; and

for each image color, identifying areas of the image having a threshold spatial distribution for that color; and

tracking, using the controller, a location of each color blob with respect to the imaging sensor across the sequence of images; and

issuing a move command from the controller to a drive system of the robot to move the robot with resect to at least one color blob.

41. The method of claim 40, wherein segmenting the image into color blobs further comprises only color quantizing pixels in a lower portion of the image oriented vertically.

42. The method of claim 40, wherein segmenting the image into color blobs further comprises only color quantizing pixels outside of a center portion of the image.

43. The method of claim 40, wherein segmenting the image into color blobs further comprises dividing the image into regions and separately color quantizing the pixels of each region.

44. The method of claim 40, wherein segmenting the image into color blobs further comprises executing a bit shifting operation to convert each pixel from a first color set to second color set smaller than the first color set.

45. The method of claim 44, wherein the bit shifting operation retains the three most significant bits of each of a red, green and blue channel.

46. The method of claim 40, wherein the image sensor comprises a camera arranged to have a field of view along a forward drive direction of the robot.

47. The method of claim 46, further comprising scanning the camera imaging sensor side-to-side or up-and-down with respect to the forward drive direction of the robot.

48. The method of claim 40, wherein tracking a location of the color blobs comprises:

determining a velocity vector of each color blob with respect to the imaging; and

recording determined color blob locations for each image of the image sequence.

49. The method of claim 40, further comprising determining a size of each color blob.

50. The method of claim 40, further comprising issuing a drive command from the controller to the drive system to maneuver the robot based on the location of one or more color blobs.

51. The method of claim 40, further comprising issuing a drive command from the controller to the drive system to maneuver the robot toward a nearest color blob.

52. The method of claim 51, wherein the nearest color blob is identified in a threshold number of images of the image sequence.

53. The method of claim 40, further comprising:

determining, using the controller, a size of each color blob;

determining, using the controller, a velocity vector of each color blob with respect to the imaging sensor; and

issuing a drive command from the controller to the drive system to maneuver the robot based on the size and the velocity vector of one or more color blobs.

54. The method of claim 53, wherein the drive command to maneuver comprises maneuvering the robot toward a color blob having the largest size and velocity vector toward the robot.

55. The method of claim 53, further comprising executing a heuristic related to color blob size and color blob speed to filter out color blobs non-indicative of debris on the floor surface.

56. The method of claim 40, further comprising assigning a numerical representation for the color of each pixel in a color space.

57. The method of claim 56, further comprising color quantizing the image pixels in a red-green-blue color space, reducing the image to a 9-bit red-green-blue image.

58. The method of claim 56, further comprising color quantizing the image pixels in a LAB color space.

59. The method of claim 40, further comprising executing a control system having a control arbitration system and a behavior system in communication with each other, the behavior system executing a cleaning behavior, the cleaning behavior influencing execution of commands by the control arbitration system based on the image segmentation to identify color blobs corresponding to a dirty floor area and color blob tracking to maneuver over the dirty floor area for cleaning using a cleaning system of the robot.