Autonomous Coverage Robot

• US 20140124004A1
• Filed: 03/08/2013
• Published: 05/08/2014
• Est. Priority Date: 11/02/2012
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Rosenstein, Michael T., Chiappetta, Mark, Pastore, Andrew, Schnittman, Mark
• Granted Patent: US 8,972,061 B2
• Time in Patent Office: Days
• US Class Current: 134/18
• CPC Class Codes: A47L 11/4011 Regulation of the cleaning machine by electric means; Control systems and remote control systems therefor
• CPC Class Codes: A47L 11/4061 Steering means; Means for avoiding obstacles; Details related to the place where the driver is accommodated
• CPC Class Codes: A47L 2201/06 Control of the cleaning action for autonomous devices; Automatic detection of the surface condition before, during or after cleaning
• CPC Class Codes: A47L 9/2826 the condition of the floor
• CPC Class Codes: A47L 9/2852 Elements for displacement of the vacuum cleaner or the accessories therefor, e.g. wheels, casters or nozzles
• CPC Class Codes: G05D 1/0219 ensuring the processing of the whole working surface
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30261 Obstacle
• CPC Class Codes: G06T 7/00 Image analysis
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06T 7/62 of area, perimeter, diameter or volume
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: Y10S 901/01 Mobile robot

Abstract

A mobile floor cleaning robot includes a robot body supported by a drive system configured to maneuver the robot over a floor surface. The robot also includes a cleaning system supported by the robot body, an imaging sensor disposed on the robot body, and a controller in communicates with the drive system and the imaging sensor. The controller receives a sequence of images of the floor surface; each image has an array of pixels. For each image, the controller segments the image into color blobs by color quantizing pixels of the image, determines a spatial distribution of each color of the image based on corresponding pixel locations; and for each image color, identifies areas of the image having a threshold spatial distribution for that color. The controller then tracks a location of the color blobs with respect to the imaging sensor across the sequence of images.

Claims

1. A method of operating mobile floor cleaning robot, the method comprising:

identifying a location of an object on a floor surface away from the robot;

driving across the floor surface to clean the floor surface at the identified location of the object;

determining whether the object persists on the floor surface; and

when the object persists, driving across the floor surface to re-clean the floor surface at the identified location of the object.

2. The method of claim 1, further comprising, after cleaning the floor surface at the identified object location, maneuvering to determine whether the object persists on the floor surface.

3. The method of claim 1, further comprising:

receiving a sequence of images of the floor surface, each image having an array of pixels;

for each image, segmenting the image into color blobs by;

color quantizing pixels of the image;

determining a spatial distribution of each color of the image based on corresponding pixel locations; and

for each image color, identifying areas of the image having a threshold spatial distribution for that color; and

tracking a location of each color blob with respect to the imaging sensor across the sequence of images.

4. The method of claim 3, further comprising identifying the location of the object based on the tracked locations of the color blobs.

5. The method of claim 3, wherein segmenting the image into color blobs further comprises only color quantizing pixels in a lower portion of the image oriented vertically.

6. The method of claim 3, wherein segmenting the image into color blobs further comprises only color quantizing pixels outside of a center portion of the image.

7. The method of claim 3, wherein segmenting the image into color blobs further comprises dividing the image into regions and separately color quantizing the pixels of each region.

8. The method of claim 3, wherein segmenting the image into color blobs further comprises executing a bit shilling operation to convert each pixel from a first color set to second color set smaller than the first color set.

9. The method of claim 8, wherein the hit shifting operation retains the three most significant bits of each of a red, green and blue channel.

10. The method of claim 3, wherein tracking a location of each color blob comprises:

determining a velocity vector of each color blob with respect to the imaging; and

recording determined color blob locations for each image of the image sequence.

11. The method of claim 3, further comprising determining a size of each color blob.

12. The method of claim 3, further comprising issuing a drive command to maneuver the robot based on the location of one or more color blobs.

13. The method of claim 12, further comprising issuing a drive command to maneuver the robot toward a nearest color blob.

14. The method of claim 13, wherein the nearest color blob is identified in a threshold number of images of the image sequence.

15. The method of claim 3, further comprising:

determining a size of each color blob;

determining a velocity vector of each color blob with respect to the imaging sensor; and

issuing a drive command to maneuver the robot based on the size and the velocity vector of one or more color blobs.

16. The method of claim 15, further comprising issuing a drive command to maneuver the robot toward a color blob having the largest size and velocity vector toward the robot.

17. The method of claim 15, further comprising executing a heuristic related to color blob size and color blob speed to filter out color blobs non-indicative of debris on the floor surface.

18. The method of claim 3, further comprising assigning a numerical representation for the color of each pixel in a color space.

19. The method of claim 18, further comprising color quantizing the image pixels in a red-green-blue color space, reducing the image to a 9-bit red-green-blue image.

20. The method of claim 3, further comprising executing a control system having a control arbitration system and a behavior system in communication with each other, the behavior system executing a cleaning behavior, the cleaning behavior influencing execution of commands by the control arbitration system based on the image segmentation to identify color blobs corresponding to a dirty floor area and color blob tracking to maneuver over the dirty floor area for cleaning using a cleaning system of the robot.

29. The mobile floor cleaning robot of claim 1, wherein tracking a location of the color blobs comprises:

determining a velocity vector of each color blob with respect to the imaging; and

recording determined color blob locations for each image of the image sequence.

21. A mobile floor cleaning robot comprising:

a robot body defining a forward drive direction;

a drive system supporting the robot body and configured to maneuver the robot over a floor surface;

a cleaning system supported by the robot body;

an imaging sensor disposed on the robot body; and

a controller in communication with the drive system and the imaging sensor, the controller;

receiving a sequence of images of to floor surface, each image having an array of pixels;

for each image, segmenting the image into color blobs by;

color quantizing pixels of the image;

determining a spatial distribution of each color of the image based on corresponding pixel locations; and

for each image color, identifying areas of the image having a threshold spatial distribution for that color; and

tracking a location of each color blob with respect to the imaging sensor across the sequence of images.

22. The mobile floor cleaning robot of claim 21, wherein segmenting the image into color blobs further comprises only color quantizing pixels in a lower portion of the image oriented vertically.

23. The mobile floor cleaning robot of claim 21, wherein segmenting the image into color blobs further comprises only color quantizing pixels outside of a center portion of the image.

24. The mobile floor cleaning robot of claim 21, wherein segmenting the image into color blobs further comprises dividing the image into regions and separately color quantizing the pixels of each region.

25. The mobile floor cleaning robot of claim 21, wherein segmenting the image into color blobs further comprises executing a bit shifting operation to convert each pixel from a first color set to second color set smaller than the first color set.

26. The mobile floor cleaning robot of claim 25, wherein the bit shifting operation retains the three most significant bits of each of a red, green and blue channel.

27. The mobile floor cleaning robot of claim 21, wherein the image sensor comprises a camera arranged to have a field of view along a forward drive direction of the robot.

28. The mobile floor cleaning robot of claim 27, wherein the camera scans side-to-side or up-and-down with respect to the forward drive direction of the robot.

30. The mobile floor cleaning robot of claim 21, wherein the controller determines a size of each color blob.

31. The mobile floor cleaning robot of claim 21, wherein the controller issues a drive command to maneuver the robot based on the location of one or more color blobs.

32. The mobile floor cleaning robot of claim 31, wherein the controller issues a drive command to maneuver the robot toward a nearest color blob.

33. The mobile floor cleaning robot of claim 32, wherein the controller identifies the nearest color blob in a threshold number of images of the image sequence.

34. The mobile floor cleaning robot of claim 21, wherein the controller:

determines a size of each color blob;

determines a velocity vector of each color blob with respect to the imaging sensor; and

issues a drive command to maneuver the robot based on the size and the velocity vector of one or more color blobs.

35. The mobile floor cleaning robot of claim 34, wherein the controller issues a drive command to maneuver the robot toward a color blob having the largest size and velocity vector toward the robot.

36. The mobile floor cleaning robot of claim 34, wherein the controller executes a heuristic related to color blob size and color blob speed to filter out color blobs non-indicative of debris on the floor surface.

37. The mobile floor cleaning robot of claim 21, wherein the controller assigns a numerical representation for the color of each pixel in a color space.

38. The mobile floor cleaning robot of claim 37, wherein the controller color quantizes the image pixels in a red-green-blue color space, reducing the image to a 9-bit red-green-blue image.

39. The mobile floor cleaning robot of claim 37, wherein the controller color quantizes the image pixels in a LAB color space.

40. A mobile floor cleaning robot comprising:

a robot body defining a forward drive direction;

a drive system supporting the robot body and configured to maneuver the robot over a floor surface;

a controller in communication with the drive system and executing a control system;

a cleaning system supported by the robot body and in communication with the controller; and

an imaging sensor in communication with the controller;

wherein the control system comprises a control arbitration system and a behavior system in communication with each other, the behavior system executing a cleaning behavior influencing execution of commands by the control arbitration system based on a sequence of images of the floor surface received from the imaging sensor to identify a dirty floor area and maneuver the cleaning system over the dirty floor area, the cleaning behavior identifying the dirty floor area by;

for each image, segmenting the image into color blobs by;

color quantizing pixels of the image;

determining a spatial distribution of each color of the image based on corresponding pixel locations; and

for each image color, identifying areas of the image having a threshold spatial distribution for that color; and

tracking a location of each color blob with respect to the imaging sensor across the sequence of images.

41. A method of operating a mobile cleaning robot having an imaging sensor, the method comprising:

receiving a sequence of images of a floor surface supporting the robot, each image having an array of pixels;

for each image, segmenting the image into color blobs by;

color quantizing pixels of the image;

determining a spatial distribution of each color of the image based on corresponding pixel locations; and

for each image color, identifying areas of the image having a threshold spatial distribution for that color; and

tracking a location of each color blob with respect to the imaging sensor across the sequence of images.

42. The method of claim 41, wherein segmenting the image into color blobs further comprises only color quantizing pixels in a lower portion of the image oriented vertically.

43. The method of claim 41, wherein segmenting the image into color blobs further comprises only color quantizing pixels outside of a center portion of the image.

44. The method of claim 41, wherein segmenting the image into color blobs further comprises dividing the image into regions and separately color quantizing the pixels of each region.

45. The method of claim 41, wherein segmenting the image into color blobs further comprises executing a bit shilling operation to convert each pixel from a first color set to second color set smaller than the first color set.

46. The method of claim 45, wherein the bit shifting operation retains the three most significant bits of each of a red, green and blue channel.

47. The method of claim 41, wherein the image sensor comprises a camera arranged to have a field of view along a forward drive direction of the robot.

48. The method of claim 47, further comprising scanning the camera side-to-side or up-and-down with respect to the forward drive direction of the robot.

49. The method of claim 41, wherein tracking a location of the color blobs comprises:

determining a velocity vector of each color blob with respect to the imaging; and

recording determined color blob locations for each image of the image sequence.

50. The method of claim 41, further comprising determining a size of each color blob.

51. The method of claim 41, further comprising issuing a drive command to maneuver the robot based on the location of one or more color blobs.

52. The method of claim 51, further comprising issuing a drive command to maneuver the robot toward a nearest color blob.

53. The method of claim 52, wherein the nearest color blob is identified in a threshold number of images of the image sequence.

54. The method of claim 41, further comprising:

determining a size of each color blob;

determining a velocity vector of each color blob with respect to the imaging sensor; and

issuing a drive command to maneuver the robot based on the size and the velocity vector of one or more color blobs.

55. The method of claim 54, further comprising issuing a drive command to maneuver the robot toward a color blob having the largest size and velocity vector toward the robot.

56. The method of claim 54, further comprising executing a heuristic related to color blob size and color blob speed to filter out color blobs non-indicative of debris on the floor surface.

57. The method of claim 41, further comprising assigning a numerical representation for the color of each pixel in a color space.

58. The method of claim 57, further comprising color quantizing the image pixels in a red-green-blue color space, reducing the image to a 9-bit red-green-blue image.

59. The method of claim 57, further comprising color quantizing the image pixels in a LAB color space.

60. The method of claim 41, further comprising executing a control system having a control arbitration system and a behavior system in communication with each other, the behavior system executing a cleaning behavior, the cleaning behavior influencing execution of commands by the control arbitration system based on the image segmentation to identify color blobs corresponding to a dirty floor area and color blob tracking to maneuver over the dirty floor area tier cleaning using a cleaning system of the robot.

61. A computer program product encoded on a non-transitory computer readable storage medium comprising instructions that when executed by a data processing apparatus cause the data processing apparatus to perform operations comprising:

receiving a sequence of images of a floor surface, each image having an array of pixels;

for each image, segmenting the image into color blobs by;

color quantizing pixels of the image;

determining a spatial distribution of each color of the image based on corresponding pixel locations; and

for each image color, identifying areas of the image having a threshold spatial distribution for that color; and

tracking a location of each color blob with respect to the imaging sensor across the sequence of images.

62. The computer program product of claim 61, wherein segmenting the image into color blobs further comprises only color quantizing pixels in a lower portion of the image oriented vertically.

63. The computer program product of claim 61, wherein segmenting the image into color blobs further comprises only color quantizing pixels outside of a center portion of the image.

64. The computer program product of claim 61, wherein segmenting the image into color blobs further comprises dividing the image into regions and separately color quantizing the pixels of each region.

65. The computer program product of claim 61, wherein segmenting the image into color blobs further comprises executing a bit shifting operation to convert each pixel from a first color set to second color set smaller than the first color set.

66. The computer program product of claim 65, wherein the hit shifting operation retains the three most significant bits of each of a red, green and blue channel.

67. The computer program product of claim 61, wherein tracking a location of the color blobs comprises:

determining a velocity vector of each color blob with respect to the imaging; and

recording determined color blob locations for each image of the image sequence.

68. The computer program product of claim 61, further comprising determining a size of each color blob.

69. The computer program product of claim 61, further comprising issuing a drive command to maneuver a robot based on the location of one or more color blobs.

70. The computer program product of claim 69, further comprising issuing a drive command to maneuver the robot toward a nearest color blob.

71. The computer program product of claim 70, wherein the nearest color blob is identified in a threshold number of images of the image sequence.

72. The computer program product of claim 61, wherein the operations further comprise:

determining a size of each color blob;

determining a velocity vector of each color blob with respect to an imaging sensor capturing the received image sequence; and

issuing a drive command to maneuver a robot based on the size and the velocity vector of one or more color blobs.

73. The computer program product of claim 72, wherein the operations further comprise issuing a drive command to maneuver the robot toward a color blob having the largest size and velocity vector toward the robot.

74. The computer program product of claim 72, wherein the operations further comprise executing a heuristic related to color blob size and color blob speed to filter out color blobs non-indicative of debris on the floor surface.

75. The computer program product of claim 61, further comprising assigning a numerical representation for the color of each pixel in a color space.

76. The computer program product of claim 75, wherein the operations further comprise color quantizing the image pixels in a red-green-blue color space, reducing the image to a 9-bit red-green-blue image.

77. The computer program product of claim 75, wherein the operations further comprise color quantizing the image pixels in a LAB color space.