Systems and methods for capturing images and annotating the captured images with information

• US 9,704,043 B2
• Filed: 12/16/2014
• Issued: 07/11/2017
• Est. Priority Date: 12/16/2014
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Schnittman, Mark S.
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US14/572,712
• Publication Number:
  US 20160167226A1
• Time in Patent Office: 938 Days
• Field of Search: None
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

The present teachings provide an autonomous mobile robot that includes a drive configured to maneuver the robot over a ground surface within an operating environment; a camera mounted on the robot having a field of view including the floor adjacent the mobile robot in the drive direction of the mobile robot; a frame buffer that stores image frames obtained by the camera while the mobile robot is driving; and a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in image frames corresponding to portions of the operating environment and determined by mobile robot sensor events.

Claims

1. A method for training a classifier of a mobile robot, the method comprising:

obtaining a plurality of image frames along a drive direction of the mobile robot,

the plurality of image frames comprising a base image frame corresponding to an initial pose of the mobile robot and subsequent image frames obtained at intervals during forward travel of the mobile robot,

the mobile robot having a forward facing camera mounted thereon for obtaining the image frames,

the camera having a field of view including a floor in front of the mobile robot, and

the mobile robot having a memory device configured to store a learned data set of a plurality of descriptors determined by mobile robot events;

assuming that a location is traversable floor, wherein the mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more sensors mounted on the mobile robot;

determining that the location is non-traversable non-floor based on a robot sensor event at the location, the robot sensor event comprising detection of a collision;

retrieving from a frame buffer an image frame obtained immediately prior to the robot sensor event;

generating a floor descriptor corresponding to characteristics of the floor at a bottom of the image frame captured by the camera immediately prior to the robot sensor event;

generating a non-floor descriptor corresponding to characteristics of the non-traversable non-floor at a top of the image frame captured by the camera immediately prior to the robot sensor event; and

storing the floor descriptor and the non-floor descriptor in the learned data set.

2. The method of claim 1, wherein the detection of the collision comprises determining that the mobile robot has collided with an obstacle, and further includes:

receiving a bumper signal indicating the collision from a bumper sensor of the mobile robot; and

verifying that the mobile robot was traveling straight prior to the collision for at least one second.

3. The method of claim 1, wherein the location corresponds to a patch of pixels and one or more characteristics of the patch of pixels includes color and/or texture.

4. The method of claim 1, wherein the learned dataset is continuously updated and oldest descriptors are replaced by new descriptors.

5. The method of claim 1, wherein the learned dataset is stored in memory and remains accessible by the mobile robot between runs.

6. The method of claim 1, wherein the learned dataset is unpopulated at a start of each new run and the classifier trains the learned dataset with descriptors over the run of the mobile robot.

7. An autonomous mobile robot comprising:

a drive configured to maneuver the autonomous mobile robot over a floor within an operating environment;

a camera mounted on the autonomous mobile robot having a field of view including the floor adjacent the autonomous mobile robot in a drive direction of the autonomous mobile robot;

a frame buffer that stores image frames obtained by the camera while the autonomous mobile robot is driving;

a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in ones of the image frames corresponding to portions of the operating environment and determined by mobile robot sensor events; and

one or more processors configured to execute a training process for a classifier of the learned data set, the training process comprising;

determining based on one or more mobile robot sensor events that the autonomous mobile robot collided with an obstacle;

retrieving a pre-collision frame from the frame buffer; and

generating a descriptor corresponding to at least part of the obstacle observed in the pre-collision frame.

8. The autonomous mobile robot of claim 7,

identifying a lower portion of the pre-collision frame and an upper portion of the pre-collision frame;

generating a first descriptor corresponding to the floor that the autonomous mobile robot is traveling on based on the lower portion;

generate a second descriptor corresponding to at least part of the obstacle observed in the pre-collision frame based on the upper portion; and

storing the first descriptor and the second descriptor in the learned data set.

9. An autonomous mobile robot comprising:

a drive configured to maneuver the autonomous mobile robot over a floor within an operating environment;

a camera mounted on the autonomous mobile robot having a field of view including the floor adjacent the autonomous mobile robot in a drive direction of the autonomous mobile robot;

a frame buffer that stores image frames obtained by the camera while the autonomous mobile robot is driving; and

a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in ones of the image frames corresponding to portions of the operating environment and determined by sensor events;

one or more processors executing a training process for a classifier of the learned data set, the training process comprising;

assuming that a portion of a base image frame is traversable floor, wherein the autonomous mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more sensors mounted on the autonomous mobile robot;

determining whether the autonomous mobile robot has traversed a threshold distance in a same direction since obtaining the base image frame;

identifying an upper portion and a lower portion of the base image frame;

identifying a section of the lower portion of the base image frame corresponding to a current pose of the autonomous mobile robot, the section being an area of the floor depicted in the lower portion of the base image frame at a depth corresponding to a drive distance traversed by the mobile robot from an initial pose to the current pose;

generating a floor descriptor of the section;

generating a non-floor descriptor corresponding to characteristics of the non-traversable non-floor within the upper portion of the base image frame responsive to detecting a sensor event comprising a collision; and

storing the floor descriptor and the non-floor descriptor in the learned data set.

10. The autonomous mobile robot of claim 9, wherein the training process further comprises not detecting a sensor event of a collision with an obstacle while traversing the threshold distance.

11. The autonomous mobile robot of claim 9, wherein the training process further comprises:

detecting a new heading of the autonomous mobile robot; and

obtaining a plurality of image frames along the new heading of the autonomous mobile robot, the plurality of image frames comprising a new base image frame corresponding to a new initial pose of the autonomous mobile robot at the new heading.

12. The autonomous mobile robot of claim 9, wherein determining whether the autonomous mobile robot has traversed the threshold distance in the same direction since obtaining the base image frame without detecting a sensor event comprises determining a distance between the initial pose of the mobile robot and the current pose of the mobile robot.

13. The autonomous mobile robot of claim 9, wherein identifying the upper portion and the lower portion of the base image frame comprises identifying a horizon within the base image frame.

14. A method for training a classifier of a mobile robot, the method comprising:

obtaining a plurality of image frames along a drive direction of the mobile robot,

the plurality of image frames comprising a base image frame corresponding to an initial pose of the mobile robot and subsequent image frames obtained at intervals during forward travel of the mobile robot,

the mobile robot having a forward facing camera mounted thereon for obtaining the image frames,

the camera having a field of view including a floor in front of the mobile robot, and

the mobile robot having a memory device configured to store a learned data set of a plurality of descriptors determined by robot sensor events;

tracking a location as non-traversable non-floor based on the plurality of descriptors, wherein the mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more sensors mounted on the mobile robot;

determining that the location is traversable floor after traveling a distance to the location and not detecting a robot sensor event at the location;

generating a floor descriptor corresponding to characteristics of the traversable floor at the location within one of the image frames captured by the camera;

determining that a new location is non-traversable non-floor based on detecting a robot sensor event comprising a collision at the new location;

generating a non-floor descriptor corresponding to characteristics of the non-traversable non-floor at the new location within one of the image frames captured by the camera; and

updating the descriptors in the learned data set.

15. The method of claim 14, wherein the distance traveled is based on an expected change in a wheel encoder.

16. The method of claim 14, further comprising detecting a heading change and resetting a tracking of the location to the new location that corresponds to the non-traversable non-floor.

17. The method of claim 14, wherein the location corresponds to a patch of pixels and one or more characteristics of the patch of pixels includes at least one characteristic selected from the group consisting of:

color; and

texture.

18. The method of claim 14, wherein not detecting the robot sensor event comprises determining that the mobile robot has not collided with an obstacle.

19. An autonomous mobile robot comprising:

a drive configured to maneuver the autonomous mobile robot over a floor within an operating environment;

a camera mounted on the mobile robot having a field of view including the floor adjacent the autonomous mobile robot in a drive direction of the autonomous mobile robot;

a frame buffer that stores image frames obtained by the camera while the autonomous mobile robot is driving; and

a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in ones of the image frames corresponding to portions of the operating environment and determined by robot sensor events;

one or more processors executing a training process for a classifier of the learned data set, the training process comprising;

tracking a location as non-traversable non-floor based on the plurality of descriptors, wherein the autonomous mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more sensors mounted on the mobile robot;

determining that the location is traversable floor after traveling a distance to the location and not detecting a robot sensor event at the location;

generating a floor descriptor corresponding to characteristics of the traversable floor at the location within one of the image frames captured by the camera;

determining that a new location is non-traversable non-floor based on detecting a robot sensor event comprising a collision at the new location;

generating a non-floor descriptor corresponding to characteristics of the non-traversable non-floor at the new location within one of the image frames captured by the camera; and

updating the descriptors in the learned data set.

20. The autonomous mobile robot of claim 19, wherein the distance traveled is based on an expected change in a wheel encoder.

21. The autonomous mobile robot of claim 19, wherein the training process further comprises detecting a heading change and resetting a tracking of the location to a new location that corresponds to non-traversable non-floor.

22. The autonomous mobile robot of claim 19, wherein the location corresponds to a patch of pixels and one or more characteristics of the patch of pixels includes at least one characteristic selected from the group consisting of:

color; and

texture.

23. The autonomous mobile robot of claim 19, wherein not detecting the sensor event comprises determining that the autonomous mobile robot has not collided with an obstacle.

24. An autonomous mobile robot comprising:

a drive configured to maneuver the autonomous mobile robot over a floor within an operating environment;

a camera mounted on the mobile robot having a field of view including the floor adjacent the autonomous mobile robot in a drive direction of the autonomous mobile robot;

a frame buffer that stores image frames obtained by the camera while the autonomous mobile robot is driving; and

a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in one of the image frames corresponding to portions of the operating environment and determined by mobile robot sensor events;

one or more processors executing a training process for a classifier of the learned data set, the training process comprising;

detecting a plurality of obstacles located at a plurality of distances from the autonomous mobile robot;

tracking a first obstacle that is closest to the autonomous mobile robot and buffering descriptors of the first obstacle in the learned data set;

traveling a threshold distance and detecting a second obstacle that is closer to the autonomous mobile robot than the first obstacle;

resetting the autonomous mobile robot to not track the first obstacle; and

tracking the second obstacle and buffering descriptors of the second obstacle in the learned data set, wherein the descriptors of the first obstacle and/or the second obstacle are determined by at least one mobile robot sensor event comprising a collision.

25. The autonomous mobile robot of claim 24, wherein an obstacle corresponds to a patch of pixels and one or more characteristics of the patch of pixels includes at least one characteristic selected from the group consisting of:

color; and

texture.

26. The autonomous mobile robot of claim 24, wherein tracking the first obstacle and/or the second obstacle comprises tracking a movement of a patch of pixels through a plurality of the image frames based on movement of the autonomous mobile robot.

27. The autonomous mobile robot of claim 24, wherein the training process further comprises traveling a threshold distance prior to capturing a new image.

28. The autonomous mobile robot of claim 24, wherein the training process further comprises buffering descriptors of a closest obstacle and updating the learned dataset with the descriptors.