Systems and Methods for Capturing Images and Annotating the Captured Images with Information

• US 20160167226A1
• Filed: 12/16/2014
• Published: 06/16/2016
• Est. Priority Date: 12/16/2014
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Schnittman, Mark S.
• Granted Patent: US 9,704,043 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

The present teachings provide an autonomous mobile robot that includes a drive configured to maneuver the robot over a ground surface within an operating environment; a camera mounted on the robot having a field of view including the floor adjacent the mobile robot in the drive direction of the mobile robot; a frame buffer that stores image frames obtained by the camera while the mobile robot is driving; and a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in image frames corresponding to portions of the operating environment and determined by mobile robot sensor events.

Claims

1. A method for training a classifier of a mobile robot, the method comprising:

obtaining a plurality of image frames along a drive direction of the mobile robot,

the plurality of image frames comprising a base image frame corresponding to an initial pose of the mobile robot and subsequent image frames obtained at intervals during forward travel of the mobile robot,

the mobile robot having a forward facing camera mounted thereon for obtaining the image frames,

the camera having a field of view including the floor in front of the robot, and

the robot having a memory device configured to store a learned data set of a plurality of descriptors determined by mobile robot events;

assuming that a location is traversable floor, wherein the mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more sensors mounted on the mobile robot;

determining that the location is non-floor based on a robot sensor event at the location;

retrieving from a frame buffer an image frame obtained immediately prior to the sensor event;

generating a floor descriptor corresponding to the characteristics of the floor at the bottom of the image frame captured by the camera immediately prior to the sensor event;

generating a non-floor descriptor corresponding to characteristics of the non-floor at the top of the image frame captured by the camera immediately prior to the sensor event; and

storing the floor descriptor and the non-floor descriptor in the learned data set.

2. The method of claim 1, wherein the sensor event is detection of a collision and determining that the mobile robot has collided with the obstacle includes:

receiving a bumper signal indicating a collision from a bumper sensor of the mobile robot; and

verifying that the mobile robot was traveling straight prior to the collision for at least one second.

3. The method of claim 1, wherein the location corresponds to a patch of pixels and the one or more characteristics of the patch of pixels includes color and/or texture.

4. The method of claim 1, wherein the learned dataset is continuously updated and the oldest descriptors are replaced by new descriptors.

5. The method of claim 1, wherein the learned dataset is stored in memory and remains accessible by the robot between runs.

6. The method of claim 1, wherein the learned dataset is unpopulated at the start of each new run and the classifier trains the dataset with descriptors over the run of the mobile robot.

7. An autonomous mobile robot comprising:

a drive configured to maneuver the robot over a ground surface within an operating environment;

a camera mounted on the robot having a field of view including the floor adjacent the mobile robot in the drive direction of the mobile robot;

a frame buffer that stores image frames obtained by the camera while the mobile robot is driving; and

a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in image frames corresponding to portions of the operating environment and determined by mobile robot sensor events.

8. The mobile robot of claim 7, further comprising one or more processors executing a training process for a classifier of the learned data set, the process comprising:

determining based on one or more sensor events that the mobile robot collided with an obstacle;

retrieving a pre-collision frame from the frame buffer;

identifying a lower portion of the pre-collision frame and an upper portion of the pre-collision frame;

generating a first descriptor corresponding to the ground surface that the mobile robot is traveling on based on the lower portion;

generate a second descriptor corresponding to at least part of the obstacle observed in the pre-collision frame based on the upper portion; and

storing the first descriptor and the second descriptor in the learned data set.

9. An autonomous mobile robot comprising:

a drive configured to maneuver to mobile robot over a ground surface within an operating environment;

a camera mounted on the mobile robot having a field of view including the floor adjacent the mobile robot in the drive direction of the mobile robot;

a frame buffer that stores image frames obtained by the camera while the mobile robot is driving; and

a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in image frames corresponding to portions of the operating environment and determined by mobile robot sensor events;

one or more processors executing a training process for a classifier of the learned data set, the process comprising;

assuming that a portion of the base image frame is traversable floor, wherein the mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more sensors mounted on the mobile robot;

determining whether the mobile robot has traversed a threshold distance in a same direction since obtaining the base image frame;

identifying an upper portion and a lower portion of the base image frame;

identifying a section of the lower portion of the base image frame corresponding to a current pose of the mobile robot, the section being an area of a ground surface depicted in the lower portion of the base image frame at a depth corresponding to a drive distance traversed by the mobile robot from the initial pose to the current pose;

generating a floor descriptor of the section; and

storing the floor descriptor in the learned data set.

10. The mobile robot of claim 9, wherein the process further comprises not detecting a sensor event of a collision with an obstacle while traversing the threshold distance.

11. The mobile robot of claim 9, wherein the process further comprises:

detecting a new heading of the mobile robot; and

obtaining a plurality of image frames along the new heading of the mobile robot, the plurality of image frames comprising a new base image frame corresponding to a new initial pose of the mobile robot at the new heading.

12. The mobile robot of claim 9, wherein determining whether the mobile robot has traversed a threshold distance in the same direction since obtaining the base image without detecting a sensor event comprises determining a distance between the initial pose of the mobile robot and a current pose of the mobile robot.

13. The mobile robot of claim 9, wherein the process further comprises generating a non-floor descriptor corresponding to characteristics of the non-floor within the upper portion of the base image frame.

14. The mobile robot of claim 9, wherein identifying the upper portion and the lower portion of the base image frame comprises identifying a horizon within the base image.

15. A method for training a classifier of a mobile robot, the method comprising:

obtaining a plurality of image frames along a drive direction of the mobile robot,

the plurality of image frames comprising a base image frame corresponding to an initial pose of the mobile robot and subsequent image frames obtained at intervals during forward travel of the mobile robot,

the mobile robot having a forward facing camera mounted thereon for obtaining the image frames,

the camera having a field of view including the floor in front of the robot, and

the robot having a memory device configured to store a learned data set of a plurality of descriptors determined by mobile robot events;

tracking a location as non-traversable non-floor based on the plurality of descriptors, wherein the mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more sensors mounted on the mobile robot;

determining that the location is traversable floor after traveling a distance to the location and not detecting a robot sensor event at the location;

generating a floor descriptor corresponding to the characteristics of the floor at the location within the image frame captured by the camera; and

updating the floor descriptors in the learned data set.

16. The method of claim 15, wherein the distance traveled is based on an expected change in a wheel encoder.

17. The method of claim 15, further comprising detecting a heading change and resetting a tracking of the location to a new location that corresponds to non-traversable non-floor.

18. The method of claim 15, wherein the location corresponds to a patch of pixels and the one or more characteristics of the patch of pixels includes at least one characteristic selected from the group consisting of:

color; and

texture.

19. The method of claim 15, wherein not detecting the sensor event comprises determining that the mobile robot has not collided with an obstacle.

20. An autonomous mobile robot comprising:

a drive configured to maneuver to mobile robot over a ground surface within an operating environment;

a camera mounted on the mobile robot having a field of view including the floor adjacent the mobile robot in the drive direction of the mobile robot;

a frame buffer that stores image frames obtained by the camera while the mobile robot is driving; and

a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in image frames corresponding to portions of the operating environment and determined by mobile robot sensor events.

one or more processors executing a training process for a classifier of the learned data set, the process comprising;

tracking a location as non-traversable non-floor based on the plurality of descriptors, wherein the mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more sensors mounted on the mobile robot;

determining that the location is traversable floor after traveling a distance to the location and not detecting a robot sensor event at the location;

generating a floor descriptor corresponding to the characteristics of the floor at the location within the image frame captured by the camera; and

updating the floor descriptors in the learned data set.

21. The mobile robot of claim 20, wherein the distance traveled is based on an expected change in a wheel encoder.

22. The mobile robot of claim 20, wherein the process further comprises detecting a heading change and resetting a tracking of the location to a new location that corresponds to non-traversable non-floor.

23. The mobile robot of claim 20, wherein the location corresponds to a patch of pixels and the one or more characteristics of the patch of pixels includes at least one characteristic selected from the group consisting of:

color; and

texture.

24. The method of claim 20, wherein not detecting the sensor event comprises determining that the mobile robot has not collided with an obstacle.

25. An autonomous mobile robot comprising:

a drive configured to maneuver to mobile robot over a ground surface within an operating environment;

a camera mounted on the mobile robot having a field of view including the floor adjacent the mobile robot in the drive direction of the mobile robot;

a frame buffer that stores image frames obtained by the camera while the mobile robot is driving; and

a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in image frames corresponding to portions of the operating environment and determined by mobile robot sensor events;

one or more processors executing a training process for a classifier of the learned data set, the process comprising;

detecting a plurality of obstacles located at a plurality of distances from the mobile robot;

tracking a first obstacle that is closest to the mobile robot and buffering descriptors of the first obstacle in the learned data set;

traveling a threshold distance and detecting a second obstacle that is closer to the mobile robot than the first obstacle;

resetting the mobile robot to not track the first obstacle; and

tracking the second obstacle and buffering descriptors of the second obstacle in the learned data set.

26. The mobile robot of claim 25, wherein an obstacle corresponds to a patch of pixels and the one or more characteristics of the patch of pixels includes at least one characteristic selected from the group consisting of:

color; and

texture.

27. The mobile robot of claim 25, wherein tracking an obstacle comprises tracking a movement of a patch of pixels through a plurality of image frames based on movement of the mobile robot.

28. The mobile robot of claim 25, wherein the process further comprises traveling a threshold distance prior to capturing a new image.

29. The method of claim 25, wherein the process further comprises buffering descriptors of the closest obstacle and updating the learned dataset with the descriptors.