Systems and methods for capturing images and annotating the captured images with information

• US 9,836,653 B2
• Filed: 03/29/2017
• Issued: 12/05/2017
• Est. Priority Date: 12/16/2014
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Schnittman, Mark S.
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US15/473,327
• Publication Number:
  US 20170206418A1
• Time in Patent Office: 251 Days
• Field of Search: None
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

The present teachings provide an autonomous mobile robot that includes a drive configured to maneuver the robot over a ground surface within an operating environment; a camera mounted on the robot having a field of view including the floor adjacent the mobile robot in the drive direction of the mobile robot; a frame buffer that stores image frames obtained by the camera while the mobile robot is driving; and a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in image frames corresponding to portions of the operating environment and determined by mobile robot sensor events.

Claims

1. An autonomous mobile robot, comprising:

a drive configured to maneuver the autonomous mobile robot over a floor surface within an operating environment;

a camera having a field of view in a drive direction of the autonomous mobile robot;

a plurality of sensors configured to detect obstacles in the drive direction of the autonomous mobile robot;

a frame buffer configured to store image frames obtained by the camera while the mobile robot is driving, the image frames corresponding to portions of the operating environment;

a controller configured to maintain a map of the operating environment and associate metadata with a respective image frame obtained by the camera, the metadata comprising a time and pose of the autonomous mobile robot when the respective image frame was captured; and

a memory device configured to store a learned data set comprising a plurality of descriptors corresponding to pixel patches in the image frames, wherein one or more of the descriptors are updated based on sensor events indicative of one or more of the obstacles detected by the sensors that do not correspond with information in the image frames,

wherein the controller is configured to update the map with locations of the one or more of the obstacles in the portions of the operating environment that are indicated by the sensor events and do not correspond with the information in the image frames.

2. The autonomous mobile robot of claim 1, wherein the plurality of sensors comprises an imaging sensor.

3. The autonomous mobile robot of claim 1, wherein the plurality of sensors comprises an inertial measurement unit (IMU), an odometry sensor, a piezoelectric sensor, a structured light optical sensor, an infrared (IR) proximity sensor, a laser sensor, a lidar sensor, a volumetric point cloud sensor, a time of flight sensor, a PIXART imaging sensor, a PRIMESENSE sensor, and/or an RGBD sensor.

4. The autonomous mobile robot of claim 1, wherein the controller is configured to generate the map of the operating environment using visual simultaneous localization and mapping (VSLAM) based on the image frames.

5. The autonomous mobile robot of claim 1, wherein the controller is further configured to identify different angles of the operating environment from the map and execute a retraining process for one or more classifiers of the learned data set based on the different angles.

6. The autonomous mobile robot of claim 5, wherein the controller is configured to execute the retraining process for the one or more classifiers of the learned data set based on a time of day.

7. The autonomous mobile robot of claim 6, wherein the controller is further configured to track movement of one of the pixel patches through a plurality of the image frames based on movement of the autonomous mobile robot, wherein the one of the pixel patches comprises sunlight moving across the floor surface.

8. The autonomous mobile robot of claim 1, wherein the camera comprises one or more cameras mounted on a top and/or bottom of the autonomous mobile robot.

9. The autonomous mobile robot of claim 1, wherein the autonomous mobile robot further comprises an input/output interface that is configured to wirelessly communicate with at least one remote computing device.

10. The autonomous mobile robot of claim 1, wherein the controller is further configured to continuously update the descriptors of the learned data set based on subsequent image frames obtained by the camera, and execute a retraining process for one or more classifiers of the learned data set responsive thereto.

11. The autonomous mobile robot of claim 1, wherein the controller is further configured to:

track a location as traversable floor based on the plurality of descriptors, wherein the autonomous mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more of the sensors;

determine that the location is non-traversable non-floor based on a sensor event detected at the location;

retrieve, from the frame buffer, an image frame captured immediately prior to the sensor event;

generate a non-floor descriptor corresponding to the location in the image frame captured immediately prior to the sensor event, wherein the image frame captured immediately prior to the sensor event has a pose of the autonomous mobile robot associated therewith; and

store the non-floor descriptor in the learned data set.

12. The autonomous mobile robot of claim 11, wherein the camera has a field of view including a portion of the floor surface, and wherein the controller is further configured to:

generate the non-floor descriptor corresponding to characteristics of the non-traversable non-floor at a top of the image frame captured immediately prior to the sensor event;

generate a floor descriptor corresponding to characteristics of the floor surface at a bottom of the image frame captured immediately prior to the sensor event; and

store the floor descriptor and the non-floor descriptor in the learned data set.

13. The autonomous mobile robot of claim 1, wherein the controller is further configured to:

track a location as non-traversable non-floor based on the plurality of descriptors, wherein the autonomous mobile robot is configured to detect traversable floor and non-traversable non-floor with one or more of the sensors;

determine that the location is traversable floor after traveling a distance to the location and not detecting a sensor event at the location;

generate a floor descriptor corresponding to the location within one of the image frames having a pose of the autonomous mobile robot associated therewith; and

store the floor descriptor in the learned data set.

19. The autonomous mobile robot of claim 1, wherein the controller is further configured to execute a retraining process for one or more classifiers of the learned data set using the one or more of the descriptors updated based on the sensor events.

14. A method of operating an autonomous mobile robot, the method comprising:

maneuvering, by a drive system, the autonomous mobile robot over a floor surface within an operating environment;

obtaining, by a camera having a field of view in a drive direction of the autonomous mobile robot, a plurality of image frames corresponding to portions of the operating environment;

storing the image frames in a frame buffer;

detecting, by a plurality of sensors, one or more obstacles in the drive direction of the autonomous mobile robot;

maintaining, by a controller, a map of the operating environment and associating metadata with a respective image frame obtained by the camera, the metadata comprising a time and pose of the autonomous mobile robot when the respective image frame was captured;

storing, in a memory device, a learned data set comprising a plurality of descriptors corresponding to pixel patches in the image frames, wherein one or more of the descriptors are updated based on sensor events indicative of one or more of the obstacles detected by the sensors that do not correspond with information in the image frames; and

updating the map with locations of the one or more of the obstacles in the portions of the operating environment that are indicated by the sensor events and do not correspond with the information in the image frames.

15. The method of claim 14, further comprising:

identifying, by the controller, different angles of the operating environment from the map; and

executing, by the controller, a retraining process for one or more classifiers of the learned data set based on the different angles.

16. The method of claim 15, wherein executing the retraining process for the one or more classifiers of the learned data set is further based on a time of day.

17. The method of claim 16, further comprising:

tracking, by the controller, movement of one of the pixel patches through a plurality of the image frames based on movement of the autonomous mobile robot, wherein the one of the pixel patches comprises sunlight moving across the floor surface.

18. The method of claim 14, wherein the plurality of sensors comprises an inertial measurement unit (IMU), an odometry sensor, a piezoelectric sensor, a structured light optical sensor, an infrared (IR) proximity sensor, a laser sensor, a lidar sensor, a volumetric point cloud sensor, a time of flight sensor, a PIXART imaging sensor, a PRIMESENSE sensor, and/or an RGBD sensor.

20. The method of claim 14, further comprising:

executing, by the controller, a retraining process for one or more classifiers of the learned data set using the one or more of the descriptors updated based on the sensor events.