Systems and methods for capturing images and annotating the captured images with information

• US 10,102,429 B2
• Filed: 11/06/2017
• Issued: 10/16/2018
• Est. Priority Date: 12/16/2014
• Status: Active Grant

• Current Assignee: iRobot Corporation
• Original Assignee: iRobot Corporation
• Inventors: Schnittman, Mark S.
• Primary Examiner(s): Gilliard, Delomia L
• Application Number:
  US15/804,495
• Publication Number:
  US 20180114064A1
• Time in Patent Office: 344 Days
• Field of Search: None
• CPC Class Codes: B25J 19/023 including video camera means
• CPC Class Codes: B25J 5/00 Manipulators mounted on wheels or on carriages B25J1/00 takes precedence; programme-controlled manipulators B25J9/00 ; vehicle aspects B60, B62, e.g. remote-controlled steering for motor vehicles B62D1/24; control of position of vehicles G05D1/00
• CPC Class Codes: G05D 1/0088 characterized by the autonomous decision making process, e.g. artificial intelligence, predefined behaviours using knowledge based models G06N5/00
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 in augmented reality scenes
• CPC Class Codes: G06V 20/64 Three-dimensional objects
• CPC Class Codes: G06V 30/1916 Validation; Performance evaluation
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/47 Optical

Abstract

The present teachings provide an autonomous mobile robot that includes a drive configured to maneuver the robot over a ground surface within an operating environment; a camera mounted on the robot having a field of view including the floor adjacent the mobile robot in the drive direction of the mobile robot; a frame buffer that stores image frames obtained by the camera while the mobile robot is driving; and a memory device configured to store a learned data set of a plurality of descriptors corresponding to pixel patches in image frames corresponding to portions of the operating environment and determined by mobile robot sensor events.

Claims

1. An autonomous mobile robot, comprising:

a drive configured to maneuver the autonomous mobile robot in an operating environment;

a camera having a field of view in a drive direction of the autonomous mobile robot;

at least one sensor configured to detect obstacles in the drive direction responsive to sensor events;

a frame buffer configured to store image frames obtained by the camera and corresponding to regions of the operating environment;

a memory device configured to store a learned data set comprising a plurality of descriptors corresponding to pixel patches of the image frames and one or more classifiers therefor; and

a controller configured to update one or more of the descriptors based on a sensor event indicative of one or more of the obstacles detected by the at least one sensor responsive to operation of the drive, and to operate the drive to maneuver the autonomous mobile robot to traverse one of the regions previously avoided or avoid one of the regions previously attempted to traverse responsive to a retraining process for the one or more classifiers based on the one or more of the descriptors that were updated.

2. The autonomous mobile robot of claim 1, wherein the controller is further configured to identify ones of the image frames as corresponding to different views of a same region among the regions of the operating environment, and wherein the retraining process is performed responsive to inconsistency of respective outputs of the one or more classifiers of the learned data set for the ones of the image frames.

3. The autonomous mobile robot of claim 2, wherein the controller is configured to execute the retraining process for the one or more classifiers of the learned data set based on a time of day.

4. The autonomous mobile robot of claim 3, wherein the controller is further configured to track movement of one of the pixel patches through the ones of the image frames, wherein the one of the pixel patches comprises sunlight moving across a floor surface based on the time of day.

5. The autonomous mobile robot of claim 1, wherein the controller is further configured to continuously update the descriptors of the learned data set based on subsequent image frames obtained by the camera, and execute the retraining process for the one or more classifiers of the learned data set responsive thereto.

6. The autonomous mobile robot of claim 5, wherein the controller is configured to execute a training process for the one or more classifiers based on an initial data set prior to executing the retraining process, wherein the initial data set does not correspond to the regions of the operating environment.

7. The autonomous mobile robot of claim 1, wherein the controller is configured to generate a map of the operating environment using visual simultaneous localization and mapping (VSLAM) based on the image frames, and is configured to annotate one or more of the pixel patches in the image frames as corresponding to the one or more of the obstacles such that the map is indicative of a presence of the one or more of the obstacles.

8. The autonomous mobile robot of claim 1, wherein the camera comprises one or more cameras mounted on a top and/or bottom of the autonomous mobile robot.

9. The autonomous mobile robot of claim 1, wherein the autonomous mobile robot further comprises an input/output interface that is configured to wirelessly communicate with at least one remote computing device that performs the retraining process.

10. The autonomous mobile robot of claim 1, wherein the controller is further configured to:

track ones of the pixel patches of the image frames as corresponding to traversable floor based on the one or more of the descriptors thereof;

determine that the ones of the pixel patches correspond to non-traversable non-floor based on the sensor event detected at a corresponding region among the regions of the operating environment;

retrieve, from the frame buffer, an image frame among the image frames, wherein the image frame is captured immediately prior to the sensor event; and

update the one or more of the descriptors by generating a non-floor descriptor corresponding to the ones of the pixel patches in the image frame captured immediately prior to the sensor event, wherein the image frame captured immediately prior to the sensor event has a pose of the autonomous mobile robot associated therewith, and storing the non-floor descriptor in the learned data set.

11. The autonomous mobile robot of claim 10, wherein the field of view includes a portion of a floor surface, and wherein the controller is further configured to:

generate the non-floor descriptor corresponding to characteristics of the non-traversable non-floor at a top of the image frame captured immediately prior to the sensor event;

generate a floor descriptor corresponding to characteristics of the floor surface at a bottom of the image frame captured immediately prior to the sensor event; and

store the floor descriptor and the non-floor descriptor in the learned data set.

12. The autonomous mobile robot of claim 1, wherein the controller is further configured to:

track ones of pixel patches of the image frames as corresponding to non-traversable non-floor based on the one or more of the descriptors corresponding thereto;

determine that the ones of the pixel patches correspond to traversable floor based on absence of the sensor event at a corresponding region among the regions of the operating environment; and

update the one or more of the descriptors by generating a floor descriptor corresponding to the ones of the pixel patches in the image frames, the image frames having respective poses of the autonomous mobile robot associated therewith, and storing the floor descriptor in the learned data set.

13. The autonomous mobile robot of claim 1, wherein the at least one sensor comprises an inertial measurement unit (IMU), an odometry sensor, a piezoelectric sensor, a structured light optical sensor, an infrared (IR) proximity sensor, a laser sensor, a lidar sensor, a volumetric point cloud sensor, a time of flight sensor, a PIXART imaging sensor, a PRIMESENSE sensor, and/or an RGBD sensor.

14. A method of operating an autonomous mobile robot, the method comprising:

maneuvering, by a drive system, the autonomous mobile robot in an operating environment;

obtaining, by a camera having a field of view in a drive direction of the autonomous mobile robot, a plurality of image frames corresponding to regions of the operating environment;

storing the image frames in a frame buffer;

detecting, by at least one sensor, one or more obstacles in the drive direction;

storing, in a memory device, a learned data set comprising a plurality of descriptors corresponding to pixel patches in the image frames;

updating, by a controller, one or more of the descriptors based on a sensor event indicative of the one or more of the obstacles detected by the at least one sensor responsive to the maneuvering; and

operating, by the controller, the drive to maneuver the autonomous mobile robot to traverse one of the regions previously avoided or avoid one of the regions previously attempted to traverse responsive to a retraining process for one or more classifiers based on the one or more of the descriptors that were updated.

15. The method of claim 14, further comprising:

identifying, by the controller, ones of the image frames as corresponding to different views of a same region among the regions of the operating environment,

wherein the retraining process is performed responsive to inconsistency of respective outputs of the one or more classifiers of the learned data set for the ones of the image frames.

16. The method of claim 15, further comprising:

executing, by the controller, the retraining process for the one or more classifiers of the learned data set based on a time of day.

17. The method of claim 16, further comprising:

tracking, by the controller, movement of one of the pixel patches through the ones of the image frames, wherein the one of the pixel patches comprises sunlight moving across a floor surface based on the time of day.

18. The method of claim 14, further comprising:

generating, by a controller, a map of the operating environment using visual simultaneous localization and mapping (VSLAM) based on the image frames; and

annotating, by the controller, one or more of the pixel patches in the image frames as corresponding to the one or more of the obstacles such that the map is indicative of a presence of the one or more of the obstacles.

19. The method of claim 14, wherein the at least one sensor comprises an inertial measurement unit (IMU), an odometry sensor, a piezoelectric sensor, a structured light optical sensor, an infrared (IR) proximity sensor, a laser sensor, a lidar sensor, a volumetric point cloud sensor, a time of flight sensor, a PIXART imaging sensor, a PRIMESENSE sensor, and/or an RGBD sensor.