ASYNCHRONOUS IMAGE CLASSIFICATION

• US 20180348783A1
• Filed: 05/31/2017
• Published: 12/06/2018
• Est. Priority Date: 05/31/2017
• Status: Abandoned Application

• Current Assignee: Neato Robotics, Inc. (Vorwerk & Company KG)
• Original Assignee: Neato Robotics, Inc. (Vorwerk & Company KG)
• Inventors: Pitzer, Charles Albert, Brooks, Griswald, Capriles, Jose, Lucas, Rachel, Yee, Kingman
• Application Number:
  US15/610,401
• Publication Number:
  US 20180348783A1
• Time in Patent Office: Days
• CPC Class Codes: A47L 2201/04 Automatic control of the travelling movement; Automatic obstacle detection
• CPC Class Codes: A47L 9/009 Carrying-vehicles; Arrangements of trollies or wheels; Means for avoiding mechanical obstacles
• CPC Class Codes: A47L 9/0477 Rolls
• CPC Class Codes: A47L 9/14 Bags or the like; Rigid filtering receptacles; Attachment of, or closures for, bags or receptacles
• CPC Class Codes: A47L 9/2826 the condition of the floor
• CPC Class Codes: A47L 9/2847 Surface treating elements
• CPC Class Codes: A47L 9/2852 Elements for displacement of the vacuum cleaner or the accessories therefor, e.g. wheels, casters or nozzles
• CPC Class Codes: A47L 9/2857 User input or output elements for control, e.g. buttons, switches or displays
• CPC Class Codes: A47L 9/2873 Docking units or charging stations for autonomous or robotic vacuum cleaners A47L2201/02
• CPC Class Codes: A47L 9/2884 Details of arrangements of batteries or their installation
• CPC Class Codes: A47L 9/2889 Safety or protection devices or systems, e.g. for prevention of motor over-heating or for protection of the user
• CPC Class Codes: A47L 9/2894 Details related to signal transmission in suction cleaners
• CPC Class Codes: G05D 1/0214 in accordance with safety or protection criteria, e.g. avoiding hazardous areas monitoring the location of vehicles within a certain area, e.g. forbidden or allowed areas, in traffic control systems for road vehicles G08G1/13
• CPC Class Codes: G05D 1/0221 involving a learning process
• CPC Class Codes: G05D 1/0227 using mechanical sensing means, e.g. for sensing treated area
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0246 using a video camera in combination with image processing means
• CPC Class Codes: G05D 1/0248 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0274 using mapping information stored in a memory device navigation using map-matching G01C21/30
• CPC Class Codes: G05D 1/0285 using signals transmitted via a public communication network, e.g. GSM network
• CPC Class Codes: G06T 2207/20132 : Image cropping
• CPC Class Codes: G06T 7/11 : Region-based segmentation
• CPC Class Codes: G06V 20/10 : Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: G06V 20/20 : in augmented reality scenes
• CPC Class Codes: G06V 20/35 : Categorising the entire scene, e.g. birthday party or wedding scene
• CPC Class Codes: H04N 23/50 : Constructional details
• CPC Class Codes: H04N 23/51 : Housings
• CPC Class Codes: H04N 5/38 : Transmitter circuitry for the transmission of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

Embodiments provide methods and apparatus for asynchronously classifying images provided by a robot. In a reconnaissance/exploratory or first cleaning pass, unidentified objects are avoided. Images of the object are uploaded over the Internet to a remote object detection and classification system, and the location is indicated by the cleaning robot. When the remote system subsequently returns an object identification or classification, the object can be indicated as something to be avoided, or the cleaning robot can return to the location and clean over the object if it is determined not to be a hazard. In one embodiment, the object is passed over at reduced speed or with a cleaning brush turned off.

Claims

1. A mobile cleaning robot comprising:

a robotic apparatus with a housing;

a drive motor mounted in the housing;

a drive system, coupled to the drive motor, for moving the robotic apparatus;

a processor;

a memory;

a distance and object detection sensor;

an image sensor;

a wireless transceiver;

a non-transitory computer readable media, coupled to the processor, containing instructions for;

creating a map of an operating environment using data from the distance and object detection sensor,

performing image processing to recognize objects in images captured by the image sensor,

cropping the objects in the images,

transmitting the cropped images using the wireless transceiver,

tagging a location of the cropped image on the map,

receiving an object classification of the cropped images from a remote object classifier, with the object classification indicating whether the objects are a potential hazard,

returning to and cleaning tagged locations of objects not indicated as a potential hazard.

2. The mobile cleaning robot of claim 1 further comprising:

a cleaning brush mounted in the housing;

a brush motor coupled to the cleaning brush; and

the non-transitory computer readable media further containing instructions for;

turning off the brush motor when the mobile cleaning robot passes over one of the objects.

3. The mobile cleaning robot of claim 1 wherein the turning off the brush motor when the robot passes over one of the objects is performed before the object is classified to indicate whether the object is a potential hazard.

4. The mobile cleaning robot of claim 1 wherein the non-transitory computer readable media further containing instructions for:

slowing down the drive motor when the mobile cleaning robot passes over one of the objects.

5. The mobile cleaning robot of claim 1 wherein indicating whether the object is a potential hazard comprises a confidence rating.

6. The mobile cleaning robot of claim 1 wherein the non-transitory computer readable media further contains instructions for obtaining at least one additional image of the potential object from a different viewpoint.

7. The mobile cleaning robot of claim 1 further comprising:

a pressure sensor for providing a signal corresponding to contact with an object;

wherein the non-transitory computer readable media further comprises instructions for;

maneuvering the cleaning robot to initiate contact by the pressure sensor with the object;

recording the amount of pressure detected by the pressure sensor;

associating data corresponding to the amount of pressure with the object and the location of the object; and

transmitting data corresponding to the amount of pressure.

8. The mobile cleaning robot of claim 1 wherein the non-transitory computer readable media further comprises instructions for:

receiving an impairment indication that the operability of the robot has been impaired;

retrieving pre-impairment images obtained for a period of time prior to the impairment indication; and

transmitting at least a portion of the pre-impairment images for remote storage in a hazards database.

9. A method for operating a mobile cleaning robot comprising:

creating a map of an operating environment using data from a distance and object detection sensor in the mobile cleaning robot;

performing image processing to recognize objects in images captured by an image sensor in the mobile cleaning robot;

cropping the objects in the images;

transmitting the cropped images using a wireless transceiver in the mobile cleaning robot;

tagging locations of the cropped images on the map;

receiving an object classification of the cropped images from a remote object classifier, with the object classification indicating whether the objects are a potential hazard; and

returning to and cleaning tagged locations of objects not indicated as a potential hazard.

10. The method of claim 9 further comprising:

performing one of turning off a brush motor and reducing a mobile cleaning robot speed when the mobile cleaning robot passes over one of the objects.

11. The method of claim 10 wherein the turning off the brush motor when the robot passes over one of the objects is performed before the object is classified to indicate whether the object is a potential hazard.

12. The method of claim 9 further comprising:

maintaining a database of object images with tagged classifications;

comparing the cropped images to the object images in the database; and

obtaining responses from a user questionnaire; and

limiting the comparing to object images in accordance with the responses from the user questionnaire.

13. The method of claim 9 wherein the object classification indicates a type of object.

14. The method of claim 9 further comprising:

providing at least one cropped image to a user device;

receiving from the user device an user hazard indication of whether the at least one cropped image is a hazard; and

storing the user hazard indication in association with the cropped image in a hazard database; and

comparing subsequent images to the stored cropped image in the hazard database; and

providing a hazard indication probability for each of the subsequent images based at least in part on the degree of similarity to the stored cropped image in the hazard database.

15. The method claim 9 further comprising:

receiving an impairment indication that the operability of the robot has been impaired;

retrieving pre-impairment images obtained for a period of time prior to the impairment indication; and

transmitting at least a portion of the pre-impairment images for remote storage in a hazards database.

16. The method of claim 15 further comprising:

controlling the cleaning robot to take a remedial action to attempt to overcome the impairment;

if the remedial action is successful, tagging the pre-impairment images with a first hazard probability; and

if the remedial action is unsuccessful, tagging the pre-impairment image with a second hazard probability, wherein the second hazard probability is higher than the first hazard probability.

17. The method of claim 9 further comprising:

returning to and cleaning tagged locations of objects indicated as a potential hazard with a hazard probability below a set level; and

receiving a user input to adjust the set level.

18. The method of claim 9 further comprising:

maneuvering the cleaning robot to initiate contact by a pressure sensor with an object;

recording the amount of pressure detected by the pressure sensor;

associating data corresponding to the amount of pressure with an image of the object and the location of the object; and

transmitting data corresponding to the amount of pressure to a remote database.

19. The method of claim 9 further comprising:

maneuvering the cleaning robot to initiate contact with an object;

recording whether the object moved as a result of the contact;

associating data corresponding to the amount of movement of the object with an image of the object and the location of the object; and

transmitting data corresponding to the amount of movement to a remote database.

20. A mobile cleaning robot comprising:

a housing;

a drive motor mounted in the housing;

a drive system, coupled to the drive motor, for moving the mobile cleaning robot;

a cleaning element, mounted in the housing;

a processor;

a distance and object detection sensor comprising a source providing collimated light output in an emitted light beam and a detector sensor operative to detect a reflected light beam from the emitted light beam incident on an object, and further comprising;

a rotating mount to which said source and said detector sensor are attached;

an angular orientation sensor operative to detect an angular orientation of the rotating mount;

a first non-transitory, computer readable media including instructions for

computing distance between the rotating mount and the object,

determining a direction of the stationary object relative to the robotic device using the angular orientation of the rotating mount, and applying a simultaneous localization and mapping (SLAM) algorithm to the distance and the direction to determine a location of the robotic device and to map an operating environment;

a second non-transitory computer readable media, coupled to the processor, containing instructions for;

creating a map of an operating environment using data from the distance and object detection sensor,

performing image processing to recognize objects in images captured by the image sensor,

cropping the objects in the images,

transmitting the cropped images using the wireless transceiver,

tagging a location of the cropped image on the map,

receiving an object classification of the cropped images from a remote object classifier, with the object classification indicating whether the objects are a potential hazard,

returning to and cleaning tagged locations of objects not indicated as a potential hazard.