Conversion of cleaning robot camera images to floorplan for user interaction

• US 10,638,906 B2
• Filed: 12/15/2017
• Issued: 05/05/2020
• Est. Priority Date: 12/15/2017
• Status: Active Grant

• Current Assignee: Vorwerk & Company Interholding Gesellschaft Mit Beschrã¤Nkter Haftung (Vorwerk & Company KG)
• Original Assignee: Neato Robotics, Inc. (Vorwerk & Company KG)
• Inventors: Herron, Matthew Allison, Lucas, Rachel, Pitzer, Charles Albert, Brooks, Griswald, Yee, Kingman
• Primary Examiner(s): Matt, Marnie A
• Application Number:
  US15/844,437
• Publication Number:
  US 20190183310A1
• Time in Patent Office: 872 Days
• Field of Search: 348114
• CPC Class Codes: A47L 11/4011 Regulation of the cleaning machine by electric means; Control systems and remote control systems therefor
• CPC Class Codes: A47L 11/4061 Steering means; Means for avoiding obstacles; Details related to the place where the driver is accommodated
• CPC Class Codes: A47L 2201/00 Robotic cleaning machines, i.e. with automatic control of the travelling movement or the cleaning operation
• CPC Class Codes: A47L 9/2805 Parameters or conditions being sensed
• CPC Class Codes: A47L 9/2852 Elements for displacement of the vacuum cleaner or the accessories therefor, e.g. wheels, casters or nozzles
• CPC Class Codes: A47L 9/2857 User input or output elements for control, e.g. buttons, switches or displays
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/63 : by using electronic viewfinders
• CPC Class Codes: H04N 23/71 : Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

In one embodiment, a method is disclosed for a robot (e.g., cleaning robot) to produce a bird'"'"'s eye view (planar) map by transforming from a robot camera view and stitching together images tagged by location. The planar map can then be presented on a user interface as a floorplan, showing the location of objects (such as rugs). A camera is mounted in the cleaning robot sufficiently high in the cleaning robot housing to provide an angled view downward at a floor. The camera'"'"'s field of view is captured as an image, and a portion or segment of that image is cropped. The cropped segment is transformed from the camera perspective to a planar view (before or after cropping), and is combined with other images to form a map of a floorplan.

Claims

1. A method for providing a floorplan populated with images captured by a floor-traversing cleaning robot, comprising:

providing a camera mounted in the floor-traversing cleaning robot sufficiently high in a robot housing to provide an angled view downward at a floor;

creating localization data using a SLAM algorithm to localize the floor-traversing cleaning robot;

creating a rectilinear grid of the floorplan using the localization data;

capturing a series of images of the floor with the camera at different locations as the floor-traversing cleaning robot traverses the floor;

determining a location of the floor-traversing cleaning robot for each of the images;

tagging each of the series of images with a location of the floor-traversing cleaning robot at a time of image capture;

cropping each of the images to identify at least one image segment in each of the images;

transforming the image segment from a robot camera perspective view to a planar view;

populating rectangles in the rectilinear grid of the floorplan with each of the image segments at the robot location in the planar view to provide an image populated floorplan, with an offset corresponding to a difference between the robot position and the position of the image segment on the floor, utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan;

displaying the image populated floorplan to a user; and

accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing cleaning robot;

wherein the field of view of the camera begins 5-25 centimeters in front of the floor-traversing cleaning robot.

2. The method of claim 1 further comprising:

providing a lens in front of the robot camera;

applying a distortion algorithm to the series of images to correct for an image distortion due to the lens.

3. The method of claim 1 further comprising:

detecting tilting of the floor-traversing cleaning robot beyond a first threshold; and

correcting the image pixel position or discarding an image.

4. The method of claim 1 further comprising:

capturing segments for an area that is the same size or larger than a minimum resolution of the floorplan.

5. The method of claim 1, further comprising:

capturing overlapping segments; and

adjusting alignment adjustment by matching an overlapping portion of one segment with an overlapping portion of an adjacent segment.

6. The method of claim 1 further comprising:

providing a source of illumination on the floor-traversing cleaning robot for a field of view of the camera;

storing a baseline image brightness value for the source of illumination;

detecting variances in illumination of different segments from a baseline; and

making any needed corrections in the brightness of the segments due to illumination variances.

7. The system of claim 1 further comprising:

a LIDAR module mounted in the floor-traversing cleaning robot and configured to determine a location of the floor-traversing cleaning robot using a SLAM algorithm.

8. The method of claim 1 wherein the user input comprises the user indicating at least one of areas where the floor-traversing cleaning robot should not go and areas that require more intensive cleaning.

9. A method for providing a floorplan from images captured by a floor-traversing robot, comprising:

providing a camera mounted in the floor-traversing robot sufficiently high in a robot housing to provide an angled view downward at a floor;

capturing a series of images of the floor with the camera at different locations as the floor-traversing robot traverses the floor;

determining a location of the floor-traversing robot for each of the images;

tagging each of the series of images with a location of the floor-traversing robot at a time of image capture;

cropping each of the images to identify at least one segment in each of the images;

transforming the segment from a robot camera perspective view to a planar view;

populating the floorplan with each of the segments at the robot location in the planar view, with an offset corresponding to a difference between the robot position and the position of the segment on the floor, utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan;

displaying the floorplan to a user;

accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing robot;

capturing the same segment multiple times to provide multiple versions of the same segment;

determining a weighted quality of each version of the same segment, with the weighed quality including at least one of the speed of the robot when an image was captured, vibration, angle (tilt), agreement with overlapping portions of other segments, and closeness of the segment to the camera and illumination; and

replacing a lower quality version with a higher quality version.

10. The method of claim 9, wherein the weighted quality is determined using at least one of a speed of the robot when an image was captured, vibration, tilt of the robot, agreement with overlapping portions of other segments, closeness of the segment to the camera and illumination.

11. A method for providing a floorplan populated with images captured by a floor-traversing cleaning robot, comprising:

providing a camera mounted in the floor-traversing cleaning robot sufficiently high in a robot housing to provide an angled view downward at a floor;

creating localization data using a SLAM algorithm to localize the floor-traversing cleaning robot;

creating a rectilinear grid of the floorplan using the localization data;

capturing a series of images of the floor with the camera;

providing a lens in front of the robot camera;

applying a distortion algorithm to the series of images to correct for an image distortion due to the lens;

determining a location of the floor-traversing cleaning robot for each of the images;

tagging each of the series of images with the location of the floor-traversing cleaning robot at a time of image capture;

detecting tilting of the cleaning robot beyond a first threshold;

correcting an image pixel position or discarding the image;

cropping each of the images to identify at least one segment in each of the images;

capturing segments for an area that is the same size or larger than a minimum resolution of the floorplan;

transforming the segment from a robot camera perspective view to a planar view;

populating rectangles in the rectilinear grid of the floorplan with each of the segments at the robot location in the planar view to provide an image populated floorplan by utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan, with an offset corresponding to a difference between the robot position and the position of the segment on the floor;

capturing overlapping segments;

adjusting alignment adjustment by matching an overlapping portion of one segment with an overlapping portion of an adjacent segment;

displaying the image populated floorplan to a user; and

accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing cleaning robot.

12. A method for providing a floorplan from images captured by a floor-traversing robot, comprising:

providing a camera mounted in the floor-traversing robot sufficiently high in a robot housing to provide an angled view downward at a floor;

capturing a series of images of the floor with the camera at different locations as the floor-traversing robot traverses the floor;

determining a location of the floor-traversing robot for each of the images;

synchronizing a frame rate of image capture with a rate of determining a location of the robot;

tagging each of the series of images with a location of the floor-traversing robot at a time of image capture;

cropping each of the images to identify at least one segment in each of the images;

transforming the segment from a robot camera perspective view to a planar view;

populating the floorplan with each of the segments at the robot location in the planar view, with an offset corresponding to a difference between the robot position and the position of the segment on the floor, utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan;

displaying the floorplan to a user; and

accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing robot.

13. A method for providing a floorplan populated with images captured by a floor-traversing cleaning robot, comprising:

providing a camera mounted in the floor-traversing cleaning robot sufficiently high in a robot housing to provide an angled view downward at a floor;

creating localization data using a SLAM algorithm to localize the floor-traversing cleaning robot;

creating a rectilinear grid of the floorplan using the localization data;

capturing a series of images of the floor with the camera at different locations as the floor-traversing cleaning robot traverses the floor;

determining a location of the floor-traversing cleaning robot for each of the images;

tagging each of the series of images with a location of the floor-traversing cleaning robot at a time of image capture;

cropping each of the images to identify at least one image segment in each of the images;

transforming the image segment from a robot camera perspective view to a planar view;

populating rectangles in the rectilinear grid of the floorplan with each of the image segments at the robot location in the planar view to provide an image populated floorplan, with an offset corresponding to a difference between the robot position and the position of the image segment on the floor, utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan;

displaying the image populated floorplan to a user; and

accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing cleaning robot;

wherein the resolution of the camera is less than 1280×

720 pixels.

14. A method for providing a floorplan populated with images captured by a floor-traversing cleaning robot, comprising:

providing a camera mounted in the floor-traversing cleaning robot sufficiently high in a robot housing to provide an angled view downward at a floor;

creating localization data using a SLAM algorithm to localize the floor-traversing cleaning robot;

creating a rectilinear grid of the floorplan using the localization data;

capturing a series of images of the floor with the camera at different locations as the floor-traversing cleaning robot traverses the floor;

determining a location of the floor-traversing cleaning robot for each of the images;

tagging each of the series of images with a location of the floor-traversing cleaning robot at a time of image capture;

cropping each of the images to identify at least one image segment in each of the images;

transforming the image segment from a robot camera perspective view to a planar view;

populating rectangles in the rectilinear grid of the floorplan with each of the image segments at the robot location in the planar view to provide an image populated floorplan, with an offset corresponding to a difference between the robot position and the position of the image segment on the floor, utilizing a known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan;

displaying the image populated floorplan to a user; and

accepting a user input on the floorplan for modifying an operating characteristic of the floor-traversing cleaning robot;

wherein the field of view of the camera is between 100 and 180 degrees and the camera is mounted 5-20 centimeters above the floor.

15. The system of claim 14 further comprising:

a lens mounted in front of the robot camera; and

non-transitory, computer-readable code having instructions for

applying a distortion algorithm to the series of images to correct for an image distortion due to the lens.

16. The system of claim 14 further comprising:

a tilt sensor mounted in the floor-traversing cleaning robot; and

non-transitory, computer-readable code imbedded in computer readable media having instructions for

detecting tilting of the floor-traversing cleaning robot beyond a first threshold; and

correcting the image pixel position or discarding an image.

17. The system of claim 14 further comprising:

a second camera mounted in the floor-traversing cleaning robot sufficiently high in the robot housing to provide an angled view downward at a floor with a field of view that overlaps with the first-mentioned camera.

18. The system of claim 14 further comprising:

a light mounted in the floor-traversing cleaning robot to provide illumination of a field of view of the camera;

non-transitory, computer-readable code imbedded in computer readable media having instructions for

storing a baseline image brightness value for a source of illumination,

detecting variances in illumination of different segments from a baseline; and

making any needed corrections in the brightness of the segments due to illumination variances.