PHOTOMOSAIC FLOOR MAPPING

• US 20190183310A1
• Filed: 12/15/2017
• Published: 06/20/2019
• Est. Priority Date: 12/15/2017
• Status: Active Grant

• Current Assignee: Vorwerk & Company Interholding Gesellschaft Mit Beschrã¤Nkter Haftung (Vorwerk & Company KG)
• Original Assignee: Neato Robotics, Inc. (Vorwerk & Company KG)
• Inventors: Herron, Matthew Allison, Lucas, Rachel, Pitzer, Charles Albert, Brooks, Griswald, Yee, Kingman
• Granted Patent: US 10,638,906 B2
• Time in Patent Office: Days
• CPC Class Codes: A47L 11/4011 Regulation of the cleaning machine by electric means; Control systems and remote control systems therefor
• CPC Class Codes: A47L 11/4061 Steering means; Means for avoiding obstacles; Details related to the place where the driver is accommodated
• CPC Class Codes: A47L 2201/00 Robotic cleaning machines, i.e. with automatic control of the travelling movement or the cleaning operation
• CPC Class Codes: A47L 9/2805 Parameters or conditions being sensed
• CPC Class Codes: A47L 9/2852 Elements for displacement of the vacuum cleaner or the accessories therefor, e.g. wheels, casters or nozzles
• CPC Class Codes: A47L 9/2857 User input or output elements for control, e.g. buttons, switches or displays
• CPC Class Codes: G01S 17/42 Simultaneous measurement of distance and other co-ordinates indirect measurement G01S17/46
• CPC Class Codes: G05D 1/0044 by providing the operator with a computer generated representation of the environment of the vehicle, e.g. virtual reality, maps maps used for automatic navigation G05D1/0274; flight directors G01C23/005
• CPC Class Codes: G05D 1/0231 using optical position detecting means position-fixing by using electromagnetic waves other than radio waves, e.g. optical position detecting means G01S5/16
• CPC Class Codes: G05D 1/024 in combination with a laser lasers per se H01S
• CPC Class Codes: G05D 1/0253 extracting relative motion information from a plurality of images taken successively, e.g. visual odometry, optical flow determining position or orientation from images G06T7/70
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20132 Image cropping
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/63 : by using electronic viewfinders
• CPC Class Codes: H04N 23/71 : Circuitry for evaluating the brightness variation
• CPC Class Codes: H04N 23/90 : Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums

Abstract

In one embodiment, a method is disclosed for a robot (e.g., cleaning robot) to produce a bird'"'"'s eye view (planar) map by transforming from a robot camera view and stitching together images tagged by location. The planar map can then be presented on a user interface as a floorplan, showing the location of objects (such as rugs). A camera is mounted in the cleaning robot sufficiently high in the cleaning robot housing to provide an angled view downward at a floor. The camera'"'"'s field of view is captured as an image, and a portion or segment of that image is cropped. The cropped segment is transformed from the camera perspective to a planar view (before or after cropping), and is combined with other images to form a map of a floorplan.

Claims

1. A method for providing a floorplan from images captured by a floor- traversing robot, comprising:

providing a camera mounted in the robot sufficiently high in a robot housing to provide an angled view downward at a floor;

capturing a series of images of the floor with the camera;

determining a location of the robot for each of the images;

tagging each of the series of images with a location of the robot at the time of image capture;

cropping each of the images to identify at least one segment in each of the images;

transforming the segment from a robot camera perspective view to a planar view;

populating the floorplan with each of the segments at the robot location in the planar view, with an offset corresponding to a difference between the robot position and the position of the segment on the floor;

displaying the floorplan to a user; and

accepting a user input on the floorplan for modifying an operating characteristic of the robot.

2. The method of claim 1 further comprising:

providing a lens in front of the robot camera;

applying a distortion algorithm to the series of images to correct for the image distortion due to the lens.

3. The method of claim 1 wherein populating the floorplan with each of the segments at the robot location in the planar view further comprises:

utilizing the known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan.

4. The method of claim 3 further comprising:

detecting tilting of the robot beyond a first threshold; and

correcting the image pixel position or discarding the image.

5. The method of claim 1 further comprising:

capturing segments for an area that is the same size or larger than a minimum resolution of the floorplan.

6. The method of claim 1, further comprising:

capturing overlapping segments; and

adjusting alignment adjustment by matching an overlapping portion of one segment with an overlapping portion of an adjacent segment.

7. The method of claim 1 further comprising:

capturing the same segment multiple times to provide multiple versions of the same segment;

determining a weighted quality of each version of the same segment;

replacing a lower quality version with a higher quality version.

8. The method of claim 7, wherein the weighted quality is determined using at least one of a speed of the robot when the image was captured, vibration, tilt of the robot, agreement with overlapping portions of other segments, closeness of the segment to the camera and illumination.

9. The method of claim 1 further comprising:

providing a source of illumination on the robot for the field of view of the camera;

storing a baseline image brightness value for the source of illumination;

detecting variances in illumination of different segments from the baseline; and

making any needed corrections in the brightness of the segments due to illumination variances.

10. A method for providing a floorplan from images captured by a floor- traversing robot, comprising:

providing a camera mounted in the robot sufficiently high in a robot housing to provide an angled view downward at a floor;

capturing a series of images of the floor with the camera;

providing a lens in front of the robot camera;

applying a distortion algorithm to the series of images to correct for the image distortion due to the lens;

determining a location of the robot for each of the images;

tagging each of the series of images with a location of the robot at the time of image capture;

detecting tilting of the robot beyond a first threshold;

correcting the image pixel position or discarding the image;

cropping each of the images to identify at least one segment in each of the images;

capturing segments for an area that is the same size or larger than a minimum resolution of the floorplan;

transforming the segment from a robot camera perspective view to a planar view;

populating the floorplan with each of the segments at the robot location in the planar view by utilizing the known manufactured height of the camera off the floor and known downward angle of the camera to correlate an image pixel position with a planar floor position in the floorplan, with an offset corresponding to a difference between the robot position and the position of the segment on the floor;

capturing overlapping segments;

adjusting alignment adjustment by matching an overlapping portion of one segment with an overlapping portion of an adjacent segment;

displaying the floorplan to a user; and

accepting a user input on the floorplan for modifying an operating characteristic of the robot.

11. A system for providing a floorplan from images, comprising:

a floor-traversing robot;

a robot housing of the floor-traversing robot;

a motor mounted in the housing for moving the floor-traversing robot;

a battery connected to provide electrical power to the motor;

a camera mounted in the robot sufficiently high in the robot housing to provide an angled view downward at a floor;

a processor mounted in the robot;

a memory in the robot;

a localization module in the robot;

non-transitory, computer-readable code in the memory having instructions for

capturing a series of images of the floor with the camera,

determining a location of the robot for each of the images from data provided by the localization module, and

tagging each of the series of images with a location of the robot at the time of image capture;

non-transitory, computer-readable code having instructions for

cropping each of the images to identify at least one segment in each of the images,

transforming the segment from a robot camera perspective view to a planar view,

populating the floorplan with each of the segments at the robot location in the planar view, with an offset corresponding to a difference between the robot position and the position of the segment on the floor,

displaying the floorplan to a user, and

accepting a user input on the floorplan for modifying an operating characteristic of the robot.

12. The system of claim 11 further comprising:

a lens mounted in front of the robot camera; and

non-transitory, computer-readable code having instructions for

applying a distortion algorithm to the series of images to correct for the image distortion due to the lens.

13. The system of claim 11 further comprising:

a tilt sensor mounted in the robot; and

non-transitory, computer-readable code imbedded in computer readable media having instructions for

detecting tilting of the robot beyond a first threshold; and

correcting the image pixel position or discarding the image.

14. The system of claim 11 further comprising:

a second camera mounted in the robot sufficiently high in the robot housing to provide an angled view downward at a floor with a field of view that overlaps with the first- mentioned camera.

15. The system of claim 11 further comprising:

a light mounted in the robot to provide illumination of the field of view of the camera;

non-transitory, computer-readable code imbedded in computer readable media having instructions for

storing a baseline image brightness value for the source of illumination,

detecting variances in illumination of different segments from the baseline; and

making any needed corrections in the brightness of the segments due to illumination variances.

16. The system of claim 11 further comprising:

a LIDAR module mounted in the robot and configured to determine a location of the robot using a SLAM algorithm.