Methods and systems for providing landmarks to facilitate robot localization and visual odometry

• US 10,059,006 B2
• Filed: 08/02/2016
• Issued: 08/28/2018
• Est. Priority Date: 08/25/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Rublee, Ethan
• Primary Examiner(s): Rink, Ryan
• Application Number:
  US15/226,353
• Publication Number:
  US 20160339587A1
• Time in Patent Office: 756 Days
• CPC Class Codes: B05B 12/122 responsive to presence or shape of target B05B12/124 takes precedence
• CPC Class Codes: B05B 13/0431 with spray heads moved by robots or articulated arms, e.g. for applying liquid or other fluent material to 3D-surfaces B05B13/0436, B05B13/0442, B05B13/0447, B05B13/0463 take precedence
• CPC Class Codes: B25J 11/0075 Manipulators for painting or coating
• CPC Class Codes: B25J 13/089 Determining the position of the robot with reference to its environment
• CPC Class Codes: B25J 19/021 Optical sensing devices
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 2203/041 Camera
• CPC Class Codes: B65G 59/02 De-stacking from the top of the stack
• CPC Class Codes: B65G 61/00 Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
• CPC Class Codes: G05B 2219/40298 Manipulator on vehicle, wheels, mobile
• CPC Class Codes: G05B 2219/40543 Identification and location, position of components, objects
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/43 Spray painting or coating
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Methods and systems for providing landmarks to facilitate robot localization and visual odometry are provided herein. At least one area of a physical environment in which a robotic device resides may be determined to include surfaces that lack sufficient discernable features to determine a location of the at least one area. Instructions may responsively be provided to the robotic device for the robotic device to provide a material in respective patterns onto one or more surfaces of the at least one area. Instructions can responsively be provided for the robotic device to provide the material in respective textures as well. The respective patterns or textures may include sufficient discernable features to determine a location of the at least one area, and the material may remain on the one or more surfaces for a predetermined period of time.

Claims

1. A system comprising:

a robotic device;

an applicator configured to provide a material onto a surface of an object, wherein the material, once provided on the surface, is designed to disappear from the surface within a predetermined period of time;

one or more optical sensors;

at least one processor;

data storage comprising instructions executable by the at least one processor to cause the system to perform operations comprising;

causing the robotic device to provide, using the applicator, the material in a pattern onto one or more surfaces in a physical environment; and

based on the material being designed to disappear from the one or more surfaces within the predetermined period of time;

detecting, within the predetermined period of time and using the one or more optical sensors, the pattern on the one or more surfaces, wherein the pattern includes a discernable feature sufficient to determine a first location of the one or more surfaces in the physical environment;

determining, within the predetermined period of time, the discernable feature from the detected pattern on the one or more surfaces, and

using, within the predetermined period of time, the discernable feature to determine the first location of the one or more surfaces with respect to a second location of the robotic device in the physical environment.

2. The system of claim 1, wherein the one or more surfaces are one or more surfaces of at least one object, the operations further comprising:

in response to using the discernable feature to determine the first location of the one or more surfaces with respect to the second location of the robotic device in the physical environment, causing the robotic device to manipulate the at least one object.

3. The system of claim 1, wherein the one or more surfaces are one or more surfaces of at least one moving object, and

wherein using the discernable feature to determine the first location of the one or more surfaces with respect to the second location of the robotic device in the physical environment comprises using the discernable feature to sense the first location of the at least one moving object with respect to the second location of the robotic device over a period of time as the at least one moving object moves in the physical environment.

4. The system of claim 1, further comprising a projector, wherein detecting, using the one or more optical sensors, the pattern on the one or more surfaces comprises:

using the projector to project light onto the one or more surfaces at substantially a location of the pattern, wherein the projected light includes one or more of ultraviolet (UV) light and infrared (IR) light; and

while projecting the light onto the one or more surfaces, detecting, using the one or more optical sensors, the pattern on the one or more surfaces.

5. The system of claim 1, wherein the pattern is unique.

6. The system of claim 1, wherein the material includes one or more of:

non-reflective paint, reflective paint, invisible ink, and disappearing ink.

7. The system of claim 1, wherein causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment comprises causing the robotic device to provide, using the applicator, the material as one or more of a two-dimensional (2D) pattern or a three-dimensional (3D) texture onto the one or more surfaces in the physical environment.

8. The system of claim 1, wherein causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment comprises causing the robotic device to first provide a primer material onto the one or more surfaces and subsequently provide, using the applicator, the material in the pattern onto the one or more surfaces.

9. A method comprising:

causing a robotic device to provide, using an applicator, a material in a pattern onto one or more surfaces in a physical environment, wherein the material, once provided on the one or more surfaces, is designed to disappear from the one or more surfaces within a predetermined period of time; and

based on the material being designed to disappear from the one or more surfaces within the predetermined period of time;

detecting, within the predetermined period of time and using one or more optical sensors, the pattern on the one or more surfaces, wherein the pattern includes a discernable feature sufficient to determine a first location of the one or more surfaces in the physical environment;

determining, within the predetermined period of time, the discernable feature from the detected pattern on the one or more surfaces, and

using, within the predetermined period of time, the discernable feature to determine the first location of the one or more surfaces with respect to a second location of the robotic device in the physical environment.

10. The method of claim 9, wherein determining the first location of the one or more surfaces with respect to the second location of the robotic device in the physical environment comprises using a simultaneous localization and mapping (SLAM) process to determine (i) the first location of the one or more surfaces with respect to the second location of the robotic device in the physical environment and (ii) a map of at least a portion of the physical environment.

11. The method of claim 10, wherein the map includes a visual representation of at least the portion of the physical environment, and

wherein the map includes one or more of:

a topological map of the physical environment and a metric map of the physical environment.

12. The method of claim 9, further comprising:

before causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment;

engaging in an attempt to extract at least one discernable feature from at least one image of the one or more surfaces, whereby a number of discernable features are extracted; and

based on a determination that the number of discernable features that are extracted is below a threshold, determining that the one or more surfaces lack sufficient discernable features to determine the first location of the one or more surfaces in the physical environment,

wherein causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment comprises causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment in response to determining that the one or more surfaces lack sufficient discernable features to determine the first location of the one or more surfaces in the physical environment.

13. The method of claim 9, further comprising:

before causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment;

determining a classification for the one or more surfaces, wherein the classification is representative of (i) a type of surface of the one or more surfaces and (ii) a type of texture of the one or more surfaces; and

based on the classification, determining at least one type of pattern to be provided by the robotic device onto the one or more surfaces,

wherein causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment comprises causing the robotic device to provide, using the applicator, the material in the pattern of the at least one type of pattern onto the one or more surfaces in the physical environment.

14. The method of claim 9, further comprising:

before causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment;

estimating an amount of time to determine the first location of the one or more surfaces with respect to the second location of the robotic device in the physical environment; and

selecting the material from a plurality of materials based on the predetermined period of time of the selected material being within a threshold amount of time from the estimated amount of time.

15. A non-transitory computer-readable medium having stored thereon program instructions that when executed by a computing system that includes at least one processor cause the computing system to perform operations comprising:

causing a robotic device to provide, using an applicator, a material in a pattern onto one or more surfaces in a physical environment, wherein the material, once provided on the one or more surfaces, is designed to disappear from the one or more surfaces within a predetermined period of time; and

based on the material being designed to disappear from the one or more surfaces within the predetermined period of time;

detecting, within the predetermined period of time and using one or more optical sensors, the pattern on the one or more surfaces, wherein the pattern includes a discernable feature sufficient to determine a first location of the one or more surfaces in the physical environment;

determining, within the predetermined period of time, the discernable feature from the detected pattern on the one or more surfaces, and

using, within the predetermined period of time, the discernable feature to determine the first location of the one or more surfaces with respect to a second location of the robotic device in the physical environment.

16. The non-transitory computer-readable medium of claim 15, wherein the one or more surfaces are one or more surfaces of at least one object, the operations further comprising:

in response to using the discernable feature to determine the first location of the one or more surfaces with respect to the second location of the robotic device in the physical environment, causing the robotic device to manipulate the at least one object.

17. The non-transitory computer-readable medium of claim 15, wherein the one or more surfaces are one or more surfaces of at least one moving object, and

wherein using the discernable features to determine the first location of the one or more surfaces with respect to the second location of the robotic device in the physical environment comprises using the discernable feature to sense the first location of the at least one moving object with respect to the second location of the robotic device over a period of time as the at least one moving object moves in the physical environment.

18. The non-transitory computer-readable medium of claim 15, wherein the pattern is unique.

19. The non-transitory computer-readable medium of claim 15, the operations further comprising:

before causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment;

engaging in an attempt to extract at least one discernable feature from at least one image of the one or more surfaces, whereby a number of discernable features are extracted; and

based on a determination that the number of discernable features that are extracted is below a threshold, determining that the one or more surfaces lack sufficient discernable features to determine the first location of the one or more surfaces in the physical environment,

wherein causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment comprises causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment in response to determining that the one or more surfaces lack sufficient discernable features to determine the first location of the one or more surfaces in the physical environment.

20. The non-transitory computer-readable medium of claim 15, further comprising:

before causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment;

determining a classification for the one or more surfaces, wherein the classification is representative of (i) a type of surface of the one or more surfaces and (ii) a type of texture of the one or more surfaces; and

based on the classification, determining at least one type of pattern to be provided by the robotic device onto the one or more surfaces,

wherein causing the robotic device to provide, using the applicator, the material in the pattern onto the one or more surfaces in the physical environment comprises causing the robotic device to provide, using the applicator, the material in the pattern of the at least one type of pattern onto the one or more surfaces in the physical environment.