Methods and systems for providing landmarks to facilitate robot localization and visual odometry

• US 9,427,874 B1
• Filed: 11/14/2014
• Issued: 08/30/2016
• Est. Priority Date: 08/25/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Rublee, Ethan
• Primary Examiner(s): Tran, Khoi
• Assistant Examiner(s): RINK, RYAN J
• Application Number:
  US14/541,576
• Time in Patent Office: 655 Days
• US Class Current: 1/1
• CPC Class Codes: B05B 12/122 responsive to presence or shape of target B05B12/124 takes precedence
• CPC Class Codes: B05B 13/0431 with spray heads moved by robots or articulated arms, e.g. for applying liquid or other fluent material to 3D-surfaces B05B13/0436, B05B13/0442, B05B13/0447, B05B13/0463 take precedence
• CPC Class Codes: B25J 11/0075 Manipulators for painting or coating
• CPC Class Codes: B25J 13/089 Determining the position of the robot with reference to its environment
• CPC Class Codes: B25J 19/021 Optical sensing devices
• CPC Class Codes: B25J 9/1697 Vision controlled systems
• CPC Class Codes: B65G 2203/041 Camera
• CPC Class Codes: B65G 59/02 De-stacking from the top of the stack
• CPC Class Codes: B65G 61/00 Use of pick-up or transfer devices or of manipulators for stacking or de-stacking articles not otherwise provided for
• CPC Class Codes: G05B 2219/40298 Manipulator on vehicle, wheels, mobile
• CPC Class Codes: G05B 2219/40543 Identification and location, position of components, objects
• CPC Class Codes: Y10S 901/01 Mobile robot
• CPC Class Codes: Y10S 901/43 Spray painting or coating
• CPC Class Codes: Y10S 901/47 Optical

Abstract

Methods and systems for providing landmarks to facilitate robot localization and visual odometry are provided herein. At least one area of a physical environment in which a robotic device resides may be determined to include surfaces that lack sufficient discernable features to determine a location of the at least one area. Instructions may responsively be provided to the robotic device for the robotic device to provide a material in respective patterns onto one or more surfaces of the at least one area. Instructions can responsively be provided for the robotic device to provide the material in respective textures as well. The respective patterns or textures may include sufficient discernable features to determine a location of the at least one area, and the material may remain on the one or more surfaces for a predetermined period of time.

Claims

1. A method, comprising:

acquiring, by a computing device, at least one image of one or more surfaces of at least one area of a physical environment in which a robotic device resides;

engaging in an attempt to extract discernable features from the at least one image, whereby a number of discernable features are extracted;

based on a determination that the number of discernable features that are extracted is below a threshold, determining that the one or more surfaces lack sufficient discernable features to determine a location of the at least one area in the physical environment; and

in response to determining that the one or more surfaces lack sufficient discernable features to determine a location of the at least one area in the physical environment, providing instructions to the robotic device to provide a material in respective patterns onto the one or more surfaces of the at least one area, wherein the respective patterns include sufficient discernable features to determine a location of the at least one area in the physical environment, and wherein the material is predicted to remain on the one or more surfaces for a predetermined period of time.

2. The method of claim 1, wherein each of the respective patterns are unique.

3. The method of claim 1, wherein the material includes one or more of:

non-reflective paint, reflective paint, invisible ink, and disappearing ink.

4. The method of claim 1, wherein providing instructions to the robotic device to provide the material in respective patterns onto the one or more surfaces of the at least one area comprises providing instructions to the robotic device to provide the material as one or more of a two-dimensional (2D) pattern or a three-dimensional (3D) texture onto the one or more surfaces of the at least one area.

5. The method of claim 1, wherein providing instructions to the robotic device to provide the material in the respective patterns onto the one or more surfaces of the at least one area comprises providing instructions to the robotic device to first provide a primer material onto the one or more surfaces of the at least one area and subsequently provide the material in the respective patterns onto the one or more surfaces of the at least one area.

6. The method of claim 1, further comprising:

before providing the instructions to the robotic device, determining a classification for the one or more surfaces of the at least one area, wherein the classification is representative of (i) a type of surface of the one or more surfaces that lack sufficient discernable features to determine a location of the at least one area and (ii) a type of texture of the type of surface; and

based on the classification, determining at least one type of pattern to be provided by the robotic device onto the one or more surfaces of the type of the at least one area,

wherein providing instructions to the robotic device to provide the material in the respective patterns onto the one or more surfaces of the at least one area comprises providing instructions to the robotic device to provide the material in respective patterns of the at least one type of pattern onto the one or more surfaces of the type of the at least one area.

7. A method, comprising:

determining, by a computing device, that at least one area of a physical environment in which a robotic device resides includes one or more surfaces that lack sufficient discernable features to determine a location of the at least one area in the physical environment;

determining a classification for the one or more surfaces of the at least one area, wherein the classification is representative of (i) a type of surface of the one or more surfaces and (ii) a type of texture of the one or more surfaces;

based on the classification, determining at least one type of pattern to be provided by the robotic device onto the one or more surfaces; and

in response to determining the at least one type of pattern, providing instructions to the robotic device to provide a material in respective patterns of the at least one type of pattern onto the one or more surfaces, wherein the respective patterns include sufficient discernable features to determine a location of the at least one area in the physical environment, and wherein the material is predicted to remain on the one or more surfaces for a predetermined period of time.

8. The method of claim 7, wherein each of the respective patterns are unique.

9. The method of claim 7, wherein the material includes one or more of:

non-reflective paint, reflective paint, invisible ink, and disappearing ink.

10. The method of claim 7, wherein providing instructions to the robotic device to provide the material in respective patterns of the at least one type of pattern onto the one or more surfaces comprises providing instructions to the robotic device to provide the material as one or more of a two-dimensional (2D) pattern or a three-dimensional (3D) texture onto the one or more surfaces.

11. The method of claim 7, wherein providing instructions to the robotic device to provide the material in respective patterns of the at least one type of pattern onto the one or more surfaces comprises providing instructions to the robotic device to first provide a primer material onto the one or more surfaces and subsequently provide the material in the respective patterns of the at least one type of pattern onto the one or more surfaces.

12. A robotic device, comprising:

at least one processor;

data storage comprising instructions executable by the at least one processor to cause the system to perform operations comprising;

acquiring at least one image of one or more surfaces of at least one area of a physical environment in which a robotic device resides;

engaging in an attempt to extract discernable features from the at least one image, whereby a number of discernable features are extracted;

based on a determination that the number of discernable features that are extracted is below a threshold, determining that the one or more surfaces lack sufficient discernable features to determine a location of the at least one area in the physical environment; and

in response to determining that the one or more surfaces lack sufficient discernable features to determine a location of the at least one area in the physical environment, causing the robotic device to provide a material in respective patterns onto the one or more surfaces of the at least one area, wherein the respective patterns include sufficient discernable features to determine a location of the at least one area in the physical environment, and wherein the material is predicted to remain on the one or more surfaces for a predetermined period of time.

13. The system of claim 12, wherein each of the respective patterns are unique.

14. The system of claim 12, wherein the material includes one or more of:

non-reflective paint, reflective paint, invisible ink, and disappearing ink.

15. The system of claim 12, wherein providing instructions to the robotic device to provide the material in respective patterns onto the one or more surfaces of the at least one area comprises providing instructions to the robotic device to provide the material as one or more of a two-dimensional (2D) pattern or a three-dimensional (3D) texture onto the one or more surfaces of the at least one area.

16. The system of claim 12, wherein causing the robotic device to provide the material in the respective patterns onto the one or more surfaces of the at least one area comprises causing the robotic device to first provide a primer material onto the one or more surfaces of the at least one area and subsequently provide the material in the respective patterns onto the one or more surfaces of the at least one area.

17. The system of claim 12, further comprising:

before causing the robotic device to provide the material in the respective patterns onto the one or more surfaces of the at least one area, determining a classification for the one or more surfaces of the at least one area, wherein the classification is representative of (i) a type of surface of the one or more surfaces that lack sufficient discernable features to determine a location of the at least one area and (ii) a type of texture of the type of surface; and

based on the classification, determining at least one type of pattern to be provided by the robotic device onto the one or more surfaces of the type of the at least one area,

wherein causing the robotic device to provide the material in the respective patterns onto the one or more surfaces of the at least one area comprises causing the robotic device to provide the material in respective patterns of the at least one type of pattern onto the one or more surfaces of the type of the at least one area.