Methods and systems for using projected patterns to facilitate mapping of an environment

• US 10,025,886 B1
• Filed: 09/30/2015
• Issued: 07/17/2018
• Est. Priority Date: 09/30/2015
• Status: Active Grant

• Current Assignee: Boston Dynamics, Inc. (SoftBank Group Corp.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Rublee, Ethan, Strasdat, Hauke Malte
• Primary Examiner(s): Nguyen, Bao Long T
• Application Number:
  US14/870,626
• Time in Patent Office: 1,021 Days
• Field of Search: 700250, 700253-254, 700258-259, 901 44, 901 46- 47, 901 50, 31856816, 31856822-56823, 382153, 382173, 382174
• CPC Class Codes: G01B 11/254 Projection of a pattern, viewing through a pattern, e.g. moiré

Abstract

Methods and systems for using projected patterns to facilitate mapping of an environment are provided herein. A computing system may cause fixedly-posed projectors to each provide, onto a respective area of the environment, a predetermined respective distinct pattern. The system may determine respective poses of the projectors, and further determine a map of the environment that identifies, for each distinct pattern, respective locations on one or more surfaces in the environment on which the distinct pattern is detectable. Based on sensor data the system may identify a portion of a particular distinct pattern in the environment. The system may use the map and the respective pose of a particular projector that is providing the particular pattern to make a determination that the portion is located at a new, different location compared to the map. The system may then transmit an output signal indicating the determination.

Claims

1. A method performed by a computing system having one or more processors and a memory, the method comprising:

causing a plurality of projectors that are fixedly posed in an environment to each provide, onto a respective area of the environment, a predetermined respective distinct pattern;

determining respective poses of the plurality of projectors;

determining a map of the environment that identifies, for each distinct pattern, respective locations on one or more surfaces in the environment on which the distinct pattern is detectable;

based on sensor data from at least one sensor, identifying a portion of a particular distinct pattern in the environment;

using the map and the determined respective pose of a particular projector of the plurality that is providing the particular distinct pattern to make a determination that the portion is located in the environment at a new location different from the respective location identified by the map for the particular distinct pattern; and

transmitting an output signal indicating the determination that the portion is located in the environment at the new location different from the respective location identified by the map for the particular distinct pattern.

2. The method of claim 1, wherein the respective locations on the one or more surfaces in the environment are represented in the map as respective three-dimensional (3D) locations.

3. The method of claim 1, wherein the map is a three-dimensional (3D) virtual reconstruction of the environment.

4. The method of claim 1, wherein the at least one sensor includes one or more cameras fixedly posed on a ceiling of the environment, the method further comprising:

determining respective poses of the one or more cameras;

using the determined respective poses of the one or more cameras to determine the map; and

using the determined respective poses of the one or more cameras to make the determination that the portion is located in the environment at the new location different from the respective location identified by the map for the particular distinct pattern,

wherein determining the respective poses of the plurality of projectors, determining respective poses of the one or more cameras, and determining the map are performed as part of an initial calibration process before making the determination that the portion is located in the environment at the new location.

5. The method of claim 1, wherein the at least one sensor includes a camera coupled to a robotic device configured to travel in the environment, and wherein the sensor data includes at least two sequential images of one or more other portions of the distinct patterns acquired by the camera as the robotic device travels in the environment, the method further comprising:

determining a pose of the camera by performing stereo matching using the at least two sequential images of the one or more other portions; and

using the determined pose of the camera to make the determination that the portion is located in the environment at the new location different from the respective location identified by the map for the particular distinct pattern.

6. The method of claim 1, wherein each distinct pattern includes a respective distinct shape, the method further comprising:

using the respective distinct shape of the particular distinct pattern to identify the particular projector of the plurality that is providing the particular distinct pattern.

7. The method of claim 1, wherein causing the plurality of projectors that are fixedly posed in the environment to each provide, onto the respective area of the environment, the predetermined respective distinct pattern comprises causing the plurality of projectors to each provide the predetermined respective distinct pattern with a respective distinct color, and wherein the at least one sensor is configured to detect the respective distinct colors, the method further comprising:

using the respective distinct color of the particular distinct pattern to identify the particular projector of the plurality that is providing the particular distinct pattern.

8. A computing system comprising:

a plurality of projectors that are fixedly posed in an environment;

at least one processor; and

data storage comprising instructions executable by the at least one processor to cause the computing system to perform operations comprising;

causing the plurality of projectors to each provide, onto a respective area of the environment, a predetermined respective distinct pattern;

determining respective poses of the plurality of projectors;

determining a map of the environment that identifies, for each distinct pattern, respective locations on one or more surfaces in the environment on which the distinct pattern is detectable;

based on sensor data from at least one sensor, identifying a portion of a particular distinct pattern in the environment;

using the map and the determined respective pose of a particular projector of the plurality that is providing the particular distinct pattern to make a determination that the portion is located in the environment at a new location different from the respective location identified by the map for the particular distinct pattern; and

transmitting an output signal indicating the determination that the portion is located in the environment at the new location different from the respective location identified by the map for the particular distinct pattern.

9. The computing system of claim 8, the operations further comprising:

updating the map to identify the new location.

10. The computing system of claim 8, the operations further comprising:

determining a three-dimensional (3D) virtual reconstruction of one or more surfaces in the environment at the new location of the portion and on which the portion is detectable.

11. The computing system of claim 8, wherein each distinct pattern is divided into a plurality of distinct detectable sub-patterns, and

wherein the portion includes at least one distinct sub-pattern of the particular distinct pattern.

12. The computing system of claim 11, wherein each distinct sub-pattern is a distinct QR code.

13. The computing system of claim 8, wherein each distinct pattern is divided into a plurality of distinct detectable sub-patterns, and wherein determining the map comprises, for each distinct pattern:

causing one or more cameras to capture one or more images of each distinct sub-pattern;

determining a two-dimensional (2D) location of each distinct sub-pattern within the respective distinct pattern; and

using triangulation between the one or more images and the determined 2D locations to determine, for each distinct sub-pattern, respective locations on one or more surfaces in the environment on which the distinct sub-pattern is detectable.

14. The computing system of claim 13, wherein the one or more cameras are coupled to a robotic device configured to travel in the environment, and

wherein causing the one or more cameras to capture the one or more images of each distinct sub-pattern comprises causing (i) the robotic device to travel in the environment and (ii) causing the one or more cameras to capture the one or more images of each distinct sub-pattern as the robotic device travels in the environment.

15. A non-transitory computer readable medium having stored thereon instructions that, upon execution by a computing system, cause the computing system to perform operations comprising:

causing a plurality of projectors that are fixedly posed in an environment to each provide, onto a respective area of the environment, a predetermined respective distinct pattern;

determining respective poses of the plurality of projectors;

determining a map of the environment that identifies, for each distinct pattern, respective locations on one or more surfaces in the environment on which the distinct pattern is detectable;

based on sensor data from at least one sensor, identifying a portion of a particular distinct pattern in the environment;

using the map and the determined respective pose of a particular projector of the plurality that is providing the particular distinct pattern to make a determination that the portion is located in the environment at a new location different from the respective location identified by the map for the particular distinct pattern; and

transmitting an output signal indicating the determination that the portion is located in the environment at the new location different from the respective location identified by the map for the particular distinct pattern.

16. The non-transitory computer readable medium of claim 15, wherein the plurality of projectors are fixedly posed on a ceiling of the environment and separated by predetermined distances.

17. The non-transitory computer readable medium of claim 15, wherein transmitting the output signal comprises transmitting, to at least one robotic device, the output signal along with instructions to cause the at least one robotic device to perform an action at the new location.

18. The non-transitory computer readable medium of claim 15, wherein the at least one sensor includes one or more cameras fixedly posed on a ceiling of the environment, the operations further comprising:

determining respective poses of the one or more cameras;

using the determined respective poses of the one or more cameras to determine the map; and

using the determined respective poses of the one or more cameras to make the determination that the portion is located in the environment at the new location different from the respective location identified by the map for the particular distinct pattern,

wherein determining the respective poses of the plurality of projectors, determining respective poses of the one or more cameras, and determining the map are performed as part of an initial calibration process before making the determination that the portion is located in the environment at the new location.

19. The non-transitory computer readable medium of claim 15, wherein the at least one sensor includes a camera coupled to a robotic device configured to travel in the environment, and wherein the sensor data includes at least two sequential images of one or more other portions of the distinct patterns acquired by the camera as the robotic device travels in the environment, the operations further comprising:

determining a pose of the camera by performing stereo matching using the at least two sequential images of the one or more other portions; and

using the determined pose of the camera to make the determination that the portion is located in the environment at the new location different from the respective location identified by the map for the particular distinct pattern.

20. The non-transitory computer readable medium of claim 15, wherein the respective locations on the one or more surfaces in the environment are represented in the map as respective three-dimensional (3D) locations.