Combination of stereo and structured-light processing

• US 9,507,995 B2
• Filed: 12/29/2014
• Issued: 11/29/2016
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: X Development LLC (Alphabet Inc.)
• Inventors: Konolige, Kurt, Rublee, Ethan
• Primary Examiner(s): AHMED, SAMIR ANWAR
• Application Number:
  US14/584,016
• Publication Number:
  US 20160063309A1
• Time in Patent Office: 701 Days
• Field of Search: 382/154, 345419-427, 348/42, 356/12
• US Class Current: 1/1
• CPC Class Codes: G06T 1/0014 Image feed-back for automatic industrial control, e.g. robot with camera robots B25J19/023
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30108 Industrial image inspection
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06V 20/64 Three-dimensional objects

Abstract

Methods and systems for determining depth information using a combination of stereo and structured-light processing are provided. An example method involves receiving a plurality of images captured with at least two optical sensors, and determining a first depth estimate for at least one surface based on corresponding features between a first image and a second image. Further, the method involves causing a texture projector to project a known texture pattern, and determining, based on the first depth estimate, at least one region of at least one image of the plurality of images within which to search for a particular portion of the known texture pattern. And the method involves determining points corresponding to the particular portion of the known texture pattern within the at least one region, and determining a second depth estimate for the at least one surface based on the determined points corresponding to the known texture pattern.

Claims

1. A method comprising:

causing, by a computing device, a texture projector to project a known texture pattern onto an environment;

receiving, by the computing device, a plurality of images captured with at least two optical sensors, wherein the plurality of images comprises a first image of the environment as perceived from a first viewpoint of a first optical sensor and a second image of the environment as perceived from a second viewpoint of a second optical sensor;

determining, by the computing device, a first depth estimate for at least one surface in the environment based on corresponding features between the first image and the second image;

determining, by the computing device and based on the first depth estimate, at least one region of at least one image of the plurality of images within which to search for a particular portion of the known texture pattern, wherein the at least one image is indicative of the environment and the known texture pattern projected onto the environment;

determining, by the computing device, points corresponding to the particular portion of the known texture pattern within the at least one region of the at least one image; and

determining, by the computing device, a second depth estimate for the at least one surface in the environment based on the determined points corresponding to the particular portion of the known texture pattern within the at least one region of the at least one image.

2. The method of claim 1, wherein determining, based on the first depth estimate, the at least one region of the at least one image within which to search for the particular portion of the known texture pattern comprises determining the at least one region using triangulation based on the first depth estimate and based on a position of the texture projector with respect to a position of an optical sensor used to capture the at least one image, wherein the optical sensor is the first optical sensor, the second optical sensor, or a third optical sensor.

3. The method of claim 1:

wherein the at least one image comprises the first image, the second image, or both.

4. The method of claim 1, further comprising causing the texture projector to project a random texture pattern onto the environment such that the first image of the environment and the second image of the environment are indicative of the environment and the random texture pattern projected onto the environment.

5. The method of claim 4, wherein the at least one image comprises a third image of the plurality of images.

6. The method of claim 1, wherein the at least two optical sensors comprises the first optical sensor, the second optical sensor, and a third optical sensor and wherein the at least one image is captured with the third optical sensor.

7. The method of claim 1, further comprising determining a size of the known texture pattern based on a size of the at least one region of the at least one image.

8. The method of claim 1:

wherein the first image of the environment and the second image of the environment are indicative of the environment and the known texture pattern projected onto the environment;

wherein the at least one image comprises the first image and the second image;

wherein the at least one region of the at least one image comprises a first region of the first image and a second region of the second image; and

wherein determining points corresponding to the particular portion of the known texture pattern within the at least one region of the at least one image comprises;

(1) determining a first set of points corresponding to the particular portion of the known texture pattern within the first region of the first image, and (2) determining a second set of points corresponding to the particular portion of the known texture pattern within the second region of the second image.

9. The method of claim 8, wherein determining the second depth estimate for the at least one surface in the environment comprises:

(1) determining a first structured-light depth estimate based on the first set of points, (2) determining a second structured-light depth estimate based on the second set of points, and (3) determining a second depth estimate that combines the first structured-light depth estimate and the second structured-light depth estimate.

10. The method of claim 1, wherein the first optical sensor, the second optical sensor, and the texture projector are coupled to a robotic manipulator.

11. The method of claim 1, further comprising determining a third depth estimate by averaging the first depth estimate and the second depth estimate.

12. A non-transitory computer-readable medium having stored therein instructions, that when executed by one or more processors of a computing device, cause the computing device to perform functions comprising:

causing a texture projector to project a known texture pattern onto an environment;

receiving a plurality of images captured with at least two optical sensors, wherein the plurality of images comprises a first image of the environment as perceived from a first viewpoint of a first optical sensor and a second image of the environment as perceived from a second viewpoint of a second optical sensor;

determining a first depth estimate for at least one surface in the environment based on corresponding features between the first image and the second image;

determining, based on the first depth estimate, at least one region of at least one image of the plurality of images within which to search for a particular portion of the known texture pattern, wherein the at least one image is indicative of the environment and the known texture pattern projected onto the environment;

determining points corresponding to the particular portion of the known texture pattern within the at least one region of the at least one image; and

determining a second depth estimate for the at least one surface in the environment based on the determined points corresponding to the particular portion of the known texture pattern within the at least one region of the at least one image.

13. The non-transitory computer-readable medium of claim 12, wherein determining, based on the first depth estimate, the at least one region of the at least one image within which to search for the particular portion of the known texture pattern comprises determining the at least one region using triangulation based on the first depth estimate and based on a position of the texture projector with respect to a position of an optical sensor used to capture the at least one image, wherein the optical sensor is the first optical sensor, the second optical sensor, or a third optical sensor.

14. The non-transitory computer-readable medium of claim 12:

wherein the at least one image comprises the first image, the second image, or both.

15. The non-transitory computer-readable medium of claim 12, wherein the functions further comprise determining a size of the known texture pattern based on a size of the at least one region of the at least one image.

16. A system comprising:

at least two optical sensors;

a texture projector configured to project a known texture pattern onto an environment; and

a computing device configured to;

cause the texture projector to project a known texture pattern onto the environment,

receive a plurality of images captured with the at least two optical sensors, wherein the plurality of images comprises a first image of the environment as perceived from a first viewpoint of a first optical sensor and a second image of the environment as perceived from a second viewpoint of a second optical sensor,

determine a first depth estimate for at least one surface in the environment based on corresponding features between the first image and the second image,

determine, based on the first depth estimate, at least one region of at least one image of the plurality of images within which to search for a particular portion of the known texture pattern, wherein the at least one image is indicative of the environment and the known texture pattern projected onto the environment,

determine points corresponding to the particular portion of the known texture pattern within the at least one region of the at least one image, and

determine a second depth estimate for the at least one surface in the environment based on the determined points corresponding to the particular portion of the known texture pattern within the at least one region of the at least one image.

17. The system of claim 16:

wherein the at least one image comprises the first image, the second image, or both.

18. The system of claim 16, further comprising a robotic manipulator, wherein the texture projector is coupled to the robotic manipulator.

19. The system of claim 18, wherein the at least two optical sensors are coupled to the robotic manipulator.

20. The system of claim 16, wherein the computing device is further configured to determine a size of the known texture pattern based on a size of the at least one region of the at least one image.