Replicated dot maps for simplified depth computation using machine learning

• US 10,643,341 B2
• Filed: 03/22/2018
• Issued: 05/05/2020
• Est. Priority Date: 03/22/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Bleyer, Michael, Pekelny, Yuri, Price, Raymond Kirk
• Primary Examiner(s): Vu, Kim Y
• Assistant Examiner(s): Bloom, Nathan J
• Application Number:
  US15/928,787
• Publication Number:
  US 20190295269A1
• Time in Patent Office: 775 Days
• Field of Search: None
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01B 11/2545 with one projection direction and several detection directions, e.g. stereo
• CPC Class Codes: G02B 2027/0127 comprising devices increasing the depth of field
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/586 from multiple light sources, e.g. photometric stereo
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: H04N 13/254 in combination with electromagnetic radiation sources for illuminating objects
• CPC Class Codes: H04N 13/366 using viewer tracking

Abstract

Disclosed embodiments include methods and systems for utilizing a structured projection pattern to perform depth detection. In some instances, the structured projection pattern forms a dot pattern, which is projected by an infrared (IR) illuminator, wherein the dot pattern includes a replicated sub-pattern having a predefined height and width. The sub-pattern is replicated in at least one direction such that the dot pattern comprises a plurality of replicated sub-patterns that are adjacently positioned.

Claims

1. A system configured for utilizing a dot pattern to perform depth detection, the system comprising:

one or more processors;

a camera system comprising at least one camera having a corresponding field of view;

a light illuminator comprising a light source and a dot pattern generator that emits the dot pattern generated by the light source, the light illuminator being configured to project a dot pattern on one or more objects in a region within the field of view, the dot pattern comprising a replicated sub-pattern having a predefined height and width, the replicated sub-pattern comprising a plurality of dots arranged such that the replicated sub-pattern omits any two patches that include an identical arrangement of dots, the any two patches having a height that corresponds with the predefined height of the replicated sub-pattern and including at least one dot of the plurality of dots, the replicated sub-pattern being replicated in at least one of a first direction or a second direction that is perpendicular to the first direction, and such that the dot pattern comprises a plurality of replicated sub-patterns that are adjacently positioned when projected by the light illuminator within the field of view; and

one or more computer-readable media having stored computer-executable instructions that are operable, when executed by the one or more processors, to cause the computer system to perform the following;

project the dot pattern with the light illuminator;

obtain a first image of the dot pattern, which is reflected in the region from the projection of the dot pattern, with the camera system; and

perform depth detection based on the dot pattern.

2. The system of claim 1, wherein the dot pattern comprises the replicated sub-pattern replicated in both the first direction and the second direction.

3. The system of claim 1, wherein the camera system includes a first camera that obtains the first image of the dot pattern and a second camera that obtains a second image of the dot pattern and wherein the predefined width of the sub-pattern corresponds to a predetermined maximum disparity search distance over which the system will search for matching pixel patches.

4. The system of claim 3, wherein the computer-executable instructions are further operable to cause the system to perform depth detection with stereo matching between the first image and the second image.

5. The system of claim 3, wherein the computer-executable instructions are further operable to cause the system to determine a first pixel patch which is obtained by the first camera matches a second pixel patch which is obtained by the second camera.

6. The system of claim 5, wherein the determining the first pixel patch matches the second pixel patch is based on determining a similarity of intensity of light detected in the first image patch and the second image patch.

7. The system of claim 6, wherein the first and second image patches have illuminated pixels that span an entire width of overlapping regions of a field of view of the first camera and a field of view of the second camera.

8. The system of claim 3, wherein the computer-executable instructions are further operable to cause the system to:

reproject the first and second images to generate a first rectified image and a second rectified image, respectively, wherein the first rectified image includes a first plurality of scanlines and the second rectified image includes a corresponding second plurality of scanlines, the first and second pluralities of scanlines extending parallel to each other; and

search for a match between a first pixel patch in the first plurality of scanlines and a second pixel patch in the second plurality of scanlines.

9. The system of claim 1, wherein the light illuminator comprises an infrared (IR) light illuminator and wherein the at least one camera comprises an IR light sensing camera.

10. The system of claim 1, wherein the light illuminator comprises a visible light illuminator and wherein the at least one camera comprises a visible light sensing camera.

11. The system of claim 1, wherein the light illuminator comprises an infrared (IR) light illuminator and wherein the at least one camera comprises at least one camera configured for sensing IR light and visible light.

12. The system of claim 11, wherein the light illuminator comprises an infrared (IR) light illuminator and a visible light illuminator and wherein the at least one camera comprises at least one camera configured for sensing IR light and visible light.

13. A method for utilizing a dot pattern to perform depth detection with a camera system, the method comprising:

projecting an infrared (IR) dot pattern into an environment having one or more objects within a field of view of the camera system and that is reflected off of the one or more objects as reflected dot pattern light, the dot pattern comprising a replicated sub-pattern having a predefined height and width, the replicated sub-pattern comprising a plurality of dots arranged such that the replicated sub-pattern omits any two patches that include an identical arrangement of dots, the any two patches having a height that corresponds with the predefined height of the replicated sub-pattern and including at least one dot of the plurality of dots, the replicated sub-pattern being replicated in at least one of a first direction or a second direction that is perpendicular to the first direction, and such that the dot pattern comprises a plurality of replicated sub-patterns that are adjacently positioned within the field of view;

obtaining a first image of the reflected dot pattern light with a camera system;

using the first image of the dot pattern light to perform at least depth detection.

14. The method of claim 13, wherein the dot pattern comprises the replicated sub-pattern replicated in both the first direction and the second direction.

15. The method of claim 13, wherein the camera system includes a first camera that obtains the first image of the reflected dot pattern light and a second camera that obtains a second image of the reflected dot pattern light and wherein the predefined width of the sub-pattern corresponds to a predetermined maximum disparity search distance over which the system will search for matching pixel patches.

16. The method of claim 15, wherein the method further includes performing stereo matching between the first image and the second image to further perform depth detection.

17. The method of claim 15, wherein the method further includes determining a first pixel patch matches a second pixel patch.

18. The method of claim 17, wherein the determining the first pixel patch matches the second pixel patch is based on determining a similarity of dot pattern of light detected in the first image patch and the second image patch.

19. The method of claim 18, wherein the first and second image patches have illuminated pixels that span an entire height that corresponds to the predefined height of the linear section.

20. The method of claim 15, wherein the method further comprises:

reprojecting the first and second images to generate a first rectified image and a second rectified image, respectively, wherein the first rectified image includes a first plurality of scanlines and the second rectified image includes a corresponding second plurality of scanlines, the first and second pluralities of scanlines extending parallel to each other; and

searching for a match between a first pixel patch in the first plurality of scanlines and a second pixel patch in the second plurality of scanlines.