Reducing interference between multiple infra-red depth cameras

• US 10,049,458 B2
• Filed: 01/20/2016
• Issued: 08/14/2018
• Est. Priority Date: 01/31/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Izadi, Shahram, Molyneaux, David, Hilliges, Otmar, Kim, David, Shotton, Jamie Daniel Joseph, Hodges, Stephen Edward, Butler, David Alexander, Fitzgibbon, Andrew, Kohli, Pushmeet
• Primary Examiner(s): Vo, Tung
• Assistant Examiner(s): BECKER, JOSEPH W
• Application Number:
  US15/001,645
• Publication Number:
  US 20160163054A1
• Time in Patent Office: 937 Days
• Field of Search: None
• CPC Class Codes: G01B 11/22 for measuring depth
• CPC Class Codes: G01B 9/08 Optical projection comparators
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/20 for generating image signals from infrared radiation only
• CPC Class Codes: H04N 23/80 Camera processing pipelines; Components thereof
• CPC Class Codes: H04N 23/90 Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Systems and methods for reducing interference between multiple infra-red depth cameras are described. In an embodiment, the system comprises multiple infra-red sources, each of which projects a structured light pattern into the environment. A controller is used to control the sources in order to reduce the interference caused by overlapping light patterns. Various methods are described including: cycling between the different sources, where the cycle used may be fixed or may change dynamically based on the scene detected using the cameras; setting the wavelength of each source so that overlapping patterns are at different wavelengths; moving source-camera pairs in independent motion patterns; and adjusting the shape of the projected light patterns to minimize overlap. These methods may also be combined in any way. In another embodiment, the system comprises a single source and a mirror system is used to cast the projected structured light pattern around the environment.

Claims

1. A depth camera system comprising:

a plurality of cameras with at least partially overlapping field of views;

a plurality of sources, each arranged to project a structured light pattern; and

a controller arranged to control a plurality of source-camera pairs to reduce the interference between projected structured light patterns and to receive signals from each of the cameras for use in depth determination, controlling a plurality of source-camera pairs comprising controlling a plurality of sources and a plurality of cameras, the controller being further arranged to control the movement of a source-camera pair relative to other source-camera pairs such that the structured light pattern projected by the source is fixed relative to the camera.

2. A depth camera system according to claim 1, wherein the plurality of cameras comprise infra-red cameras and wherein the plurality of sources comprise infra-red sources, each arranged to project an infra-red structured light pattern.

3. A depth camera system according to claim 1, wherein the controller is arranged to cycle between sources with overlapping structured light patterns.

4. A depth camera system according to claim 1, wherein the controller is arranged to cycle between source-camera pairs with overlapping structured light patterns, the controller being further arranged to adjust the cycle dynamically based on motion detected by at least one source-camera pair.

5. A depth camera system according to claim 3, wherein the controller is arranged to adjust the cycle dynamically based on the signals received from each of the cameras.

6. A depth camera system according to claim 3, wherein the controller is arranged to cycle between sources with overlapping structured light patterns in a first phase of operation and to operate a single source in a second phase of operation.

7. A depth camera system according to claim 3, wherein the controller is arranged to cycle between sources with overlapping structured light patterns in a first phase of operation and to operate all of the sources in a second phase of operation.

8. A depth camera system according to claim 1, further comprising a plurality of depth camera units, each unit comprising at least one source and at least one camera and wherein the controller is arranged to set an operating wavelength of each depth camera unit, such that at least some overlapping structured light patterns are at different operating wavelengths.

9. A depth camera system according to claim 8, wherein each unit comprises at least one wavelength tunable element and wherein the controller is arranged to tune the at least one wavelength tunable element to set an operating wavelength of each depth camera unit.

10. A depth camera system according to claim 8, wherein each unit comprises a plurality of sources with different operating wavelengths and wherein the controller is arranged to select one of the plurality of sources to set an operating wavelength of each depth camera unit.

11. A depth camera system according to claim 8, wherein each unit comprises a plurality of cameras with different operating wavelengths and wherein the controller is arranged to select one of the plurality of cameras to set an operating wavelength of each depth camera unit.

12. A depth camera system according to claim 1, wherein the controller is arranged to move each source-camera pair in a motion pattern which is different from motion patterns of other source-camera pairs.

13. A depth camera system according to claim 12, further comprising a plurality of depth camera units, each unit comprising a source-camera pair and an optical-path movement device arranged to move the source-camera pair under control of the controller.

14. A depth camera system according to claim 1, wherein the controller is arranged to modulate each source-camera pair with a different pseudo-random modulation pattern.

15. A depth camera system according to claim 1, wherein the controller is arranged to control a shape of each projected structured light pattern to reduce overlap of patterns when all of the sources are projecting a structured light pattern simultaneously.

16. A computer-implemented method of controlling a depth camera system, the system comprising a plurality of infra-red cameras and a plurality of infra-red sources, and the method comprising:

controlling a plurality of source-camera pairs to reduce interference between projected infra-red structured light patterns from each source, controlling the plurality of source-camera pairs comprising controlling a plurality of infra-red cameras and a plurality of infra-red sources;

controlling the plurality of source-camera pairs including controlling the movement of a source-camera pair relative to other source-camera pairs such that the infra-red structured light pattern projected by the infra-red source belonging to the pair is fixed relative to the infra-red camera belonging to the pair; and

receiving signals from the plurality of infra-red cameras for use in depth determination.

17. A computer-implemented method according to claim 16, wherein controlling the plurality of infra-red sources comprises cycling between infra-red sources with overlapping structured light patterns.

18. A computer-implemented method according to claim 17, wherein controlling the plurality of infra-red sources further comprises dynamically adjusting a cycle used for cycling between infra-red sources based on a dense 3D model of an environment of the depth camera system which is formed using signals from the plurality of infra-red cameras.

19. A computer-implemented method according to claim 16, wherein the system further comprises a plurality of depth camera units, each unit comprising at least one infra-red source and at least one infra-red camera and wherein controlling the plurality of infra-red sources comprises setting an operating wavelength of each depth camera unit, such that overlapping structured light patterns are at different operating wavelengths.

20. A depth camera system comprising:

a plurality of depth camera units, each unit comprising an optical-path movement device and a source-camera pair, the source-camera pair comprising an infra-red camera and an infra-red source arranged to project an infra-red structured light pattern; and

a computing-based device comprising;

a processor and a memory, wherein the memory is arranged to store device-executable instructions which, when executed, cause the processor to control the optical-path movement device to move the optical path of each source-camera pair in a motion pattern such that movement of the infra-red structured light pattern projected by the source in a source-camera pair is fixed relative to the camera in the source-camera pair, the motion pattern for the depth camera unit being different from the motion pattern of every other depth camera unit in the plurality of depth camera units.