AUTOMATIC MULTIPLE DEPTH CAMERAS SYNCHRONIZATION USING TIME SHARING

• US 20150373322A1
• Filed: 01/16/2015
• Published: 12/24/2015
• Est. Priority Date: 06/20/2014
• Status: Active Grant

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Goma, Sergiu Radu, Atanassov, Kalin Mitkov
• Granted Patent: US 10,419,703 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G01B 11/245 using a plurality of fixed, simultaneously operating transducers G01B11/2408 - G01B11/2425, G01B11/255 take precedence
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01S 17/10 using transmission of interrupted, pulse-modulated waves determination of distance by phase measurements G01S17/32
• CPC Class Codes: G01S 17/87 Combinations of systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/4814 of transmitters alone
• CPC Class Codes: G01S 7/483 Details of pulse systems
• CPC Class Codes: G03B 17/38 Releasing-devices separate from shutter integral with shutter G03B9/08
• CPC Class Codes: G03B 2206/00 Systems for exchange of information between different pieces of apparatus, e.g. for exchanging trimming information, for photo finishing
• CPC Class Codes: G03B 35/00 Stereoscopic photography
• CPC Class Codes: H04N 13/271 wherein the generated image signals comprise depth maps or disparity maps
• CPC Class Codes: H04N 23/21 from near infrared [NIR] radiation
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20
• CPC Class Codes: H04N 5/40 Modulation circuits
• CPC Class Codes: H04N 5/44 Receiver circuitry for the reception of television signals according to analogue transmission standards H04N5/14 takes precedence

Abstract

Aspects relate to an depth sensing system for capturing an image containing depth information of an object. In one embodiment, a depth sensing device for use in conjunction with multiple depth sensing devices for capturing an image containing depth information of an object comprises a near-infrared transmitter comprising a laser capable of producing a near infra-red light beam, a diffractive optical element positioned to receive a light beam emitted from the laser, the diffractive optical element, and a collimating lens, and a near-infrared receiver coupled to the transmitter in a relative position, the receiver comprising a sensor assembly capable of producing an image of the received light, the depth sensing device being configured to transmit and receive near infra-red light beams during a time period that is different than any of the other of two or more transmitter-receiver pairs of devices in communication with the depth sensing device.

Claims

1. A depth sensing device for capturing an image containing depth information of a scene and for use in a depth sensing system that includes at least one other depth sensing device, comprising:

a transmitter capable of projecting light on a scene, the transmitter comprising a laser capable of producing a light beam including a series of laser pulses, each pulse having a pulse length and the series of pulses produced at a pulse frequency, and

a receiver coupled to the transmitter at a known relative orientation, the receiver comprising a shutter and a sensor assembly capable of producing an image based on sensing light projected by the transmitter and reflected from the scene; and

a controller comprising a processor, the controller coupled to the transmitter and the receiver, the controller configured to

determine the presence of light in the scene using the receiver,

control the pulse length of the series of laser pulses,

control when an exposure window begins during which the transmitter is activated to project light on the scene and activate the receiver to begin sensing light reflected from the scene, the beginning of the exposure window and the pulse length controlled based on the determined presence of the light on the scene from the at least one other depth sensing device so the exposure window is temporally different from when any other of the at least one other depth sensing devices in the system is illuminating the scene.

2. The depth sensing device of claim 1, wherein

the laser is capable of producing a near infrared (NIR) light beam,

the transmitter further comprises an optical element positioned to receive the NIR light beam emitted from the laser, the optical element including a plurality of features configured to produce a known structured light pattern when the NIR light beam propagates through the optical element, and

the controller is further configured to determine the presence of NIR light indicative of a structured light pattern in the scene using the NIR receiver.

3. The depth sensing system of claim 1, wherein the plurality of features includes a plurality of diffractive optical features.

4. The depth sensing device of claim 1, wherein the transmitter comprises a time-of-flight (TOF) transmitter.

5. The depth sensing device of claim 2, wherein the controller is further configured to activate the transmitter to project a structured light pattern on the scene and adjust the shutter to be synchronized with the pulse length of the NIR light beam if the controller determines there is no NIR light indicative of a structured light pattern present on the scene.

6. The depth sensing device of claim 5, wherein the controller is further configured to delay the transmitter from producing a NIR light beam for a delay period if the controller determines the presence of NIR light indicative of a structured light pattern on the scene, and at the end of the delay period check again for the presence of NIR light indicative of a structured light pattern on the scene.

7. The depth sensing system of claim 1, wherein the shutter comprises a rolling shutter, and wherein during the exposure time the controller activates the rolling shutter to scan across the scene when the scene is illuminated by light projected by the transmitter.

8. The depth sensing system of claim 1, wherein the controller is configured to determine an exposure window without communicating with the at least one other device or another synchronization system.

9. The depth sensing system of claim 1, wherein the controller is further configured to adjust the pulse frequency based on the determined presence of light in the scene so the that the transmitter projects light onto the scene during an exposure window that is temporally different than when any other of the at least one depth sensing device is illuminating the scene.

10. A method operative on a depth sensing device for capturing an image containing depth information of a scene in a depth sensing system that includes at least two depth sensing devices, the method comprising:

detecting light reflected from a scene using a sensor, the light indicative of the scene being illuminated by a depth sensing device;

if light from the scene is not detected, activating a transmitter of the depth sensing device to project light on the scene during an exposure window, and activating a shutter of the depth sensing device to capture information from the reflection of the light during the exposure window, the projected light including a series of pulses having a pulse length and a pulse frequency;

if light from the scene is detected,

adjusting a shutter of the depth sensing device to a detected laser pulse length, and again sensing light from the scene;

if light is again detected, delaying the start of an exposure window during which the transmitter of the depth sensing device projects light on the scene the shutter of the depth sensing device captures information from the reflection of the light from the scene, and iteratively repeating sensing if light is present on the scene and delaying the start of an exposure window until light is not detected; and

if light from the scene is not detected, activating the transmitter and the receiver for the exposure window, projecting light on the scene in a series of pulses, the series of pulses having a pulse frequency and each pulse having a pulse length, and detecting light projected light from the scene using the receiver.

11. The method of claim 10, wherein

the transmitter produces a near infrared (NIR) light beam,

the transmitter comprises an optical element positioned to receive the NIR light beam emitted from the laser, the optical element including a plurality of features configured to produce a known structured light pattern when the NIR light beam propagates through the optical element, and

wherein detecting light form the scene comprises detecting the presence of NIR light indicative of a structured light pattern in the scene using the receiver.

12. The method of claim 11, wherein the plurality of features includes a plurality of diffractive optical features.

13. The method of claim 10, wherein the transmitter comprises a time-of-flight (TOF) transmitter.

14. The method of claim 10, wherein the shutter comprises a rolling shutter, and wherein during the exposure window the controller activates the rolling shutter to scan across the scene when the scene is illuminated by light projected by the transmitter.

15. The method of claim 10, wherein a controller determines the exposure window without communicating with the at least one other device or another synchronization system.

16. The method claim 10, further comprising adjusting the pulse frequency based on the determined presence of light in the scene so the that the transmitter projects light onto the scene during an exposure window that is temporally different than when any other of the at least one depth sensing device is illuminating the scene.

17. A depth sensing device for capturing an image containing depth information of a scene and for use in a depth sensing system that includes at least one other depth sensing device, comprising:

means for projecting light on a scene, the light projecting means configured to produce a laser light beam including a series of laser pulses, each pulse having a pulse length and the series of pulses produced at a pulse frequency, and

means for receiving light coupled to the projecting means at a known relative orientation, the light receiving means configured to produce an image based on detecting light projected by the light projecting means and reflected from the scene; and

means for controlling coupled to the projecting means and the light receiving means, the controlling means configured to

determine the presence of light in the scene using the light receiving means,

control the pulse length of the series of laser pulses,

control when an exposure window begins during which the light projecting means is activated to project light on the scene and activate the light receiving means to begin sensing light reflected from the scene, the beginning of the exposure window and the pulse length controlled based on the determined presence of the light on the scene from the at least one other depth sensing device so the exposure window is temporally different from when any other of the at least one other depth sensing devices in the system is illuminating the scene.

18. The depth sensing device of claim 17, wherein the light projecting means comprises a transmitter including a laser.

19. The depth sensing device of claim 17, wherein the light receiving means comprises a receiver including a shutter and a sensor assembly capable of producing an image based on sensing light projected by the light projecting means and reflected from the scene.

20. The depth sensing device of claim 17, wherein the controlling means comprises comprising at least one processor.

21. The depth sensing system of claim 18,

wherein the laser is capable of emitting a near infrared (NIR) light beam,

wherein the light projecting means comprises a transmitter comprising an optical element positioned to be illuminated by the NIR light beam emitted from the laser, the optical element including a plurality of features configured to produce a known structured light pattern when the NIR light beam propagates through the optical element, and

wherein the controlling means comprises a controller configured to determine the presence of NIR light indicative of a structured light pattern in the scene using the NIR receiver.

22. The depth sensing system of claim 21, wherein the plurality of features includes a plurality of diffractive optical features.

23. The depth sensing device of claim 21, wherein the transmitter comprises a time-of-flight (TOF) transmitter.

24. The depth sensing device of claim 21, wherein the controller is further configured to activate the transmitter to project a structured light pattern on the scene and adjust the shutter to be synchronized with the pulse length of the NIR light beam if the controller determines there is no NIR light indicative of a structured light pattern present on the scene.

25. The depth sensing device of claim 24, wherein the controller is further configured to delay the transmitter from producing a NIR light beam for a delay period if the controller determines the presence of NIR light indicative of a structured light pattern on the scene, and at the end of the delay period check again for the presence of NIR light indicative of a structured light pattern on the scene.

26. The depth sensing system of claim 25, wherein the means for receiving comprises a rolling shutter, and wherein during the exposure time the controller activates the rolling shutter to scan across the scene when the scene is illuminated by light projected by the transmitter.

27. The depth sensing system of claim 17, wherein the controlling means is configured to determine an exposure window without communicating with the at least one other device or another synchronization system.

28. The depth sensing system of claim 27, wherein the controller is further configured to adjust the pulse frequency based on the determined presence of light in the scene so the that the transmitter projects light onto the scene during an exposure window that is temporally different than when any other of the at least one depth sensing device is illuminating the scene.

29. A computer readable medium containing non-transient instructions that control at least one processor to execute the instructions, the method comprising:

detecting light reflected from a scene using a sensor, the light indicative of the scene being illuminated by a depth sensing device;

if light from the scene is not detected, activating a transmitter of the depth sensing device to project light on the scene during an exposure window, and activating a shutter of the depth sensing device to capture information from the reflection of the light during the exposure time, the projected light including a series of pulses having a pulse length and a pulse frequency;

if light from the scene is detected,

adjusting a shutter of the depth sensing device to a detected laser pulse length, and again sensing light from the scene;

if light is again detected, delaying the start of an exposure window during which the transmitter of the depth sensing device projects light on the scene the shutter of the depth sensing device captures information from the reflection of the light from the scene, and iteratively repeating sensing if light is present on the scene and delaying the start of an exposure window until light is not detected; and

if light from the scene is not detected, activating the transmitter and the receiver for the exposure window, projecting light on the scene in a series of pulses, the series of pulses having a pulse frequency and each pulse having a pulse length, and detecting light projected light from the scene using the receiver.

30. The computer readable medium of claim 29, wherein the transmitter produces a near infrared (NIR) light beam

the transmitter comprises an optical element positioned to receive the NIR light beam emitted from the laser, the optical element including a plurality of features configured to produce a known structured light pattern when the NIR light beam propagates through the optical element, and

wherein detecting light from the scene comprises detecting the presence of NIR light indicative of a structured light pattern in the scene using the receiver.