Integrated low power depth camera and projection device

• US 8,681,255 B2
• Filed: 09/28/2010
• Issued: 03/25/2014
• Est. Priority Date: 09/28/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Katz, Sagi, Felzenshtein, Shlomo, Adler, Avishai, Yahav, Giora
• Primary Examiner(s): KHAN, USMAN A
• Application Number:
  US12/892,589
• Publication Number:
  US 20120075534A1
• Time in Patent Office: 1,274 Days
• Field of Search: 348/333.01
• US Class Current: 348/333.01
• CPC Class Codes: G01S 17/18 wherein range gates are used
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: H04N 9/3182 Colour adjustment, e.g. white balance, shading or gamut white balance per se H04N9/73; control of amplitude of colour signals H04N9/68; colour control circuits for displays in general G09G5/02, G09G3/2003, G09G3/3607
• CPC Class Codes: H04N 9/3194 including sensor feedback

Abstract

A video projector device includes a visible light projector to project an image on a surface or object, and a visible light sensor, which can be used to obtain depth data regarding the object using a time-of-flight principle. The sensor can be a charge-coupled device which obtains color images as well as obtaining depth data. The projected light can be provided in successive frames. A frame can include a gated sub-frame of pulsed light followed by continuous light, while the sensor is gated, to obtain time of flight data, an ungated sub-frame of pulsed light followed by continuous light, while the sensor is ungated, to obtain reflectivity data and a background sub-frame of no light followed by continuous light, while the sensor is gated, to determine a level of background light. A color sub-frame projects continuous light, while the sensor is active.

Claims

1. A video projector device, comprising:

a light source;

a driver which drives the light source;

an optical component which projects visible light from the light source in a field of view;

a sensor which senses visible light, the sensor comprises a plurality of pixels; and

at least one control circuit associated with the driver and the sensor, the at least one control circuit;

to project a frame of pixel data on an object in a frame period, divides the frame period into multiple sub-frames including a gated sub-frame, and causes the video projector device to operate in a projection and sensing mode followed by a projection-only mode, without sensing, during the gated sub-frame,

to provide the projection and sensing mode;

causes the driver to drive the light source in a pulsed mode in which the light source emits pulses of visible light while causing the sensor to operate in a gated mode so that the plurality of pixels integrate charge from reflections of the pulses of visible light from the object,

to provide the projection-only mode, without sensing;

causes the driver to drive the light source in a continuous mode in which the light source emits continuous visible light while reading out an integrated charge from the plurality of pixels to obtain light intensity values, and

derives depth data regarding the object in the field of view based on the light intensity values, using a time-of-flight principle.

2. The video projector device of claim 1, wherein:

the multiple sub-frames comprise an ungated sub-frame following the gated sub-frame; and

the at least one control circuit provides the ungated sub-frame by causing the driver to drive the light source in the pulsed mode while the sensor is operated in an ungated mode, followed by causing the driver to drive the light source in the continuous mode while the sensor is read to obtain light intensity values, where the at least one control circuit adjusts the light intensity values obtained during the gated sub-frame based on the light intensity values read from the sensor and obtained during the ungated sub-frame, to compensate for reflectivity of the object in the field of view.

3. The video projector device of claim 1, wherein:

the multiple sub-frames comprise a background sub-frame following the gated sub-frame; and

the at least one control circuit provides the background sub-frame, in which no visible light is emitted by the video projector device toward the object while the sensor is operated in a gated mode to sense background light, followed by causing the driver to drive the light source in the continuous mode while the sensor is read to obtain light intensity values, where the at least one control circuit adjusts the light intensity values obtained during the gated sub-frame based on the light intensity values read from the sensor and obtained during the background sub-frame, to compensate for the background light in the field of view.

4. The video projector device of claim 3, wherein:

the at least one control circuit provides two of the background sub-frames in the multiple sub-frames.

5. The video projector device of claim 1, wherein:

the multiple sub-frames comprise a color sub-frame following the gated sub-frame; and

the at least one control circuit provides the color sub-frame by causing the driver to drive the light source in the continuous mode while the sensor is operated in a standard color sensing mode, followed by causing the driver to drive the light source in the continuous mode while the sensor is read to obtain light intensity values which include color data from the field of view.

6. The video projector device of claim 1, wherein:

the multiple sub-frames comprise a color sub-frame in which the at least one control circuit causes the driver to drive the light source in the continuous mode while the sensor is active.

7. The video projector device of claim 1, wherein:

the at least one control circuit causes the driver to drive the light source using triangular shaped pulses during the gated sub-frame.

8. The video projector device of claim 1, wherein:

the at least one control circuit causes the driver to drive the light source using pulses which each have at least one of;

(a) a leading edge which transitions higher at a rate which is below a step up rate of the light source, or (b) a trailing edge which transitions lower at a rate which is below a step down rate of the light source.

9. The video projector device of claim 1, wherein:

the light source is a backlight of the video projector device.

10. The video projector device of claim 1, wherein:

the light source, the optical component, the at least one control circuit and the sensor are provided in a common housing.

11. The video projector device of claim 1, further comprising:

at least one light-transmissive LCD panel which encodes video information onto the visible light, the at least one light-transmissive LCD panel comprises a plurality of pixels.

12. The video projector device of claim 1, wherein:

the at least one control circuit focuses the optical component based on the depth data.

13. The video projector device of claim 1, wherein:

the pulses of visible light are provided during a limited period, relative to a period in which the continuous visible light is provided, so that image quality of an image projected on the object is not noticeably reduced.

14. A video projector device, comprising:

a light source;

a driver which drives the light source;

an optical component which projects visible light from the light source in a field of view;

a sensor which senses visible light, including visible light which is reflected from an object in the field of view, the sensor comprises a plurality of pixels; and

at least one control circuit which alternately causes a depth-sensing frame and a non-depth-sensing frame to be projected onto the object, the at least one control circuit;

(a) to provide the depth-sensing frame;

causes the driver to drive the light source in a pulsed mode in which the light source emits pulses of visible light during one portion of the depth-sensing frame,

causes the sensor to operate in a gated mode during the one portion of the depth-sensing frame to detect the pulses of visible light,

derives depth data regarding the object in the field of view based on light intensity values which are read from the sensor, using a time-of-flight principle, and

causes the driver to drive the light source in a continuous mode in which the light source emits continuous visible light during another portion of the depth-sensing frame, and

(b) at a different time than the depth-sensing frame, to provide the non-depth-sensing frame;

causes the driver to drive the light source in a continuous mode in which the light source emits continuous visible light, and

does not derive depth data regarding the object in the field of view.

15. The video projector device of claim 14, wherein:

the at least one control circuit causes the light source to emit visible light at a higher illumination power during the depth-sensing frame than during the non-depth-sensing frame.

16. The video projector device of claim 14, wherein:

the depth-sensing frame is longer than the non-depth-sensing frame.

17. The video projector device of claim 14, wherein:

the at least one control circuit is controllable to selectively operate in a first mode in which the at least one control circuit uses a limited number of one or more frames to obtain depth data regarding the object in the field of view, and in a second mode in which the at least one control circuit continuously obtains depth data regarding object in the field of view.