Single chip red, green, blue, distance (RGB-Z) sensor

• US 8,139,141 B2
• Filed: 01/26/2005
• Issued: 03/20/2012
• Est. Priority Date: 01/28/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Bamji, Cyrus, Zhao, Peiqian
• Primary Examiner(s): Ometz, David
• Assistant Examiner(s): DURNFORD GESZVAIN, DILLON
• Application Number:
  US11/044,996
• Publication Number:
  US 20050285966A1
• Time in Patent Office: 2,610 Days
• Field of Search: 348/274, 348/275, 348/336, 348/337, 348/348, 348/349, 348/350, 356 501- 508
• US Class Current: 348/348
• CPC Class Codes: G01C 3/08 Use of electric radiation detectors
• CPC Class Codes: G01J 3/2803 using photoelectric array detector
• CPC Class Codes: G01J 3/36 Investigating two or more bands of a spectrum by separate detectors
• CPC Class Codes: G01S 17/86 Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/894 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 17/931 of land vehicles
• CPC Class Codes: G01S 7/4811 common to transmitter and receiver
• CPC Class Codes: G01S 7/4863 Detector arrays, e.g. charge-transfer gates
• CPC Class Codes: G01S 7/4912 Receivers
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 5/2226 Determination of depth image, e.g. for foreground/background separation determining depth by image analysis in general G06T7/50; segmentation by image analysis in general G06T7/10
• CPC Class Codes: H04N 9/03 Circuitry for demodulating colour component signals modulated spatially by colour striped filters by frequency separation

Abstract

An RGB-Z sensor is implementable on a single IC chip. A beam splitter such as a hot mirror receives and separates incoming first and second spectral band optical energy from a target object into preferably RGB image components and preferably NIR Z components. The RGB image and Z components are detected by respective RGB and NIR pixel detector array regions, which output respective image data and Z data. The pixel size and array resolutions of these regions need not be equal, and both array regions may be formed on a common IC chip. A display using the image data can be augmented with Z data to help recognize a target object. The resultant structure combines optical efficiency of beam splitting with the simplicity of a single IC chip implementation. A method of using the single chip red, green, blue, distance (RGB-Z) sensor is also disclosed.

Claims

1. A CMOS-implementable sensor useable with a time-of-flight (TOF) system that includes an emitter that emits optical energy in an emitter optical path toward a target object, emitted said optical energy including a second spectral band comprising non-visible wavelengths, a fraction of which emitted said optical energy is reflected from said target object a distance Z away from said TOF system, said sensor detecting target object reflected optical energy components, including optical energy in a first spectral band comprising visible wavelengths to image said target object, and target object reflected optical energy components emitted by said emitter in said second spectral band to determine at least magnitude of said Z, said TOF system acquiring magnitude of said distance Z without a shutter, the sensor comprising:

an optical splitter disposed with respect to said emitter optical path to receive, incoming to said TOF system, reflected optical energy components in said first spectral band including red, green, and blue components, and target object reflected optical energy components in said second spectral band, and to substantially pass optical energy components in said first spectral band and to substantially reflect optical energy components in the said second spectral band, said optical splitter so disposed thus receiving incoming optical energy reflected from external to said TOF system without receiving optical energy emitted by said emitter that has not been so reflected;

an image sensor disposed to detect components from said first spectral band output from said optical splitter, optical energy components in said first spectral band including red, green, and blue components and being detected by sensors disposed in a single plane relative to one another, and to output display data useable to display at least a portion of said target object;

a Z sensor disposed to detect components from said second spectral band output by said optical splitter and to output Z data useable to determine at least magnitude of said Z;

said image sensor and said Z sensor are formed on a common integrated circuit (IC) substrate, a plane of said IC substrate being parallel to an optic axis defined by components of said first spectral band and said second spectral band input to said optical splitter so as to define a common boresight.

2. The sensor of claim 1, wherein said optical splitter is selected from a group consisting of (a) a hot mirror, (b) a cold mirror, (c) a half mirror, and (d) a fully reflective mirror.

3. The sensor of claim 1, wherein said first spectral band comprises at least one group of wavelengths selected from a group consisting of (i) red, green, blue (RGB) wavelengths, and (ii) wavelengths in a range of 400 nm to 650 nm, and wherein said second spectral band comprises at least one group of wavelengths selected from a group consisting of (i) IR wavelengths, (ii) near IR (NIR) wavelengths, and (iii) a wavelength of 800 nm.

4. The sensor of claim 1, wherein:

said image sensor includes a first plurality of red, green, blue (RGB) pixel detector diodes formed on an IC substrate and defining an RGB detection resolution;

said Z sensor includes a second plurality of Z pixel detector diodes formed on said IC substrate and defining a Z detection resolution different from said RGB detection resolution.

5. The sensor of claim 4, wherein a cross-sectional area defined by one of said Z pixel detector diodes is substantially larger than a cross-sectional area defined by one of said RGB pixel detector diodes.

6. The sensor of claim 1, further including means for substantially equalizing any discrepancy in optical path length within said sensor between a first optical path defined by components of said first spectral band and defined by a second optical path defined by said components of said second spectral band.

7. The sensor of claim 6, wherein said means for substantially equalizing includes an element having an index of refraction n≧

1.2, disposed in at least one of said first optical path and said second optical path.

8. The sensor of claim 1, further including means for displaying an image from an output of said image sensor.

9. The sensor of claim 1, further including means for determining, from an output of said Z sensor, at least one of (a) magnitude of said distance Z, (b) magnitude of Δ

Z/Δ

t of said target object, (c) approximate shape of at least a portion of said target object, and (d) approximate size of at least a portion of said target object.

10. The sensor of claim 1, further including:

means for displaying an image from an output of said image sensor;

means for determining, from an output of said Z sensor, at least one of (a) magnitude of said distance Z, (b) magnitude of Δ

Z/Δ

t of said target object, (c) approximate shape of at least a portion of said target object, and (d) approximate size of at least a portion of said target object; and

means, coupled to an output of said means for determining, for augmenting information displayed on said image.

11. The sensor of claim 10, wherein said means for augmenting includes at least one augmentation selected from a group consisting of (a) visually highlighting a portion of said image, (b) displaying an image of a warning on said display, (c) sounding an acoustic alarm, (d) outputting a signal useable to activate a safety system in a motor vehicle containing said sensor.

12. A CMOS-implementable sensor useable with a time-of-flight (TOF) system that includes an emitter that emits optical energy in an emitter optical path toward a target object, emitted said optical energy including a second spectral band comprising non-visible wavelengths, a fraction of which emitted said optical energy is reflected from said target object a distance Z away from said TOF system, said sensor detecting target object reflected optical energy components, including optical energy in a first spectral band comprising visible wavelengths an optical splitter disposed with respect to said emitter optical path to receive, incoming to said TOF system, reflected optical energy components in said first spectral band including, red, green, and blue components, to image said target object, and target object reflected optical energy components emitted by said emitter in said second spectral band to determine at least magnitude of said Z, said TOF system acquiring magnitude of said distance Z without a shutter, the sensor comprising:

an optical splitter disposed with respect to said emitter optical path to receive, incoming to said TOF system, reflected optical energy components in said first spectral band and target object reflected optical energy components in said second spectral band, and to substantially pass optical energy components from one said spectral band and to substantially reflect optical energy components from the other said spectral band, said optical splitter thus receiving incoming optical energy reflected from external to said TOF system without receiving optical energy emitted by said emitter that has not been so reflected;

an image sensor comprising image pixel detector diodes disposed on an integrated circuit (IC) substrate to detect components in said first spectral band that are output from said optical splitter, optical energy components in said first spectral band being detected by sensors disposed in a single plane relative to one another, and to output display data useable to display at least a portion of said target object; and

a Z sensor comprising Z pixel detector diodes disposed on said IC substrate to detect components in said second spectral band that are output from said optical splitter and to output Z data useable to determine at least magnitude of said Z, said Z pixel detector diodes are interspersed among said image pixel detector diodes;

wherein a plane of said IC substrate is parallel to an optic axis defined by components of said first spectral band and said second spectral band.

13. The sensor of claim 12, wherein said optical splitter includes at least one splitter selected from a group consisting of a hot mirror, a cold mirror, a half mirror, and a fully reflective mirror.

14. The sensor of claim 12, wherein said first spectral band comprises at least one group of wavelengths selected from a group consisting of (i) red, green, blue (RGB) wavelengths, and (ii) wavelengths in a range of 400 nm to 650 nm, and wherein said second spectral band comprises at least one group of wavelengths selected from a group consisting of (i) IR wavelengths, (ii) near IR (NIR) wavelengths, (iii), and a wavelength of 800 nm.

15. The sensor of claim 12, further including:

means for displaying an image from an output of said image sensor;

means for determining, from an output of said Z sensor, at least one of (a) magnitude of said distance Z, (b) magnitude of Δ

Z/Δ

t of said target object, (c) approximate shape of at least a portion of said target object, and (d) approximate size of at least a portion of said target object; and

means, coupled to an output of said means for determining, for augmenting information displayed on said image.

16. A CMOS-implementable sensor useable with a-time-of-flight (TOF) system that includes an emitter that emits optical energy in an emitter optical path toward a target object, emitted said optical energy including a second spectral band comprising non-visible wavelengths, a fraction of which emitted said optical energy is reflected from said target object a distance Z away from said TOF system, said sensor detecting target object reflected optical energy components, including optical energy in a first spectral band comprising visible wavelengths including red, green, and blue components, to image said target object, and target object reflected optical energy components emitted by said emitter in said second spectral band to determine at least magnitude of said Z, said TOF system acquiring magnitude of said distance Z without a shutter, the sensor comprising:

an integrated circuit substrate having formed thereon image pixel detector diodes disposed with respect to said emitter optical path to receive and to detect, incoming to said TOF system, reflected optical energy components in said first spectral band including red, green, and blue components, and having formed thereon Z pixel detector diodes, interspersed among said image pixel detector diodes to detect target object reflected optical energy components in said second spectral band, said integrated circuit substrate so disposed thus receiving incoming optical energy reflected from external to said TOF system without receiving optical energy emitted by said emitter that has not been so reflected, a plane of said IC substrate being parallel to an optic axis defined by components of said first spectral band and said second spectral band;

means for outputting image data from outputs of said image pixel detector diodes including red, green, and blue optical energy components, said image data useable to display at least a portion of said target object;

means for outputting Z data from outputs of said Z pixel detector diodes, said Z data useable to determine at least magnitude of said Z.

17. The sensor of claim 16, wherein:

said first spectral band comprises at least one group of wavelengths selected from a group consisting of (i) red, green, blue (RGB) wavelengths, and (ii) wavelengths in a range of 400 nm to 650 nm; and

said second spectral band comprises at least one group of wavelengths selected from a group consisting of (i) IR wavelengths, (ii) near IR (NIR) wavelengths, (iii) a wavelength of 800 nm, and (iv) at least one wavelength greater than 650 nm.

18. A CMOS-implementable sensor useable to identify a target object with a time-of-flight (TOF) system that is carried by a motor vehicle and includes an emitter that emits optical energy in an emitter optical path toward said target object, emitted said optical energy including a second spectral band comprising non-visible wavelengths, a fraction of which emitted said optical energy is reflected from said target object a distance Z away from said TOF system, the CMOS-implementable sensor detecting target object reflected optical energy components including optical energy in a first spectral band comprising red, green, blue (RGB) visible wavelengths to image said target objects, and target object reflected optical energy components in said second spectral band to determine at least magnitude of said Z, said TOF system acquiring magnitude of said distance Z without a shutter, the CMOS implementable sensor comprising:

means, disposed with respect to said emitter optical path to receive, incoming to said TOF system, reflected optical energy components in said first spectral band including red, green, and blue components, and target object reflected optical energy components in said second spectral band, for substantially passing optical energy components in one said spectral band and for substantially reflecting optical energy components in the other said spectral band, said means for substantially passing and for substantially reflecting so disposed thus receiving incoming optical energy reflected from external to said TOF system without receiving optical energy emitted by said emitter that has not been so reflected;

an image sensor disposed on an integrated circuit (IC) substrate to detect first spectral band components including red, green, and blue components, output from said means for substantially passing and for substantially reflecting and to output image data useable to display at least a portion of said target object;

a Z sensor disposed on said IC substrate to detect second spectral band components output by said means for substantially passing and for substantially reflecting and to output Z data useable to determine at least magnitude of said Z, a plane of said IC substrate being parallel to an optic axis defined by components of said first spectral band and said second spectral band;

means for displaying an image from an output of said image sensor;

means for determining, from an output of said Z sensor, at least one of (a) magnitude of said distance Z, (b) magnitude of Δ

Z/Δ

t of said target object, and (c) approximate shape of at least a portion of said target object, and (d) approximate size of at least a portion of said target object; and

means, coupled to an output of said means for determining, for augmenting information displayed on said image by carrying out at least one of (a) visually highlighting a portion of said image, (b) displaying a warning on said image, (c) sounding an acoustic alarm, (d) outputting a signal useable to sound a horn in said motor vehicle, and outputting a signal to activate brakes in said motor vehicle.