Depth sensor using modulated light projector and image sensor with color and IR sensing

• US 9,407,837 B2
• Filed: 02/21/2014
• Issued: 08/02/2016
• Est. Priority Date: 02/28/2013
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Lee, Johnny C.
• Primary Examiner(s): Dagnew, Mekonnen
• Application Number:
  US14/186,877
• Publication Number:
  US 20140240492A1
• Time in Patent Office: 893 Days
• US Class Current: 1/1
• CPC Class Codes: G01C 3/08 Use of electric radiation detectors
• CPC Class Codes: G06V 10/145 Illumination specially adapted for pattern recognition, e.g. using gratings
• CPC Class Codes: G06V 20/10 Terrestrial scenes scenes under surveillance with static cameras G06V20/52; scenes perceived from the exterior of a vehicle G06V20/56; scenes perceived from the interior of a vehicle G06V20/59
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/611 where the recognised objects include parts of the human body
• CPC Class Codes: H04N 23/63 by using electronic viewfinders
• CPC Class Codes: H04N 23/667 Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
• CPC Class Codes: H04N 23/73 by influencing the exposure time
• CPC Class Codes: H04N 23/74 by influencing the scene brightness using illuminating means
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/00 Circuitry of solid-state image sensors [SSIS]; Control thereof
• CPC Class Codes: H04N 25/131 including elements passing infrared wavelengths
• CPC Class Codes: H04N 25/135 based on four or more different wavelength filter elements
• CPC Class Codes: H04N 25/53 Control of the integration time
• CPC Class Codes: H04N 25/702 SSIS architectures characterised by non-identical, non-equidistant or non-planar pixel layout
• CPC Class Codes: H04N 25/705 Pixels for depth measurement, e.g. RGBZ

Abstract

An electronic device includes a modulated light projector to project an infrared light pattern and an image sensor. The image sensor includes an array of photosensors and a filter array overlying the array of photosensors. The filter array comprises a mosaic of color filter elements and infrared filter elements, each color filter element overlying a corresponding photosensor of a first subset of photosensors and each infrared filter element overlying a corresponding photosensor of a second subset of photosensors. The electronic device further includes a controller to, in a first mode, control the image sensor to capture a color image using the first subset of photosensors and, in a second mode, control the image sensor to capture a depth image representing a reflection of the infrared light pattern using the second subset of photosensors.

Claims

1. An electronic device comprising:

a modulated light projector to project an infrared light pattern; and

an image sensor comprising;

an array of photosensors; and

a filter array overlying the array of photosensors and comprising a mosaic of color filter elements and infrared filter elements, each color filter element overlying a corresponding photosensor of a first subset of photosensors and each infrared filter element overlying a corresponding photosensor of a second subset of photosensors, wherein each photosensor of the first subset has a first size and each photosensor of the second subset has a second size different than the first size; and

a controller to, in a first mode, control the image sensor to capture a color image using the first subset of photosensors and, in a second mode, control the image sensor to capture a depth image representing a reflection of the infrared light pattern using the second subset of photosensors.

2. The electronic device of claim 1, wherein the modulated light projector comprises:

an array of one or more vertical cavity surface emitting laser (VCSEL) diodes;

an array of one or more lenses overlying the array of one or more VCSEL diodes; and

a diffractive optical element overlying the array of one or more lenses.

3. The electronic device of claim 1, wherein the electronic device further comprises:

a display disposed at a first surface of the electronic device, wherein the electronic device is configured to present, via the display, imagery captured using the image sensor; and

wherein the modulated light projector and the image sensor are disposed at a second surface of the electronic device, the second surface opposite the first surface.

4. The electronic device of claim 3, wherein:

the image sensor is a first image sensor; and

the electronic device further comprises a second image sensor disposed at the second surface, the second image sensor having a field of view different than a field of view of the first image sensor.

5. A method comprising:

providing an electronic device comprising an image sensor, the image sensor comprising an array of photosensors and a filter array overlying the array of photosensors and comprising a mosaic of color filter elements and infrared filter elements, each color filter element overlying a corresponding photosensor of a first subset of photosensors and each infrared filter element overlying a corresponding photosensor of a second subset of photosensors;

capturing, at the electronic device, a color image using the first subset of photosensors;

projecting, from the electronic device, an infrared light pattern;

capturing, at the electronic device, a depth image representing a reflection of the infrared light pattern using the second subset of photosensors; and

wherein providing the electronic device comprises providing the image sensor with each photosensor of the first subset having a first size and each photosensor of the second subset having a second size different than the first size.

6. The method of claim 5, further comprising:

determining, at the electronic device, at least one spatial feature from at least one of the color image and the depth image; and

determining, at the electronic device, at least one of a relative position and a relative orientation of the electronic device based on the at least one spatial feature.

7. The method of claim 5, further comprising:

displaying the color image at a display of the electronic device.

8. The method of claim 5, wherein capturing the depth image comprises:

binning subarrays of M×

N photosensors of the second subset into corresponding macropixels for the depth image, M and N comprising integer numbers greater than zero and at least one of M and N being greater than one.

9. The method of claim 5, wherein providing the image sensor further comprises:

providing the image sensor with the color filter elements that are non-uniformly distributed in the filter array.