Compact color and depth imaging system

• US 10,609,310 B2
• Filed: 09/12/2019
• Issued: 03/31/2020
• Est. Priority Date: 09/05/2018
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Wan, Chung Chun, Ko, Jamyuen
• Primary Examiner(s): Lam, Hung H
• Application Number:
  US16/568,458
• Publication Number:
  US 20200077037A1
• Time in Patent Office: 201 Days
• CPC Class Codes: G01J 5/0803 Arrangements for time-dependent attenuation of radiation signals
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/10141 Special mode during image acquisition
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 23/10 for generating image signals from different wavelengths
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
• CPC Class Codes: H04N 9/646 for image enhancement, e.g. vertical detail restoration, cross-colour elimination, contour correction, chrominance trapping filters

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for depth imaging. In one aspect, a method includes obtaining, by an image sensor that includes infrared pixels and color pixels, a first image of a scene while the image sensor is in a first position, moving the image sensor to a second position, wherein, in the second position, a particular infrared pixel is located where a particular color pixel was previously located when the image sensor was in the first position, obtaining, by the image sensor, a second image of the scene while the image sensor is in the second position, generating a composite image based on the first image and the second image, and determining an estimated distance to an object within the scene based on the composite image.

Claims

1. A computer-implemented method, comprising:

obtaining, by an image sensor that includes infrared pixels and color pixels, a first image of a scene while the image sensor is in a first position;

moving the image sensor to a second position, wherein, in the second position, a particular infrared pixel is located where a particular color pixel was previously located when the image sensor was in the first position;

obtaining, by the image sensor, a second image of the scene while the image sensor is in the second position; and

generating a composite image based on the first image and the second image.

2. The method of claim 1, wherein moving the image sensor to a second position comprises:

moving the image sensor to the second position such that the particular infrared pixel is located where a particular color pixel was previously located when the image sensor was in the first position, relative to a camera that includes the image sensor.

3. The method of claim 1, wherein moving the image sensor to a second position comprises:

moving the image sensor to the second position such that light from the scene that fell onto the particular color pixel in the first position instead falls onto the particular infrared pixel in the second position.

4. The method of claim 1, wherein moving the image sensor to a second position comprises:

moving the image sensor to the second position such that light from the scene that fell onto the particular infrared pixel in the first position instead falls onto a second particular color pixel in the second position.

5. The method of claim 1, wherein generating a composite image based on the first image and the second image comprises:

generating the composite image to include a first pixel from the particular infrared pixel while the image sensor was in the first position and a second pixel, at a different location in the composite image than the first pixel, from the particular infrared pixel while the image sensor was in the second position.

6. The method of claim 1, wherein the image sensor includes a pattern of infrared pixels and color pixels, where the pattern includes an infrared pixel, a first green color pixel adjacent and to the right of the infrared pixel, a red color pixel adjacent and below the first green pixel, and a second green color pixel adjacent and below the infrared pixel.

7. The method of claim 6, comprising:

obtaining, by the image sensor, a third image of the scene while the image sensor is in a third position, wherein, in the third position, the particular infrared pixel is located where a second particular color pixel was previously located when the image sensor was in the first position; and

obtaining, by the image sensor, a fourth image of the scene while the image sensor is in a fourth position, wherein, in the fourth position, the particular infrared pixel is located where a third particular color pixel was previously located when the image sensor was in the first position,

wherein generating a composite image based on the first image and the second image comprises;

generating the composite image based on the first image, the second image, the third image, and the fourth image.

8. The method of claim 1, wherein obtaining, by an image sensor that includes infrared pixels and color pixels, a first image of a scene while the image sensor is in a first position comprises:

obtaining an infrared image of the scene from the infrared pixels in the image sensor and not from the color pixels in the image sensor.

9. The method of claim 1, wherein obtaining, by an image sensor that includes infrared pixels and color pixels, a first image of a scene while the image sensor is in a first position comprises:

obtaining a first infrared image with the infrared pixels while an infrared emitter is emitting infrared onto the scene; and

obtaining a first color image with the color pixels while the infrared emitter is not emitting infrared onto the scene,

wherein generating a composite image based on the first image and the second image comprises;

generating a composite infrared image based on the first infrared image and a second infrared image; and

generating a composite color image based on the first color image and a second color image.

10. A system comprising:

one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising;

obtaining, by an image sensor that includes infrared pixels and color pixels, a first image of a scene while the image sensor is in a first position;

moving the image sensor to a second position, wherein, in the second position, a particular infrared pixel is located where a particular color pixel was previously located when the image sensor was in the first position;

obtaining, by the image sensor, a second image of the scene while the image sensor is in the second position; and

generating a composite image based on the first image and the second image.

11. The system of claim 10, wherein moving the image sensor to a second position comprises:

moving the image sensor to the second position such that the particular infrared pixel is located where a particular color pixel was previously located when the image sensor was in the first position, relative to a camera that includes the image sensor.

12. The system of claim 10, wherein moving the image sensor to a second position comprises:

moving the image sensor to the second position such that light from the scene that fell onto the particular color pixel in the first position instead falls onto the particular infrared pixel in the second position.

13. The system of claim 10, wherein moving the image sensor to a second position comprises:

moving the image sensor to the second position such that light from the scene that fell onto the particular infrared pixel in the first position instead falls onto a second particular color pixel in the second position.

14. The system of claim 10, wherein generating a composite image based on the first image and the second image comprises:

generating the composite image to include a first pixel from the particular infrared pixel while the image sensor was in the first position and a second pixel, at a different location in the composite image than the first pixel, from the particular infrared pixel while the image sensor was in the second position.

15. The system of claim 10, wherein the image sensor includes a pattern of infrared pixels and color pixels, where the pattern includes an infrared pixel, a first green color pixel adjacent and to the right of the infrared pixel, a red color pixel adjacent and below the first green pixel, and a second green color pixel adjacent and below the infrared pixel.

16. The system of claim 15, the operations comprising:

obtaining, by the image sensor, a third image of the scene while the image sensor is in a third position, wherein, in the third position, the particular infrared pixel is located where a second particular color pixel was previously located when the image sensor was in the first position; and

obtaining, by the image sensor, a fourth image of the scene while the image sensor is in a fourth position, wherein, in the fourth position, the particular infrared pixel is located where a third particular color pixel was previously located when the image sensor was in the first position,

wherein generating a composite image based on the first image and the second image comprises;

generating the composite image based on the first image, the second image, the third image, and the fourth image.

17. The system of claim 10, wherein obtaining, by an image sensor that includes infrared pixels and color pixels, a first image of a scene while the image sensor is in a first position comprises:

obtaining an infrared image of the scene from the infrared pixels in the image sensor and not from the color pixels in the image sensor.

18. The system of claim 10, wherein obtaining, by an image sensor that includes infrared pixels and color pixels, a first image of a scene while the image sensor is in a first position comprises:

obtaining a first infrared image with the infrared pixels while an infrared emitter is emitting infrared onto the scene; and

obtaining a first color image with the color pixels while the infrared emitter is not emitting infrared onto the scene,

wherein generating a composite image based on the first image and the second image comprises;

generating a composite infrared image based on the first infrared image and a second infrared image; and

generating a composite color image based on the first color image and a second color image.

19. A non-transitory computer-readable medium storing instructions executable by one or more computers which, upon such execution, cause the one or more computers to perform operations comprising:

obtaining, by an image sensor that includes infrared pixels and color pixels, a first image of a scene while the image sensor is in a first position;

moving the image sensor to a second position, wherein, in the second position, a particular infrared pixel is located where a particular color pixel was previously located when the image sensor was in the first position;

obtaining, by the image sensor, a second image of the scene while the image sensor is in the second position; and

generating a composite image based on the first image and the second image.

20. The medium of claim 19, wherein moving the image sensor to a second position comprises:

moving the image sensor to the second position such that the particular infrared pixel is located where a particular color pixel was previously located when the image sensor was in the first position, relative to a camera that includes the image sensor.