Image demosaicing for hybrid optical sensor arrays

• US 10,275,855 B2
• Filed: 01/11/2017
• Issued: 04/30/2019
• Est. Priority Date: 01/11/2017
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Price, Raymond Kirk, Demandolx, Denis, Bleyer, Michael
• Primary Examiner(s): Mariam, Daniel G
• Application Number:
  US15/404,025
• Publication Number:
  US 20180197275A1
• Time in Patent Office: 839 Days
• Field of Search: 382300
• CPC Class Codes: G01B 11/25 by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50
• CPC Class Codes: G01B 11/2513 with several lines being projected in more than one direction, e.g. grids, patterns
• CPC Class Codes: G01B 11/2545 with one projection direction and several detection directions, e.g. stereo
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 3/4015 Image demosaicing, e.g. colour filter arrays [CFA] or Bayer patterns
• CPC Class Codes: G06T 3/403 Edge-driven scaling; Edge-based scaling
• CPC Class Codes: G06T 7/521 from laser ranging, e.g. using interferometry; from the projection of structured light
• CPC Class Codes: G06T 7/55 from multiple images
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/843 Demosaicing, e.g. interpolating colour pixel values
• CPC Class Codes: H04N 25/131 including elements passing infrared wavelengths

Abstract

An imaging device comprises a hybrid optical sensor array including a first and second set of pixels that comprise different numbers of pixels. The first set of pixels is sensitive to infrared light, while the second set of pixels comprises three subsets of pixels sensitive to RGB light. A first set of data for a scene is captured by the first set of pixels and a second set of data for the scene is captured by a second set of pixels. The first and second sets of data are jointly demosaiced, such that the higher resolution data set is utilized to increase the resolution of a lower resolution data set. This allows for high-resolution infrared and RGB images to be produced for a scene without perspective or timing discrepancies inherent in multi-camera machine vision systems.

Claims

1. A method for an imaging device, comprising:

receiving, from a hybrid optical sensor array comprising n pixels, a first set of data for a scene captured by a first set of pixels and a second set of data for the scene captured by a second set of pixels, wherein;

the first set of pixels and second set of pixels total n pixels, where n is a positive integer;

the first set of pixels and second set of pixels comprise different numbers of pixels;

the first set of pixels is sensitive to infrared light; and

the second set of pixels comprises a first subset of pixels sensitive to red light, a second subset of pixels sensitive to green light, and a third subset of pixels sensitive to blue light;

demosaicing the first set of data based on at least the second set of data and demosaicing the second set of data based on at least the first set of data, such that a higher resolution data set is utilized to increase the resolution of a lower resolution data set; and

outputting a third set of data for the scene comprising an infrared image with n pixels, and outputting a fourth set of data for the scene comprising an RGB image with n pixels.

2. The method of claim 1, wherein demosaicing the first set of data based on at least the second set of data comprises:

interpolating the first set of data independent of the second set of data; and

filtering the interpolated first set of data based on the second set of data.

3. The method of claim 2, wherein the second set of pixels comprises a greater number of pixels than the first set of pixels, and wherein filtering the interpolated first set of data based on the second set of data further comprises:

filtering the interpolated first set of data based on guidance image weights derived from the second set of data.

4. The method of claim 1, wherein demosaicing the second set of data based on at least the first set of data comprises:

interpolating the second set of data independent of the first set of data; and

filtering the interpolated second set of data based on the first set of data.

5. The method of claim 4, wherein the first set of pixels comprises a greater number of pixels than the second set of pixels, and wherein filtering the interpolated second set of data based on the first set of data further comprises:

filtering the interpolated second set of data based on guidance image weights derived from the first set of data.

6. The method of claim 1, wherein the demosaicing of the first set of data and the second set of data comprises:

jointly interpolating the first set of data and the second set of data; and

jointly re-interpolating the interpolated first set of data and the interpolated second set of data.

7. The method of claim 6, wherein the second set of pixels comprises a greater number of pixels than the first set of pixels, and wherein jointly interpolating the first set of data and the second set of data comprises:

interpolating the first set of data based on guidance image weights derived from the second set of data.

8. The method of claim 6, wherein the first set of pixels comprises a greater number of pixels than the second set of pixels, and wherein jointly interpolating the first set of data and the second set of data comprises:

interpolating the first set of data based on guidance image weights derived from the second set of data.

9. The method of claim 1, further comprising:

generating depth information for the scene based on at least the third set of data.

10. The method of claim 1, further comprising:

generating depth information for the scene based on at least the first set of data, and wherein demosaicing the second set of data based on at least the first set of data further comprises demosaicing the second set of data based on the generated depth information for the scene.

11. An imaging system, comprising:

a hybrid optical sensor array comprising n pixels, where n is a positive integer, then pixels comprising a first set of pixels and a second set of pixels, the first set of pixels comprising n/4 pixels sensitive to infrared light, the second set of pixels comprising 3n/4 pixels, including a first subset of pixels sensitive to red light, a second subset of pixels sensitive to green light, and a third subset of pixels sensitive to blue light;

a storage machine holding instructions executable by a logic machine to;

receive infrared light of a scene at the first set of pixels;

receive visible light of the scene at the second set of pixels;

generate a first RGB data set of the scene based on the received visible light;

generate a first infrared light data set of the scene based on the received infrared light;

demosaic the infrared data set of the scene based on spatial information derived from the first RGB data set of the scene;

output a second infrared light data set of the scene comprising n pixels.

12. The imaging system of claim 11, wherein demosaicing the first infrared data set of the scene based on spatial information derived from the first RGB data set of the scene comprises:

interpolating the first infrared data set independent of the first RGB data set; and

filtering the interpolated infrared data set based on the first RGB data set.

13. The imaging system of claim 11, wherein the storage machine further holds instructions executable by the logic machine to:

demosaic the first RGB data set of the scene based on the infrared data set of the scene; and

wherein demosaicing the first infrared data set of the scene and the first RGB data set of the scene comprises;

jointly interpolating the first infrared data set and the first RGB data set; and

jointly re-interpolating the interpolated infrared data set and the interpolated RGB data set.

14. The imaging system of claim 11, further comprising:

a time-of-flight depth camera, and wherein the storage machine further holds instructions executable by the logic machine to;

generate two or more infrared light data sets of the scene based on the received infrared light;

demosaic each infrared data set of the scene based on the first RGB data set of the scene, each infrared data set weighted equally during demosaicing; and

output depth values for the scene comprising n pixels based on the demosaiced infrared data sets.

15. The imaging system of claim 11, further comprising:

a structured light depth camera, and wherein the storage machine further holds instructions executable by the logic machine to;

generate depth values for the scene based on the first infrared light data set of the scene;

demosaic the first RGB data set of the scene based on the generated depth values for the scene;

demosaic the generated depth values for the scene based on spatial information derived from the first RGB data set of the scene; and

output depth values for the scene comprising n pixels based on the demosaiced generated depth values.

16. An imaging system, comprising:

a hybrid optical sensor array comprising n pixels, where n is a positive integer, then pixels comprising a first set of pixels and a second set of pixels, the first set of pixels comprising 3n/4 pixels sensitive to infrared light, the second set of pixels comprising n/4 pixels, including a first subset of pixels sensitive to red light, a second subset of pixels sensitive to green light, and a third subset of pixels sensitive to blue light;

a storage machine holding instructions executable by a logic machine to;

receive infrared light of a scene at the first set of pixels;

receive visible light of the scene at the second set of pixels;

generate a first RGB data set of the scene based on the received visible light;

generate a first infrared light data set of the scene based on the received infrared light;

demosaic the first RGB data set of the scene based on guidance image weights derived from the infrared light data set of the scene; and

output a second RGB data set of the scene comprising n pixels.

17. The imaging system of claim 16, wherein demosaicing the first RGB data set of the scene based on at least guidance image weights derived from the infrared light data set of the scene comprises:

interpolating the first RGB data set independent of the first infrared data set; and

filtering the interpolated RGB data set based on the infrared data set.

18. The imaging system of claim 16, wherein the storage machine further holds instructions executable by the logic machine to:

demosaic the first infrared data set of the scene based on the RGB data set of the scene; and

wherein demosaicing the first infrared data set of the scene and the first RGB data set of the scene comprises;

jointly interpolating the first infrared data set and the first RGB data set; and

jointly re-interpolating the interpolated infrared data set and the interpolated RGB data set.

19. The imaging system of claim 16, further comprising:

a time-of-flight depth camera, and wherein the storage machine further holds instructions executable by the logic machine to;

generate two or more infrared light data sets of the scene based on the received infrared light;

demosaic each infrared data set of the scene based on the first RGB data set of the scene, each infrared data set weighted equally during demosaicing; and

output depth values for the scene comprising n pixels based on the demosaiced infrared data sets.

20. The imaging system of claim 16, further comprising:

a structured light depth camera, and wherein the storage machine further holds instructions executable by the logic machine to;

generate depth values for the scene based on the first infrared light data set of the scene;

demosaic the first RGB data set of the scene based on guidance image weights derived from the generated depth values for the scene;

demosaic the generated depth values for the scene based on the first RGB data set of the scene; and

output depth values for the scene comprising n pixels based on the demosaiced generated depth values.