HIGH RESOLUTION MULTISPECTRAL IMAGE CAPTURE

• US 20140085502A1
• Filed: 06/15/2011
• Published: 03/27/2014
• Est. Priority Date: 06/15/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Lin, Stephen S., Tong, Xin, Cao, Xun
• Granted Patent: US 9,635,274 B2
• Time in Patent Office: Days
• US Class Current: 348/223.1
• CPC Class Codes: G02B 27/1013 for colour or multispectral image sensors, e.g. splitting an image into monochromatic image components on respective sensors spectral imaging systems G01J
• CPC Class Codes: G02B 27/1066 for enhancing image performance, like resolution, pixel numbers, dual magnifications or dynamic range, by tiling, slicing or overlapping fields of view
• CPC Class Codes: H04N 23/13 with multiple sensors
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/72 Combination of two or more compensation controls
• CPC Class Codes: H04N 23/84 for processing colour signals
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio
• CPC Class Codes: H04N 25/71 Charge-coupled device [CCD] sensors; Charge-transfer registers specially adapted for CCD sensors
• CPC Class Codes: H04N 5/265 Mixing

Abstract

The use of a hybrid camera tool may capture high resolution multispectral images without sacrificing resolution in exchange for spectral accuracy. The capture of a high resolution multispectral image may include forming a high spatial resolution image based on a portion of incoming light and generating a high spectral resolution image that includes multispectral samples based on another portion of the incoming light. The high resolution multispectral image is then synthesized by consolidating the high spatial resolution image with the high spectral resolution image.

Claims

1. A device, comprising:

a beam splitter to split incoming light into a first light portion and a second light portion;

a first camera that records a red, green, blue (RGB) image based on the first light portion;

an occlusion mask that subsamples light radiance from the second light portion to provide multispectral samples; and

a second camera that records a grayscale image that includes the multispectral samples.

2. The device of claim 1, further comprising an image consolidation engine that synthesizes a consolidated image having a spatial resolution derived from the RGB image and a spectral resolution derived from the grayscale image.

3. The device of claim 1, further comprising an opaque enclosure that includes an aperture to pass the incoming light to the beam splitter, the opaque enclosure at least enclosing the beam splitter, the occlusion mask, and a prism.

4. The device of claim 1, further comprising a mirror that reflects the first light portion from the beam splitter to the first camera at an angle that reduces a distance between the first camera and the second camera.

5. The device of claim 1, further comprising a prism or another spectrally dispersive optical component that disperses the second light portion along an initial direction and with which the multispectral samples are averaged in another direction that is perpendicular to the initial direction.

6. The device of claim 1, further comprising a filter that filters the second light portion to remove one or more spectrum bands of the second light portion.

7. A method, comprising:

forming a high spatial resolution image from a first light portion of an incoming light;

generating a high spectral resolution image that includes multispectral samples based on a second light portion of the incoming light, the high spectral resolution image having less spatial resolution and more spectral resolution than the high spatial resolution image; and

consolidating the high spatial resolution image with the high spectral resolution image to generate a consolidated image having a spatial resolution of the high spatial resolution image and a spectral resolution of the high spectral resolution image.

8. The method of claim 7, further comprising splitting the incoming light into the first light portion and the second light portion.

9. The method of claim 7, further comprising performing a color adjustment on the consolidated image.

10. The method of claim 7, wherein the high spectral resolution image is a red, green, blue (RGB) image that is encoded based on red, green, and blue color channel data, and wherein the high spectral resolution image is a grayscale image.

11. The method of claim 7, wherein the generating includes:

passing the second light portion through an occlusion mask that subsamples light radiance from the second light portion to generate the multispectral samples;

dispersing the second light portion along an initial direction and with which the multispectral samples are averaged in another direction that is perpendicular to the initial direction; and

recording the high spectral resolution image that includes the multispectral samples.

12. The method of claim 7, wherein the consolidating includes:

identifying associated pixels in the high spatial resolution image that have corresponding multispectral samples from the high spectral resolution image;

identifying unassociated pixels in the high spatial resolution image that lack corresponding multispectral samples from the high spectral resolution image; and

propagating multispectral information to each unassociated pixel from one or more neighboring associated pixels.

13. The method of claim 12, further comprising identifying another associated pixel by using optical flow to acquire an additional multispectral sample from another high spatial resolution image that temporally preceded the high spatial resolution image.

14. The method of claim 12, wherein the propagating includes propagating the multispectral information on a per-channel basis to each of a plurality of RGB channels of an unassociated pixel, and summing spectral values of the plurality of RGB channels to obtain a full spectrum for the unassociated pixel.

15. A computing device, comprising:

one or more processors; and

a memory that includes a plurality of computer-executable components, the plurality of computer-executable components comprising;

a multispectral propagation component that generates a consolidated image from a high spatial resolution image and multispectral samples captured in a high spectral resolution image, the high spatial resolution image having less spectral resolution and more spatial resolution than the high spectral resolution image; and

a temporal enhancement component that acquires one or more of the multispectral samples from at least one additional high spatial resolution image that temporally preceded the high spatial resolution image via optical flow.

16. The computing device of claim 15, further comprising a color adjustment component that automatically adjusts one or more colors of the consolidated image.

17. The computing device of claim 15, wherein a multispectral propagation component generates a plurality of consolidated images from corresponding high spatial resolution images and high spectral resolution images, further comprising a tracking component that tracks an object that is captured in the plurality of consolidated images.

18. The computing device of claim 15, wherein at least some of the multispectral samples are generated from light that is passed through an occlusion mask and a prism.

19. The computing device of claim 15, wherein the multispectral propagation component generates the consolidated image by:

identifying associated pixels in the high spatial resolution image that have corresponding multispectral samples from the high spectral resolution image;

identifying unassociated pixels in the high spatial resolution image that lack corresponding multispectral samples from the high spectral resolution image; and

propagating multispectral information to each unassociated pixel from one or more neighboring associated pixels.

20. The computing device of claim 15, wherein the multispectral propagation component propagates multispectral information in the multispectral samples on a per-channel basis to each of a plurality of RGB channels of an unassociated pixel, and sums spectral values of the plurality of RGB channels to obtain a full spectrum for the unassociated pixel.