System for multi- and hyperspectral imaging

• US 7,835,002 B2
• Filed: 10/25/2005
• Issued: 11/16/2010
• Est. Priority Date: 10/25/2004
• Status: Expired due to Fees

• Current Assignee: RP Ventures Technology Office AB
• Original Assignee: RP Ventures Technology Office AB
• Inventors: Muhammed, Hamed Hamid, Bergholm, Fredrik
• Primary Examiner(s): Chowdhury; Tarifur
• Assistant Examiner(s): LAPAGE, MICHAEL P
• Application Number:
  US11/577,860
• Publication Number:
  US 20080123097A1
• Time in Patent Office: 1,848 Days
• Field of Search: 356/416, 356418-419, 250/226, 348294-326, 348335-336, 348/340, 348/342, 359/634
• US Class Current: 356/419
• CPC Class Codes: G01J 2003/1213 Filters in general, e.g. dichroic, band
• CPC Class Codes: G01J 2003/2826 Multispectral imaging, e.g. filter imaging
• CPC Class Codes: G01J 3/02 Details
• CPC Class Codes: G01J 3/0208 using focussing or collimating elements, e.g. lenses or mirrors; performing aberration correction
• CPC Class Codes: G01J 3/0218 using optical fibers
• CPC Class Codes: G01J 3/0294 Multi-channel spectroscopy
• CPC Class Codes: G01J 3/2823 Imaging spectrometer
• CPC Class Codes: G01J 3/51 using colour filters
• CPC Class Codes: G01N 2021/1793 Remote sensing
• CPC Class Codes: G01N 2021/1797 in landscape, e.g. crops
• CPC Class Codes: G01N 2021/3177 Use of spatially separated filters in simultaneous way

Abstract

Arrangement for the production of instantaneous or non-instantaneous multi-band images, to be transformed into multi- or hyperspectral images, comprising light collecting means, an image sensor array, and an instantaneous colour separating means, positioned before the image sensor array, and uniform spectral filters, for restricting imaging to certain parts of the electromagnetic spectrum. A filter unit is positioned before the colour separating means in the optical path in, or close to, converged light. Each filter mosaic consists of a multitude of homogeneous filtering regions. The transmission curves of the filtering regions of a colour or spectral filter mosaic can be partly overlapping, in addition to overlap between these transmission curves and those belonging to the filtering regions of the colour separating means. The transmission curves of the colour or spectral filter mosaics and the colour separating means are suitably spread out in the intervals of a spectrum to be studied.

Claims

1. An arrangement for the production of instantaneous or non-instantaneous multi-band images, comprising:

light collecting means, an image sensor with at least one two dimensional sensor array with an instantaneous colour separating means, positioned before or embedded in said two dimensional sensor array in the optical path of said arrangement, and a first uniform spectral filter in said optical path, with the purpose of restricting imaging to certain parts of the electromagnetic spectrum, characterised in that a filter unit is used comprising overlapping colour or spectral filter mosaics, or a filter unit comprising overlapping colour or spectral filter mosaics and uniform colour or spectral filters, that said filter unit is either permanently or interchangeably positioned before said colour separating means in said optical path in, or close to, converged light, that said colour or spectral filter mosaics comprises a multitude of homogeneous filtering regions, where there is an overlap or partial overlap between transmission curves of said filtering regions of colour or spectral filter mosaics, said uniform colour or spectral filters and the filtering regions of said colour separating means, that said transmission curves of said colour or spectral filter mosaics, said uniform filters and said colour separating means are spread out in the intervals of a spectrum to be studied, and that the combination of said colour separating means and said colour or spectral filter mosaics produces different sets of linearly independent transmission curves, and that the multiple-filter image captured by said image sensor is demosaicked, by identifying and segmenting the image regions that are affected by the regions of the multiple filter mosaic, to obtain a multi-band image, and that said multi-band image is transformed by using one of the two approaches, the statistical approach using a partial least squares method or an artificial neural network approach, to obtain a multi- or hyperspectral image.

2. An arrangement according to claim 1, characterised in that said image sensor comprises three separate two dimensional sensor arrays, and that said colour separating means comprises means for separating incoming light into three different spectral bands, one band to respective two dimensional sensor array, such as a beam splitter with three separate uniform colour filters or a colour separating beam splitter, and that said filter unit comprises colour or spectral filter mosaics.

3. An arrangement according to claim 1, characterised in that at least one additional optical fibre is used to provide light signal to a portion of the multiple filter mosaic in said arrangement, while the rest of said multiple filter mosaic area receives light signals delivered by a lens system.

4. A system comprising an arrangement according to claim 3, characterised in that the multiple-filter image captured by said image sensor is demosaicked by identifying and segmenting the image regions that are affected by the regions of the multiple filter mosaic, and that the mean values are calculated for the image regions, detecting the optical fibres'"'"' light-signals, to generate one multi-band mean-vector for each region, while interpolation is performed on the other image regions by using one of the three approaches, the interpolation approach using distance transform, the interpolation approach using statistical transformation employing the partial least squares method, or the artificial neural network approach, to obtain a multi-band image with complete band-content in these image regions, and that said multi-band image, including said multi-band mean-vectors, is transformed by using one of the two approaches, the statistical approach using the partial least squares method or the artificial neural network approach, to obtain a multi- or hyperspectral image, and a spectrum corresponding to each said multi-band mean-vector.

5. A system comprising an arrangement according to claim 4, characterised in that at least one of the optical fibres delivers the lighting source'"'"'s signal and the resulting said multi-band mean-vector and the corresponding estimated spectrum are used to compensate for the lighting effect on said multi-band image and said multi- or hyperspectral image, respectively, and by this way, reflectance spectral responses, respectively, reflectance spectra are obtained in the pixels.

6. A system comprising an arrangement according to claim 5, characterised in that said reflectance spectral responses and said reflectance spectra are converted into R, G and B values to generate a colour corrected image by using R, G and B filters with desired or optimised transmission curves.

7. An arrangement according to claim 1, characterised in that at a number of optical fibres is used to provide light signals to said multiple filter mosaic in said arrangement.

8. An arrangement according to claim 1, characterised in that each said two dimensional sensor array comprises layered photodiodes with at least three layers, so that spectral or colour separation is embedded into said sensor array and no additional colour separating means (123, 123a, 123b, 123c or 123d, respectively) is needed.

9. An arrangement according to claim 8, characterised in that a CMYT or a CMY colour mosaic is used to achieve the desired multiple filtering effect, or that uniform C, M and Y colour filters are used to achieve the desired multiple filtering effect, that the used layered photodiodes have at least three layers, and that pixel-wise matching is preferred between said image sensor array and said colour mosaic.

10. An arrangement according to claim 8, characterised in that a CMYTBrBk colour mosaic is used to achieve the desired multiple filtering effect, when using layered photodiodes with at least three layers.

11. An arrangement according to claim 8, characterised in that a CMYT or a CMYTBk colour mosaic is used to achieve the desired multiple filtering effect, when using layered photodiodes with at least three layers where at least one of the layers is sensitive to near infrared light and at least another one of the layers is sensitive to near ultraviolet light.

12. An arrangement according to claim 1, characterised in that said colour or spectral filter mosaics and/or said uniform colour or spectral filters are produced by existing colour laser or ink printing techniques, or by other photographic printing techniques, on transparent or semitransparent materials.

13. An arrangement according to claim 1, characterised in that said image sensor comprises one sole two dimensional sensor array, and a colour separating means that is a second colour or spectral filter mosaic with elements fitting individual pixel elements on said sole two dimensional sensor array.

14. An arrangement according to claim 1, characterised in that a digital camera is a part of said arrangement, and that said image sensor and said colour separating means are parts of said digital camera.

15. An optical auxiliary device, adapted to be connected to a conventional digital camera, with or without own camera lens, in a manner to align the optical axis of said optical device with the optical axis of said digital camera, characterised in that said optical device and said camera form an arrangement according to claim 1 when connected to each other.

16. A colour or spectral mosaic comprising at least one first colour or spectral filter mosaic, characterised in that said colour or spectral mosaic is adapted to be inserted into an ocular, whereby said ocular and inserted said colour or spectral mosaic in combination with a conventional digital camera forms an arrangement according to claim 1.

17. A colour or spectral mosaic according to claim 16, characterised in that said colour or spectral mosaic is adapted to be inserted into a ocular, such as a telescope or a microscope.

18. An arrangement according to claim 1, characterised in that the sets of uniform multiple filters or multiple filter mosaics are generated by combining complementary colour or spectral filter mosaics and/or uniform colour or spectral filters, such as RGBT or RGB and CMYT or CMY colour mosaics, or uniform colour filters of these colours, and that pixel-wise matching is preferred between said image sensor array and said multiple filter mosaic.

19. An arrangement according to claim 18, characterised in that a VT mosaic is placed in converged light to contribute to building the resulting multiple filter mosaic.

20. An arrangement according to claim 1, characterised in that an RGB and a CMYTBrBk colour mosaics are combined to build the resulting multiple filter mosaic.

21. An arrangement according to claim 1, characterised in that an RGCYBrBk and a VT colour or spectral mosaics are combined to build the resulting multiple filter mosaic.

22. A system comprising an arrangement according to any one of claims 1 to 6, characterised in that the multiple-filter image captured by said image sensor is demosaicked by identifying and segmenting the image regions that are affected by the regions of the multiple filter mosaic, and that interpolation is performed by using one of the three approaches, the interpolation approach using distance transform, the interpolation approach using statistical transformation employing the partial least squares method, or the artificial neural network approach, to obtain a multi-band image with complete bands, and that said multi-band image is transformed by using one of the two approaches, the statistical approach using the partial least squares method or the artificial neural network approach, to obtain a multi- or hyperspectral image.

23. A system comprising an arrangement according to any one of claims 1 to 9, characterised in that the multiple-filter image captured by said image sensor is demosaicked by identifying and segmenting the image regions that are affected by the regions of the multiple filter mosaic, and that the mean values are calculated for certain image regions generating one multi-band mean-vector for each region, while interpolation is performed on the other image regions by using one of the three approaches, the interpolation approach using distance transform, the interpolation approach using statistical transformation employing the partial least squares method, or the artificial neural network approach, to obtain a multi-band image with complete band-content in these image regions, and that said multi-band image, including said multi-band mean-vectors, is transformed by using one of the two approaches, the statistical approach using the partial least squares method or the artificial neural network approach, to obtain a multi- or hyperspectral image, and a spectrum corresponding to each said multi-band mean-vector.

24. An arrangement for the production of instantaneous or non-instantaneous multi-band images, comprising light:

collecting means, an image sensor with at least one two dimensional sensor array with an instantaneous colour separating means, positioned before or embedded in said two dimensional sensor array in the optical path of said arrangement, and a first uniform spectral filter in said optical path, with the purpose of restricting imaging to certain parts of the electromagnetic spectrum, characterised in that a filter unit is used comprising overlapping colour or spectral filter mosaics and uniform colour or spectral filters, or a comprising overlapping uniform colour or spectral filters, such that said uniform colour or spectral filters are mounted on filter wheels or displayed by transmissive displays, that said filter unit is positioned before said colour separating means in said optical path in, or close to, converged light, that said colour or spectral filter mosaics comprises a multitude of homogeneous filtering regions, where there is an overlap or partial overlap between transmission curves of said filtering regions of colour or spectral filter mosaics, said uniform filters and the filtering regions of said colour separating means, that said transmission curves of said colour or spectral filter mosaics, said uniform colour or spectral filters and said colour separating means are spread out in the intervals of a spectrum to be studied, and that the combination of said colour separating means and said colour or spectral filter mosaics produces different sets of linearly independent transmission curves, and that the multiple-filter image captured by said image sensor is demosaicked, by identifying and segmenting the image regions that are affected by the regions of the multiple filter mosaic, to obtain a multi-band image, and that said multi-band image is transformed by using one of the two approaches, the statistical approach using a partial least squares method or an artificial neural network approach, to obtain a multi- or hyperspectral image.

25. An image sensor for the production of instantaneous multi-band images, comprising:

at least one two dimensional sensor array, and an instantaneous colour separating means comprises a RGBT colour mosaic or a CMYT colour mosaic, embedded in said image sensor array, and a filter unit, characterised in that said image sensor comprises a colour or spectral filter mosaic, and that said filter unit can either be placed directly over said colour separating means, or integrated with said colour separating means, and that the image captured by said image sensor is demosaicked, by identifying and segmenting the image regions that are affected by the regions of the multiple filter mosaic, to obtain a multi-band image, and that said multi-band image is transformed by using one of the two approaches, the statistical approach using a partial least squares method or an artificial neural network approach, to obtain a multi- or hyperspectral image, and that said colour separating means comprises an RGBT colour mosaic while the said colour or spectral filter mosaic comprises an CMYT colour mosaic, or that said colour separating means comprises an CMYT colour mosaic while the said colour or spectral filter mosaic comprises an RGBT colour mosaic.

26. An arrangement according to claim 25, characterised in that said image sensor comprises three separate two dimensional sensor arrays with embedded colour filter mosaics, and that said colour separating means (122′

) comprises means for separating incoming light into three different spectral bands, one band to respective two dimensional sensor array, such as a beam splitter with three separate uniform colour or spectral filters (125a, 125b, 125c) or a colour separating beam splitter.

27. An arrangement according to claim 26, characterised in that said image sensor comprises four separate two dimensional sensor arrays (121a, 121b, 121c, 121d), where said sensor array (121d) is covered by an instantaneous colour separating means (123d) without using any uniform colour or spectral filter for said sensor array (121d).