Compact multi-spectrum imaging with fusion

• US 10,244,190 B2
• Filed: 12/21/2013
• Issued: 03/26/2019
• Est. Priority Date: 03/02/2009
• Status: Active Grant

• Current Assignee: Teledyne FLIR LLC (Teledyne Technologies Incorporated)
• Original Assignee: FLIR Systems, Inc. (Teledyne Technologies Incorporated)
• Inventors: Boulanger, Pierre, Sharp, Barbara, Hoelter, Theodore R., Teich, Andrew C., Hogasten, Nicholas, Scott, Jeffrey S., Strademar, Katrin, Nussmeier, Mark, Kurth, Eric A.
• Primary Examiner(s): Bekele, Mekonen
• Application Number:
  US14/138,058
• Publication Number:
  US 20150358560A1
• Time in Patent Office: 1,921 Days
• Field of Search: 348164, 348159, 34833302, 382159
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/56 provided with illuminating means
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 25/671 for non-uniformity detection or correction
• CPC Class Codes: H04N 25/677 for reducing the column or line fixed pattern noise
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Techniques using small form factor infrared imaging modules are disclosed. An imaging system may include visible spectrum imaging modules, infrared imaging modules, illumination modules, and other modules to interface with a user and/or a monitoring system. Visible spectrum imaging modules and infrared imaging modules may be positioned in proximity to a scene that will be monitored while visible spectrum-only images of the scene are either not available or less desirable than infrared images of the scene. Imaging modules may be configured to capture images of the scene at different times. Image analytics and processing may be used to generate combined images with infrared imaging features and increased detail and contrast. Selectable aspects of non-uniformity correction processing, true color processing, and high contrast processing, may be performed on the captured images. Control signals based on the combined images may be presented to a user and/or a monitoring system.

Claims

1. A system comprising:

a housing;

a visible spectrum sensor assembly disposed within the housing and comprising a focal plane array (FPA) of visible spectrum sensors and adapted to capture visible spectrum images of a scene;

an infrared sensor assembly disposed within the housing and comprising a FPA of infrared sensors and adapted to capture infrared images of the scene;

a singular and discrete common circuit board disposed within the housing and physically connected to both the visible spectrum sensor assembly and the infrared sensor assembly, wherein the visible spectrum sensor assembly and the infrared sensor assembly are directly connected and parallel to and securedly fixed in position adjacent to each other by a first planar surface of the common circuit board to provide permanent alignment of the visible spectrum sensor assembly and the infrared sensor assembly to each other, such that the visible spectrum images and the infrared images are substantially rotationally aligned to each other due to the direct connection of the visible spectrum sensor assembly and the infrared sensor assembly to the first planar surface of the common circuit board; and

a logic device disposed within the housing and electrically coupled to the visible spectrum sensor assembly and the infrared sensor assembly through the common circuit board, wherein the logic device is adapted to;

receive at least one visible spectrum image captured by the visible spectrum sensor assembly directly connected to the first planar surface of the common circuit board, and

receive at least one infrared image captured by the infrared sensor assembly directly connected to the first planar surface of the common circuit board.

2. The system of claim 1, further comprising a memory in communication with the logic device, wherein the logic device is adapted to:

store the at least one visible spectrum image and the at least one infrared image to the memory in a common data file according to a common data format, wherein the common data format comprises at least one component corresponding to a radiometric component of the at least one infrared image.

3. The system of claim 2, wherein:

the common data format comprises a Red-Green-Blue-Temperature (RGBT) format encoding RGB components of the at least one visible spectrum image and a radiometric temperature component of the at least one infrared image.

4. The system of claim 1, wherein:

the logic device is adapted to register the images to each other by performing interpolation, scaling, cropping, morphing and/or filtering operations on one or more of the images to substantially match common spatial content within the images;

the visible spectrum sensor assembly comprises a first field of view, the infrared sensor assembly comprises a second field of view, and wherein the first field of view and the second field of view substantially overlap; and

the housing comprises a plurality of electrical connections embedded within the housing and configured to electrically connect the logic device to the common circuit board, wherein heat generated by the visible spectrum sensor assembly and the infrared sensor assembly is dissipated through the common circuit board and the housing via the electrical connections.

5. The system of claim 1, further comprising:

an illuminator in communication with the logic device and adapted to illuminate at least a portion of the scene in the visible and/or infrared spectrum; and

a user interface in communication with the logic device, wherein the logic device is adapted to;

receive user input from the user interface indicating a portion-of-interest of the scene imaged by the infrared sensor assembly,

control the illuminator to illuminate at least the portion-of-interest in a spectrum sensed by the visible spectrum sensor assembly,

receive at least one illuminated captured image of the portion-of-interest from the visible spectrum sensor assembly, and

generate a combined image comprising illuminated characteristics of the scene derived from the at least one illuminated captured image.

6. The system of claim 1, further comprising an additional sensor assembly physically connected to the common circuit board and directly connected and parallel to the first planar surface of the common circuit board to provide permanent alignment of the additional sensor assembly to the visible spectrum sensor assembly and the infrared sensor assembly, wherein the logic device is adapted to:

receive at least one additional image of the scene from the additional sensor assembly; and

generate a combined image comprising stereo imaging characteristics of the scene derived from the at least one additional image, the at least one visible spectrum image, and/or the at least one infrared image.

7. The system of claim 1, wherein:

the common circuit board, the visible spectrum sensor assembly, and the infrared sensor assembly are disposed within a mobile telephone, a tablet computing device, a personal digital assistant, a visible light camera, a music player, or a portable host device.

8. The system of claim 1, wherein:

the visible spectrum sensor assembly and the infrared sensor assembly are implemented with a common aperture and/or optical elements adapted to provide a common view of the scene for the visible spectrum sensor assembly and the infrared sensor assembly, and wherein the common aperture and/or optical elements are disposed within the housing.

9. The system of claim 1, wherein:

the logic device is implemented as a common processing module for the visible spectrum sensor assembly and the infrared sensor assembly; and

the logic device, the visible spectrum sensor assembly, and the infrared sensor assembly are arranged in a staggered arrangement on the common circuit board.

10. The system of claim 1, wherein the logic device is adapted to:

receive control parameters;

derive color characteristics of the scene from at least one of the images;

derive high spatial frequency content from at least one of the images; and

generate a combined image comprising relative contributions of the color characteristics and the high spatial frequency content, wherein;

the relative contributions are determined by the control parameters;

the at least one infrared image comprises one or more unblurred infrared images of the scene corrected by a plurality of non-uniform correction (NUC) terms used to remove noise from the one or more unblurred infrared images; and

the NUC terms are determined based on an intentionally blurred infrared image of the scene.

11. The system of claim 1, wherein:

the at least one visible spectrum image is captured substantially at a first time;

the at least one infrared image comprises a radiometric component and is captured substantially at a second time; and

the logic device is adapted to process the at least one visible spectrum image and the at least one infrared image to generate a combined image comprising visible spectrum characteristics of the scene derived from the at least one visible spectrum image and infrared characteristics of the scene derived from the radiometric component of the infrared image.

12. The system of claim 1, wherein:

the FPA of infrared sensors comprises an array of microbolometers adapted to receive a bias voltage selected from a range of approximately 0.2 volts to approximately 0.7 volts;

a size of the array of microbolometers is approximately 80 by 60; and

the infrared sensor assembly is adapted to be inserted into a socket having a size less than approximately 8.5 mm by 8.5 mm.

13. A method comprising:

receiving visible spectrum radiation from a scene at a focal plane array (FPA) of visible spectrum sensors of a visible spectrum sensor assembly disposed within a housing;

receiving infrared radiation from the scene at a FPA of infrared sensors of an infrared sensor assembly disposed within the housing, the visible spectrum sensor assembly and the infrared sensor assembly being aligned to each other by a singular and discrete common circuit board disposed within the housing and physically connected to both the visible spectrum sensor assembly and the infrared sensor assembly, wherein the visible spectrum sensor assembly and the infrared sensor assembly are directly connected and parallel to and securedly fixed in position adjacent to each other by a first planar surface of the common circuit board to provide permanent alignment of the visible spectrum sensor assembly and the infrared sensor assembly to each other, such that visible spectrum images captured by the visible spectrum sensor assembly and infrared images captured by the infrared sensor assembly are substantially rotationally aligned to each other due to the direct connection of the visible spectrum sensor assembly and the infrared sensor assembly to the first planar surface of the common circuit board;

generating at least one visible spectrum image of the scene in response to receiving the visible spectrum radiation;

generating at least one infrared image of the scene in response to receiving the infrared radiation; and

producing an output signal comprising data corresponding to at least one of the generated images.

14. The method of claim 13, further comprising:

storing the data corresponding to the at least one visible spectrum image and the at least one infrared image to a memory in a common data file according to a common data format, wherein the common data format comprises at least one component corresponding to a radiometric component of the at least one infrared image.

15. The method of claim 14, wherein:

the common data format comprises a Red-Green-Blue-Temperature (RGBT) format encoding RGB components of the at least one visible spectrum image and a radiometric temperature component of the at least one infrared image.

16. The method of claim 13, further comprising registering the generated images to each other by performing interpolation, scaling, cropping, morphing and/or filtering operations on one or more of the images to substantially match common spatial content within the images, wherein:

the visible spectrum sensor assembly comprises a first field of view, the infrared sensor assembly comprises a second field of view, and wherein the first field of view and the second field of view substantially overlap; and

the housing comprises a plurality of electrical connections embedded within the housing and configured to electrically connect a logic device to the common circuit board, wherein heat generated by the visible spectrum sensor assembly and the infrared sensor assembly is dissipated through the common circuit board and the housing via the electrical connections.

17. The method of claim 13, further comprising:

receiving user input from a user interface indicating a portion-of-interest of the scene imaged by the infrared sensor assembly;

controlling an illuminator to illuminate at least the portion-of-interest in a spectrum sensed by the visible spectrum sensor assembly, wherein the illuminator is adapted to illuminate at least a portion of the scene in the visible and/or infrared spectrum;

generating at least one illuminated captured image of the portion-of-interest by the visible spectrum sensor assembly; and

generating a combined image comprising illuminated characteristics of the scene derived from the at least one illuminated captured image.

18. The method of claim 13, further comprising:

generating at least one additional image of the scene from an additional sensor assembly physically connected to the common circuit board and directly connected to the first planar surface of the common circuit board to provide permanent alignment of the additional sensor assembly to the visible spectrum sensor assembly and the infrared sensor assembly; and

generating a combined image comprising stereo imaging characteristics of the scene derived from the at least one additional image, the at least one visible spectrum image, and/or the at least one infrared image.

19. The method of claim 13, wherein:

the common circuit board, the visible spectrum sensor assembly, and the infrared sensor assembly are disposed within a mobile telephone, a tablet computing device, a personal digital assistant, a visible light camera, a music player, or a portable host device.

20. The method of claim 13, wherein:

the at least one visible spectrum image and the at least one infrared image are generated by receiving the infrared radiation and the visible spectrum radiation through a common aperture and/or optical elements adapted to provide a common view of the scene for the visible spectrum sensor assembly and the infrared sensor assembly, and wherein the common aperture and/or optical elements are disposed within the housing.

21. The method of claim 13, wherein:

the images are generated and the output signal is produced by a common processing module, the visible spectrum sensor assembly, and the infrared sensor assembly arranged in a staggered arrangement on the common circuit board.

22. The method of claim 13, further comprising:

receiving control parameters;

deriving color characteristics of the scene from at least one of the images;

deriving high spatial frequency content from at least one of the images; and

generating a combined image comprising relative contributions of the color characteristics and the high spatial frequency content, wherein;

the relative contributions are determined by the control parameters,

the at least one infrared image comprises one or more unblurred infrared images of the scene corrected by a plurality of non-uniform correction (NUC) terms used to remove noise from the one or more unblurred infrared images; and

the NUC terms are determined based on an intentionally blurred infrared image of the scene.

23. The method of claim 13, wherein:

the at least one visible spectrum image is generated substantially at a first time; and

the at least one infrared image comprises a radiometric component and is generated substantially at a second time, the method further comprising;

processing the at least one visible spectrum image and the at least one infrared image to generate a combined image comprising visible spectrum characteristics of the scene derived from the at least one visible spectrum image and infrared characteristics of the scene derived from the radiometric component of the infrared image.

24. The method of claim 13, wherein:

the FPA of infrared sensors comprises an array of microbolometers adapted to receive a bias voltage selected from a range of approximately 0.2 volts to approximately 0.7 volts;

a size of the array of microbolometers is approximately 80 by 60; and

the infrared sensor assembly is adapted to be inserted into a socket having a size less than approximately 8.5 mm by 8.5 mm.

25. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which when executed by one or more processors of a system are adapted to cause the system to perform a method comprising:

receiving at least one visible spectrum image of a scene captured by a visible spectrum sensor assembly disposed within a housing and comprising a focal plane array (FPA) of visible spectrum sensors;

receiving at least one infrared image of the scene captured by an infrared sensor assembly disposed within the housing and comprising a FPA of infrared sensors, the visible spectrum sensor assembly and the infrared sensor assembly being aligned to each other by a singular and discrete common circuit board disposed within the housing and physically connected to both the visible spectrum sensor assembly and the infrared sensor assembly, wherein the visible spectrum sensor assembly and the infrared sensor assembly are directly connected and parallel to and securedly fixed in position adjacent to each other by a first planar surface of the common circuit board to provide permanent alignment of the visible spectrum sensor assembly and the infrared sensor assembly to each other, such that visible spectrum images captured by the visible spectrum sensor assembly and infrared images captured by the infrared sensor assembly are substantially rotationally aligned to each other due to the direct connection of the visible spectrum sensor assembly and the infrared sensor assembly to the first planar surface of the common circuit board; and

generating a combined image comprising visible spectrum characteristics of the scene derived from the at least one visible spectrum image and infrared characteristics of the scene derived from the at least one infrared image.