Infrared imaging enhancement with fusion

• US 9,635,285 B2
• Filed: 12/21/2013
• Issued: 04/25/2017
• Est. Priority Date: 03/02/2009
• Status: Expired due to Fees

• Current Assignee: FLIR Systems, Inc. (Teledyne Technologies Incorporated)
• Original Assignee: FLIR Systems, Inc. (Teledyne Technologies Incorporated)
• Inventors: Teich, Andrew C., Hgasten, Nicholas, Scott, Jeffrey S., Strandemar, Katrin, Nussmeier, Mark, Kurth, Eric A., Hoelter, Theodore R., Boulanger, Pierre, Sharp, Barbara
• Primary Examiner(s): Shah, Utpal
• Application Number:
  US14/138,052
• Publication Number:
  US 20150334315A1
• Time in Patent Office: 1,221 Days
• Field of Search: None
• CPC Class Codes: G06F 2218/04 Denoising
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/90 Determination of colour characteristics
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 23/84 for processing colour signals
• CPC Class Codes: H04N 25/671 for non-uniformity detection or correction
• CPC Class Codes: H04N 25/674 based on the scene itself, e.g. defocusing
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Techniques using small form factor infrared imaging modules are disclosed. An imaging system may include visible spectrum imaging modules, infrared imaging modules, and other modules to interface with a user and/or a monitoring system. Visible spectrum imaging modules and infrared imaging modules may be positioned in proximity to a scene that will be monitored while visible spectrum-only images of the scene are either not available or less desirable than infrared images of the scene. Imaging modules may be configured to capture images of the scene at different times. Image analytics and processing may be used to generate combined images with infrared imaging features and increased detail and contrast. Triple fusion processing, including selectable aspects of non-uniformity correction processing, true color processing, and high contrast processing, may be performed on the captured images. Control signals based on the combined images may be presented to a user and/or a monitoring system.

Claims

1. A system comprising:

a memory adapted to receive a visible spectrum image of a scene from a visible spectrum imager and a plurality of infrared images of the scene from an infrared imager; and

a processor configured to communicate with the memory, wherein the processor is configured to;

generate a blurred infrared image from at least one of the plurality of infrared images;

determine, for each row of the blurred infrared image, a corresponding row fixed pattern noise (FPN) correction term;

determine, for each column of the blurred infrared image, a corresponding column FPN correction term;

apply the row and column FPN correction terms to the blurred infrared image to provide a row and column FPN corrected blurred infrared image;

determine a plurality of non-uniformity correction (NUC) terms based, at least in part, on the row and column FPN corrected blurred infrared image;

apply the NUC terms to one of the plurality of the infrared images to remove noise from the one of the plurality of the infrared images to provide a corrected infrared image;

receive control parameters;

derive high spatial frequency content from at least one of the visible spectrum image and the corrected infrared image; and

generate a combined image comprising relative contributions of at least the high spatial frequency content, wherein the relative contributions are determined in real-time by a user adjusting the control parameters.

2. The system of claim 1, wherein the generating the blurred infrared image comprises:

detecting motion of the infrared imager based on motion sensor data provided by a motion sensor;

receiving a set of the plurality of infrared in response to detected motion;

generating a blurred infrared image from the set of the plurality of infrared images based on an average of the set of the plurality of infrared images.

3. The system of claim 1, wherein:

the applying the NUC terms to the one of the plurality of infrared images comprises selectively applying the NUC terms to the one of the plurality of infrared images to selectively remove noise from the one of the plurality of infrared images; and

the selectively applying the NUC terms is determined by the control parameters.

4. The system of claim 1, further comprising the infrared imager, wherein the infrared imager is configured to:

generate the blurred infrared image of the scene; and

derive the plurality of NUC terms.

5. The system of claim 1, wherein:

one or more of the control parameters are determined by one or more threshold values associated with at least one of the visible spectrum image, the plurality of infrared images, and the corrected infrared image.

6. The system of claim 1, further comprising a user interface configured to communicate with the memory and the processor, wherein the processor is configured to:

receive user input from the user interface; and

determine one or more of the control parameters from the user input.

7. The system of claim 1, further comprising deriving color characteristics of the scene from at least one of the visible spectrum image and the infrared image, wherein:

the color characteristics of the scene are derived from a chrominance component of the visible spectrum image;

the relative contributions include a luminance component of the corrected infrared image; and

the relative contribution of the luminance component of the corrected infrared image substantially matches the relative contribution of the color characteristics, as controlled by the control parameters.

8. The system of claim 7, wherein:

the luminance component of the corrected infrared image is blended with a luminance component of the visible spectrum image before being contributed to the combined image.

9. The system of claim 1, wherein:

the high spatial frequency content is derived from a luminance component of the visible spectrum image;

the relative contributions include a chrominance component of the corrected infrared image; and

the relative contribution of the chrominance component of the corrected infrared image substantially matches the relative contribution of the high spatial frequency content, as controlled by the control parameters.

10. The system of claim 9, wherein:

the high spatial frequency content is blended with a luminance component of the corrected infrared image before being contributed to the combined image.

11. The system of claim 1, further comprising the infrared imager, wherein:

the infrared imager comprises a focal plane array (FPA) configured to capture the plurality of infrared images of the scene; and

the FPA comprises an array of microbolometers adapted to receive a bias voltage selected from a range of approximately 0.2 volts to approximately 0.7 volts.

24. The system of claim 2, further comprising the visible spectrum imager and the infrared imager, wherein the infrared imager comprises the motion sensor.

12. A method comprising:

receiving a visible spectrum image of a scene from a visible spectrum imager and a plurality of infrared images of the scene from an infrared imager;

generating a blurred infrared image from at least one of the plurality of infrared images;

determining, for each row of the blurred infrared image, a corresponding row fixed pattern noise (FPN) correction term;

determining, for each column of the blurred infrared image, a corresponding column FPN correction term;

applying the row and column FPN correction terms to the blurred infrared image to provide a row and column FPN corrected blurred infrared image;

determining a plurality of non-uniformity correction (NUC) terms based, at least in part, on the row and column FPN corrected blurred infrared image;

applying the NUC terms to one of the plurality of the infrared images to remove noise from the one of the plurality of the infrared images to provide a corrected infrared image;

receiving control parameters;

deriving color characteristics of the scene from at least one of the visible spectrum image and the corrected infrared image;

and

generating a combined image comprising relative contributions of at least the color characteristics, wherein the relative contributions are determined in real-time by a user adjusting the control parameters.

13. The method of claim 12, wherein the generating the blurred infrared image comprises:

detecting motion of the infrared imager based on motion sensor data provided by a motion sensor;

receiving a set of the plurality of infrared images in response to detected motion; and

generating the blurred infrared image from the set of the plurality of infrared images based on an average of the set of the plurality of infrared images.

14. The method of claim 12, wherein the applying the NUC terms to the one of the plurality of infrared images comprises selectively applying the NUC terms to the one of the plurality of infrared images.

15. The method of claim 12, wherein the infrared imager is configured to:

generate the blurred infrared image of the scene by intentionally blurring one of the plurality of infrared images of the scene or by determining a set of the plurality of infrared images corresponding to detected motion and generating the blurred infrared image from the set of the plurality of infrared images based on an average of the set of the plurality of infrared images; and

determine the plurality of NUC terms.

16. The method of claim 12, wherein:

one or more of the control parameters are determined by one or more threshold value;

associated with at least one of the visible spectrum image, the plurality of infrared images, and the corrected infrared image.

17. The method of claim 12, further comprising:

receiving user input from a user interface, wherein one or more of the control parameters are determined by the user input.

18. The method of claim 12, further comprising:

receiving information from a sensor, wherein one or more of the control parameters are determined by the information, wherein the information is associated with an amount of available ambient light.

19. The method of claim 12, wherein:

the color characteristics of the scene are derived from a chrominance component of the visible spectrum image;

the relative contributions include a luminance component of the corrected infrared image; and

the relative contribution of the luminance component of the corrected infrared image substantially matches the relative contribution of the color characteristics, as controlled by the control parameters.

20. The method of claim 19, wherein:

the luminance component of the corrected infrared image is blended with a luminance component of the visible spectrum image before being contributed to the combined image.

21. The method of claim 12, further comprising deriving high spatial frequency content from at least one of the visible spectrum image and the infrared image, wherein:

the high spatial frequency content is derived from a luminance component of the visible spectrum image;

the relative contributions include a chrominance component of the corrected infrared image; and

the relative contribution of the chrominance component of the corrected infrared image substantially matches the relative contribution of the high spatial frequency content, as controlled by the control parameters.

22. The method of claim 21, wherein:

the high spatial frequency content is blended with a luminance component of the corrected infrared image before being contributed to the combined image.

23. A non-transitory machine-readable medium comprising a plurality of machine-readable instructions which when executed by one or more processors of a system are adapted to cause the system to perform a method comprising:

receiving a visible spectrum image of a scene from a visible spectrum imager and a plurality of infrared images of the scene from an infrared imager;

generating a blurred infrared image least one of the plurality of infrared images;

determining, for each row of the blurred infrared image, a corresponding row fixed pattern noise (FPN) correction term;

determining, for each column of the blurred infrared image, a corresponding column FPN correction term;

applying the row and column FPN correction terms to the blurred infrared image to provide a row and column FPN corrected blurred infrared image;

determining a plurality of non-uniformity correction (NUC) terms based, at least in part, on the row and column FPN corrected blurred infrared image;

applying the NUC terms to one of the plurality of the infrared images to remove noise from the one of the plurality of the infrared images to provide a corrected infrared image;

receiving control parameters;

deriving high spatial frequency content from at least one of the visible spectrum image and the corrected infrared image; and

generating a combined image comprising relative contributions of at least the high spatial frequency content, wherein the relative contributions are determined in real-time by a user adjusting the control parameters.