Systems and approaches for thermal imaging

• US 10,116,884 B2
• Filed: 04/05/2016
• Issued: 10/30/2018
• Est. Priority Date: 04/05/2015
• Status: Active Grant

• Current Assignee: Hema Imaging LLC
• Original Assignee: Hema Imaging LLC
• Inventors: Beall, Erik, Popescu, Marius
• Primary Examiner(s): Ustaris, Joseph
• Assistant Examiner(s): Sechser, Jill
• Application Number:
  US15/091,068
• Publication Number:
  US 20160295134A1
• Time in Patent Office: 938 Days
• CPC Class Codes: G01J 2005/0077 Imaging
• CPC Class Codes: G06K 7/10297 arrangements for handling protocols designed for non-contact record carriers such as RFIDs NFCs, e.g. ISO/IEC 14443 and 18092 cryptographic protocols H04L9/00; network security protocols H04L63/00; real-time communication protocols in data switching networks H04L65/00; network protocols for data switching network services H04L67/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/23 from thermal infrared radiation
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A thermal imaging system includes a thermal imaging sensor adapted to capture any number of thermal images of any number of assets in an environment, a non-thermal imaging sensor coupled to the thermal sensor and being adapted to capture any number of non-thermal images of the assets, an asset identification system adapted to identify a particular asset from the assets, an image alignment system, and a controller adapted to control operation of the thermal imaging sensor, the non-thermal imaging sensor, the asset identification system, and the image alignment system. Upon identifying the particular asset via the asset identification system, the controller is adapted to control the image alignment system to adjust an orientation of the thermal image and the non-thermal image and cause the thermal imaging sensor and the non-thermal imaging sensor to capture the image.

Claims

1. A thermal imaging system, the system comprising:

a thermal imaging sensor configured to capture a plurality of thermal images of a plurality of assets in an environment;

a non-thermal imaging sensor coupled to the thermal sensor, the non-thermal imaging sensor configured to capture a plurality of non-thermal images of the plurality of assets;

an asset identification system configured to identify a particular asset from the plurality of assets, the particular asset being previously identified by being input into the asset identification system, the asset identification system identifying the particular asset by instructing a user to capture a unique identifier image associated with the particular asset and comparing the capture to a plurality of baseline unique identifier images;

an image alignment system; and

a controller configured to control operation of the thermal imaging sensor, the non-thermal imaging sensor, the asset identification system, and the image alignment system;

wherein when the asset identification system identifies the particular asset, the controller is configured to (i) present at least one of a previously-captured baseline image or a template image of the particular asset to the user via a display such that the user may approximate an orientation of a present thermal view of the particular asset and a present non-thermal view of the particular asset to the at least one of the previously-captured baseline image or template image, (ii) control the image alignment system to adjust the orientation of the present thermal view of the particular asset and the present non-thermal view of the particular asset to match the at least one of the previously-captured baseline image or template image, and (iii) cause the thermal imaging sensor to capture a thermal image of the present thermal view of the particular asset and the non-thermal imaging sensor to capture a non-thermal image of the present non-thermal view of the particular asset.

2. The thermal imaging system of claim 1, wherein the asset identification system is configured to scan for identifying information using the non-thermal imaging sensor.

3. The thermal imaging system of claim 2, wherein the identifying information comprises at least one of a barcode, a name plate displaying visible characters and/or patterns, and other spatial pattern coding systems.

4. The thermal imaging system of claim 2, wherein the asset identification system is further configured to access a memory module and compare the scanned identifying information with data stored on the memory module to identify the particular asset.

5. The thermal imaging system of claim 1, wherein the asset identification system is configured to scan for identifying information using a radio frequency identification communication system.

6. The thermal imaging system of claim 5, wherein the identifying information comprises a radio frequency identification tag.

7. The thermal imaging system of claim 1, wherein the image alignment system is configured to apply image processing to a previously captured non-thermal image of the particular asset to establish a baseline image to compare to the present non-thermal view of the particular asset.

8. The thermal imaging system of claim 7, wherein the image alignment system is configured to apply image processing to a recently-captured non-thermal image.

9. The thermal imaging system of claim 8, wherein the image processing comprises at least one of a spatial transformation, a histogram normalization, an image segmentation, a thresholding, a spatial smoothing, and an edge detection.

10. The thermal imaging system of claim 7, wherein the image alignment system is configured to display the previously captured non-thermal image while displaying the present non-thermal view of the particular asset.

11. The thermal imaging system of claim 10, wherein the image alignment system is further configured to perform an image difference cost function to compare the processed previously captured non-thermal image to the captured non-thermal image.

12. The thermal imaging system of claim 11, wherein the image difference cost function is computed using at least one of a least-squares difference, a normalized mutual information, and a local Pearson correlation.

13. The thermal imaging system of claim 12, wherein the image alignment system computes an image coregistration that minimizes the cost function.

14. The thermal imaging system of claim 1, wherein the image alignment system is configured to access a memory module and retrieve a previously captured and stored non-thermal image of the particular asset to establish a baseline image and apply image processing to the present non-thermal image to compare the baseline non-thermal view with the present non-thermal view.

15. The thermal imaging system of claim 1, wherein the image alignment system is further configured to access a memory module and compare at least one feature derived from the present non-thermal image with feature data obtained from a previously acquired non-thermal image stored on the memory module to determine an image transformation that aligns the images.

16. A method of repeatably acquiring thermal images in an environment, the method comprising:

providing a thermal imaging system comprising a thermal imaging sensor adapted to capture a plurality of thermal images of a plurality of assets, a non-thermal imaging sensor configured to capture a plurality of non-thermal images of a plurality of assets simultaneously with the thermal image sensor, an asset identification system configured to identify a particular asset from the plurality of assets, an image alignment system, and a controller configured to control operation of the thermal imaging sensor, the non-thermal imaging sensor, the asset identification system, and the image alignment system;

identifying, via the asset identification system, the particular asset, the particular asset being previously identified by being input into the asset identification system, the asset identification system identifying the particular asset by instructing a user to capture a unique identifier image associated with the particular asset and comparing the capture to a plurality of baseline unique identifier images;

presenting, with the image alignment system, at least one of a previously-captured baseline image or a template image of the particular asset to the user via a display such that the user may approximate an orientation of a present thermal view of the particular asset and a present non-thermal view of the particular asset to the at least one of the previously-captured baseline image or template image;

adjusting, with the image alignment system, the orientation of the present thermal view of the particular asset and the present non-thermal view of the particular asset to match the at least one of the previously-captured baseline image or template image; and

causing, with the controller, the thermal imaging sensor to capture a thermal image of the present thermal view of the particular asset and the non-thermal imaging sensor to capture a non-thermal image of the present non-thermal view of the particular asset.

17. The method of claim 16, further comprising applying, with the image alignment system, image processing to at least one of a previously captured non-thermal image of the particular asset, a present non-thermal image of the particular asset, and a present non-thermal view of the particular asset.

18. The method of claim 17, further comprising displaying the previously captured non-thermal image while displaying one of the present non-thermal view and the present non-thermal view.

19. The method of claim 18, further comprising performing an image difference cost function that compares the previously captured non-thermal image to the non-thermal image.

20. The method of claim 19, wherein performing the image different cost function comprises computing an image coregistration that minimizes the cost function.

21. The method of claim 20, further comprising accessing a memory module and comparing at least one feature derived from the present non-thermal image with feature data obtained from a previously acquired non-thermal image stored on the memory module, and determining an image transformation to align the previously acquired non-thermal image and the present non-thermal image.

22. The method of claim 21, wherein the image coregistration is decomposed into an image scale component, a rotation component, and a translation component, the components being used to display indicators on the display corresponding to an intended repositioning and re-orienting of the imaging system so as to match a pose of the presently acquired imaging system with a pose of the previously acquired imaging system.

23. The method of claim 16, further comprising:

measuring, with an orientation measuring system, an orientation of the thermal imaging system to annotate the non-thermal image; and

using the orientation measuring system, initializing an image coregistration search routine.

24. The method of claim 16, further comprising applying, with the image alignment system, image processing to at least one of a previously captured thermal image of the particular asset, a present thermal image of the particular asset, and a present thermal view of the particular asset.

25. The method of claim 24, further comprising displaying the previously captured thermal image while displaying one of the present thermal image and the present thermal view.

26. A thermal imaging device, the device comprising:

a housing;

a thermal imaging sensor at least partially arranged on an exterior surface of the housing and configured to capture a plurality of thermal images of a plurality of assets in an environment;

a non-thermal imaging sensor at least partially arranged on the exterior surface of the housing and coupled to the thermal sensor, the non-thermal imaging sensor configured to capture a plurality of non-thermal images of the plurality of assets, the particular asset being previously identified by being input into the asset identification system, the asset identification system identifying the particular asset by instructing a user to capture a unique identifier image associated with the particular asset and comparing the capture to a plurality of baseline unique identifier images;

an asset identification system configured to identify a particular asset from the plurality of assets;

an image alignment system; and

a controller configured to control operation of the thermal imaging sensor, the non-thermal imaging sensor, the asset identification system, and the image alignment system;

wherein when the asset identification system identifies the particular asset, the controller is configured to (i) present at least one of a previously-captured baseline image or a template image of the particular asset to the user via a display such that the user may approximate an orientation of a present thermal view of the particular asset and a present non-thermal view of the particular asset to the at least one of the previously-captured baseline image or template image, (ii) control the image alignment system to adjust the orientation of the present thermal view of the particular asset and the present non-thermal view of the particular asset to match the at least one of the previously-captured baseline image or template image, and (iii) cause the thermal imaging sensor to capture a thermal image of the present thermal view of the particular asset and the non-thermal imaging sensor to capture a non-thermal image of the present non-thermal view of the particular asset.