Systems and approaches for thermal image corrections

• US 10,129,490 B2
• Filed: 04/05/2016
• Issued: 11/13/2018
• Est. Priority Date: 04/05/2015
• Status: Active Grant

• Current Assignee: ECB Consulting LLC
• Original Assignee: Hema Imaging LLC
• Inventors: Beall, Erik
• Primary Examiner(s): Ustaris, Joseph
• Assistant Examiner(s): Sechser, Jill
• Application Number:
  US15/091,383
• Publication Number:
  US 20160292826A1
• Time in Patent Office: 952 Days
• CPC Class Codes: G01J 2005/0077 Imaging
• CPC Class Codes: G06K 7/10297 arrangements for handling protocols designed for non-contact record carriers such as RFIDs NFCs, e.g. ISO/IEC 14443 and 18092 cryptographic protocols H04L9/00; network security protocols H04L63/00; real-time communication protocols in data switching networks H04L65/00; network protocols for data switching network services H04L67/00
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2207/30208 Marker matrix
• CPC Class Codes: G06T 2207/30244 Camera pose
• CPC Class Codes: G06T 3/60 Rotation of whole images or parts thereof
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/0004 Industrial image inspection
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: G06T 7/33 using feature-based methods
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/80 Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration
• CPC Class Codes: G06V 10/143 Sensing or illuminating at different wavelengths
• CPC Class Codes: G06V 20/52 Surveillance or monitoring of activities, e.g. for recognising suspicious objects recognising microscopic objects G06V20/69
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

A thermal imaging system includes a thermal imaging sensor configured to capture thermal images of assets in an environment, a non-thermal imaging sensor coupled to the thermal sensor and being configured to capture non-thermal images of the assets, an image alignment system, an image correction system, and a controller configured to control operation of the thermal and non-thermal imaging sensors, the image alignment system, and the image correction system. The controller is configured to determine, using the image correction system, that data representative of a reflective surface is present in a present thermal view of a particular asset, cause the thermal and non-thermal imaging sensors to capture pose-corrected images, generate pose corrected thermal and non-thermal images, with at least one having corrected thermal data corresponding to a location of the reflective surface, and associate the corrected thermal data with the at least one reflective surface.

Claims

1. A thermal imaging system, the system comprising

a thermal imaging sensor configured to capture a plurality of thermal images containing thermal data of a plurality of assets in an environment;

a non-thermal imaging sensor coupled to the thermal sensor, the non-thermal imaging sensor configured to capture a plurality of non-thermal images of the plurality of assets;

an image alignment system;

an image correction system; and

a controller configured to control operation of the thermal imaging sensor, the non-thermal imaging sensor, the image alignment system, and the image correction system by;

(i) determining, using the image correction system, that data representative of a thermally reflective surface is present in a present thermal view of a particular asset by identifying at least one portion of the surface of the particular asset having thermal data corresponding to a location different than the at least one portion of the surface of the particular asset, wherein the at least one portion of the surface of the particular asset having thermal data corresponding to the location different than the at least one portion of the surface of the particular asset corresponds to the thermally reflective surface,

(ii) causing the thermal imaging sensor to capture a plurality of thermal images and the non-thermal imaging sensor to capture a plurality of non-thermal images of the particular asset at a plurality of poses,

(iii) generating, using the image alignment system, a plurality of pose-corrected thermal images and a plurality of pose-corrected non-thermal images from the plurality of thermal images and the plurality of non-thermal images, at least one of the pose-corrected thermal images including corrected thermal data corresponding to a location of the reflective surface, and

(iv) using the image correction system and the plurality of thermal and non-thermal images, associating the corrected thermal data corresponding to the location of the reflective surface with the at least one reflective surface present in the present thermal view.

2. The thermal imaging system of claim 1, wherein the image correction system determines that data representative of a reflected surface is present by identifying an inconsistency between image motion in the captured thermal and non-thermal images using an image motion estimation technique.

3. The thermal imaging system of claim 2, wherein the image motion estimation technique comprises at least one of a feature detection and an optical flow detection that computes vector motion of a set of individual pixels within the captured plurality of thermal images.

4. The thermal imaging system of claim 3, wherein the image alignment system determines a homography between the present thermal view and the captured plurality of thermal images.

5. The thermal imaging system of claim 4, wherein the image correction system determines an optical flow for each of the plurality of pose-corrected thermal images and computes an optical flow error for each of the plurality of pose-corrected thermal images.

6. The thermal imaging system of claim 5, wherein the image correction system uses a minimum thermal value calculated from the optical flow error to remove the thermal data corresponding to at least one reflection from the present thermal view.

7. The thermal imaging system of claim 3, wherein the image alignment system determines a homography between the present non-thermal view and the plurality of pose-corrected non-thermal images, and the image correction system determines an optical flow for each of the plurality of pose-corrected non-thermal images and computes an accumulated optical flow error of the plurality of pose-corrected non-thermal images.

8. The thermal imaging system of claim 1, wherein the image correction system generates a statistic of pose corrected pixels and uses the statistic to replace the data representative of the reflective surface from the present thermal view.

9. The thermal imaging system of claim 1, wherein the plurality of pose-corrected thermal images and the plurality of pose-corrected non-thermal images are captured along a hemispherical path relative to the particular asset.

10. The thermal imaging system of claim 1, wherein the image correction system determines whether the at least one reflection is present by receiving an input from a user.

11. The thermal imaging system of claim 1, wherein the controller is further configured to cause the non-thermal imaging sensor and the thermal imaging sensor to capture a plurality of images in an opposite direction from the particular asset to be used by the image correction system.

12. A method of correcting thermal images in an environment, the method comprising:

obtaining at least one present thermal view of at least one asset using a thermal imaging sensor and at least one present non-thermal view of the at least one asset using a non-thermal imaging sensor;

determining, using the image correction system, that data representative of a thermally reflective surface is present in a present thermal view of a particular asset by identifying at least one portion of the surface of the particular asset having thermal data corresponding to a location different than the at least one portion of the surface of the particular asset, wherein the at least one portion of the surface of the particular asset having thermal data corresponding to the location different than the at least one portion of the surface of the particular asset corresponds to the thermally reflective surface;

capturing a plurality of thermal and non-thermal images of the at least one asset from a plurality of poses;

generating, using the image alignment system, a plurality of pose-corrected thermal images and a plurality of pose-corrected non-thermal images from the plurality of thermal images and the plurality of non-thermal images, at least one of the pose-corrected thermal images including corrected thermal data corresponding to a location of the reflective surface;

associating, using the image correction system and the plurality of thermal and non-thermal images, the corrected thermal data corresponding to the location of the reflective surface with the at least one reflective surface present in the present thermal view.

13. The method of claim 12, wherein determining that data representative of a reflective surface is present further comprises determining a discordance between the present thermal view and the present non-thermal view.

14. The method of claim 13, wherein determining the discordance comprises at least one of performing a feature detection and an optical flow detection.

15. The method of claim 14, wherein adjusting the orientation of the plurality of thermal and non-thermal images comprises determining a homography between the present thermal view and the plurality of captured thermal images.

16. The method of claim 15, wherein determining the adjusted temperature value of the at least one reflection comprises determining an optical flow for each of the plurality of thermal images and computing an accumulated optical flow error for each of the plurality of thermal images.

17. The method of claim 12, wherein the image correction system generates a statistic of pose corrected pixels and uses the statistic to replace the data representative of the reflective surface from the present thermal view.

18. The method of claim 12, wherein the plurality of thermal images and the plurality of non-thermal images are captured along a hemispherical path relative to the particular asset.

19. The method of claim 12, wherein identifying the reflection comprises manually notifying the image correction system.

20. The method of claim 12, further comprising capturing a plurality of thermal images and a plurality of non-thermal images in an opposite direction from the particular asset to be used by the image correction system.