Selective image correction for infrared imaging devices

• US 9,292,909 B2
• Filed: 12/21/2013
• Issued: 03/22/2016
• Est. Priority Date: 06/03/2009
• Status: Active Grant

• Current Assignee: Teledyne FLIR LLC (Teledyne Technologies Incorporated)
• Original Assignee: FLIR Systems, Inc. (Teledyne Technologies Incorporated)
• Inventors: Hgasten, Nicholas, Hoelter, Theodore R., Strandemar, Katrin, Ingerhed, Malin, Nussmeier, Mark, Kurth, Eric A., Boulanger, Pierre, Sharp, Barbara
• Primary Examiner(s): Bayat, Ali
• Application Number:
  US14/138,055
• Publication Number:
  US 20150332441A1
• Time in Patent Office: 822 Days
• Field of Search: 382/264
• US Class Current: 1/1
• CPC Class Codes: B60R 11/04 Mounting of cameras operative during drive; Arrangement of controls thereof relative to the vehicle
• CPC Class Codes: G06F 2218/04 Denoising
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20201 Motion blur correction
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06V 20/56 exterior to a vehicle by using sensors mounted on the vehicle
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/54 Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
• CPC Class Codes: H04N 23/57 Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
• CPC Class Codes: H04N 23/6812 based on additional sensors, e.g. acceleration sensors
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 25/671 for non-uniformity detection or correction
• CPC Class Codes: H04N 5/33 Transforming infrared radiation cameras or camera modules for generating image signals from infrared radiation H04N23/20; circuitry of SSIS for transforming infrared radiation into image signals H04N25/20

Abstract

Techniques are disclosed for systems and methods using small form factor infrared imaging devices to image scenes in proximity to a vehicle. An imaging system may include one or more infrared imaging devices, a processor, a memory, a display, a communication module, and modules to interface with a user, sensors, and/or a vehicle. Infrared imaging devices may be positioned in proximity to, mounted on, installed in, or otherwise fixed relative to a vehicle. Infrared imaging devices may be configured to capture infrared images of scenes in proximity to a vehicle. Various infrared image analytics and processing may be performed on captured infrared images to correct and/or calibrate the infrared images. Monitoring information, notifications, and/or control signals may be generated based on the corrected infrared images and then presented to a user and/or a monitoring and notification system, and/or used to control aspects of the vehicle.

Claims

1. A method comprising:

receiving one or more sensor signals associated with motion of a vehicle;

processing the sensor signals to determine relative motion of an infrared imaging device with respect to a scene imaged by the infrared imaging device, wherein the infrared imaging device is fixed relative to the vehicle; and

selectively performing an infrared image correction process based on the relative motion, wherein the infrared image correction process comprises;

receiving a blurred infrared image from the infrared imaging device, wherein the blurred infrared image comprises blurred infrared image data from the scene and noise introduced by the infrared imaging device;

processing the blurred infrared image to determine a plurality of non-uniformity correction (NUC) terms to reduce a portion of the noise; and

applying the NUC terms to unblurred infrared images captured by the infrared imaging device.

2. The method of claim 1, further comprising applying a high pass filter to the blurred infrared image to substantially remove the blurred infrared image data from the blurred infrared image before the NUC terms are determined.

3. The method of claim 1, wherein the infrared image correction process further comprises:

processing the blurred infrared image to determine a plurality of spatially correlated fixed pattern noise (FPN) terms to reduce a portion of the noise comprising spatially correlated FPN associated with rows or columns of infrared sensors of the infrared imaging device; and

applying the spatially correlated FPN terms to the blurred infrared image before the NUC terms are determined.

4. The method of claim 1, wherein the blurred infrared image is accumulated from a set of infrared images captured while the vehicle is in motion.

5. The method of claim 4, further comprising capturing the set of infrared images based on the determined relative motion.

6. The method of claim 1, wherein:

the relative motion comprises a rotation of the infrared imaging device relative to the scene;

the rotation is in a plane of a focal plane array (FPA) of the infrared imaging device; and

the infrared image correction process comprises determining spatial row or column fixed pattern noise (FPN) terms to correct a portion of FPN in infrared images captured by the infrared imaging device.

7. The method of claim 1, further comprising processing one or more infrared images captured by the infrared imaging device to further determine the relative motion by evaluating pixels of the infrared images, wherein the selectively performing the infrared image correction process is further based on the relative motion determined by the infrared image processing.

8. The method of claim 1, further comprising performing the following if the determined relative motion is insufficient to perform the infrared image correction process:

intentionally defocusing the infrared imaging device;

capturing a defocused infrared image; and

performing the infrared image correction process using the defocused infrared image.

9. The method of claim 1, wherein receiving the sensor signals comprises monitoring one or more motion sensors.

10. The method of claim 1, wherein receiving the sensor signals comprises monitoring an engine sensor, a transmission sensor, a drive shaft sensor, a differential sensor, a rear axle sensor, a front axle sensor, a steering angle sensor, an antenna module, a GPS module, a gyroscope, or an accelerometer.

11. The method of claim 1, wherein:

the infrared imaging device comprises a focal plane array (FPA) configured to capture the infrared images of the scene; and

the FPA comprises an array of microbolometers adapted to receive a bias voltage selected from a range of approximately 0.2 volts to approximately 0.7 volts.

12. The method of claim 11, wherein:

a size of the array of microbolometers is approximately 80 by 60; and

the infrared imaging device is adapted to be inserted into a socket having a size less than approximately 8.5 mm by 8.5 mm.

13. A system comprising:

an infrared imaging device fixed relative to a vehicle and adapted to capture infrared images of a scene; and

a processor adapted to execute instructions to;

receive one or more sensor signals associated with motion of the vehicle,

process the sensor signals to determine motion of the infrared imaging device with respect to the scene, and

selectively perform an infrared image correction process based on the relative motion, wherein;

the relative motion comprises a rotation of the infrared imaging device relative to the scene,

the rotation is in a plane of a focal plane array (FPA) of the infrared imaging device, and

the instructions to perform the infrared image correction process are adapted to cause the processor to determine spatial row or column fixed pattern noise (FPN) terms to correct a portion of FPN in infrared images captured by the infrared imaging device.

14. The system of claim 13, wherein the instructions to perform the infrared image correction process are adapted to cause the processor to:

receive a blurred infrared image from the infrared imaging device, wherein the blurred infrared image comprises blurred infrared image data from the scene and noise introduced by the infrared imaging device;

process the blurred infrared image to determine a plurality of non-uniformity correction (NUC) terms to reduce a portion of the noise; and

apply the NUC terms to unblurred infrared images captured by the infrared imaging device.

15. The system of claim 14, wherein the instructions to perform the infrared image correction process are adapted to cause the processor to apply a high pass filter to the blurred infrared image to substantially remove the blurred infrared image data from the blurred infrared image before the NUC terms are determined.

16. The system of claim 14, wherein the instructions to perform the infrared image correction process are adapted to cause the processor to:

process the blurred infrared image to determine a plurality of spatially correlated fixed pattern noise (FPN) terms to reduce a portion of the noise comprising spatially correlated FPN associated with rows or columns of infrared sensors of the infrared imaging device; and

apply the spatially correlated FPN terms to the blurred infrared image before the NUC terms are determined.

17. The system of claim 14, wherein the blurred infrared image is accumulated from a set of infrared images captured while the vehicle is in motion.

18. The system of claim 17, wherein the set of blurred infrared images are captured based on the determined relative motion.

19. The system of claim 13, wherein:

the processor is adapted to execute instructions to process one or more infrared images captured by the infrared imaging device to further determine the relative motion by evaluating pixels of the infrared images; and

the instructions to selectively perform the infrared image correction process are adapted to cause the processor to selectively perform the infrared image correction process further based on the relative motion determined by the infrared image process.

20. The system of claim 13, wherein the processor is further adapted to execute instructions to:

determine the relative motion is insufficient to perform the infrared image correction process;

intentionally defocus the infrared imaging device; and

perform the infrared image correction process using a defocused infrared image.

21. The system of claim 13, wherein the instructions to receive the sensor signals are adapted to cause the processor to monitor one or more motion sensors.

22. The system of claim 13, wherein the instructions to receive the sensor signals are adapted to cause the processor to monitor an engine sensor, a transmission sensor, a drive shaft sensor, a differential sensor, a rear axle sensor, a front axle sensor, a steering angle sensor, an antenna module, a GPS module, a gyroscope, or an accelerometer.

23. The system of claim 13, wherein:

the infrared imaging device comprises a focal plane array (FPA) configured to capture the infrared images of the scene; and

the FPA comprises an array of microbolometers adapted to receive a bias voltage selected from a range of approximately 0.2 volts to approximately 0.7 volts.

24. The system of claim 23, wherein:

a size of the array of microbolometers is approximately 80 by 60; and

the infrared imaging device is adapted to be inserted into a socket having a size less than approximately 8.5 mm by 8.5 mm.

25. A system comprising:

an infrared imaging device fixed relative to a vehicle and adapted to capture infrared images of a scene; and

a processor adapted to execute instructions to;

receive one or more sensor signals associated with motion of the vehicle,

process the sensor signals to determine relative motion of the infrared imaging device with respect to the scene, and

selectively perform an infrared image correction process based on the relative motion, wherein;

the infrared imaging device comprises a focal plane array (FPA) configured to capture the infrared images of the scene, and

the FPA comprises an array of microbolometers adapted to receive a bias voltage selected from a range of approximately 0.2 volts to approximately 0.7 volts.

26. The system of claim 25, wherein the instructions to perform the infrared image correction process comprise instructions adapted to cause the processor to:

receive a blurred infrared image from the infrared imaging device, wherein the blurred infrared image comprises blurred infrared image data from the scene and noise introduced by the infrared imaging device;

process the blurred infrared image to determine a plurality of non-uniformity correction (NUC) terms to reduce a portion of the noise; and

apply the NUC terms to unblurred infrared images captured by the infrared imaging device.

27. The system of claim 25, wherein:

the relative motion comprises a rotation of the infrared imaging device relative to the scene;

the rotation is in a plane of a focal plane array (FPA) of the infrared imaging device; and

the instructions to perform the infrared image correction process are adapted to cause the processor to determine spatial row or column fixed pattern noise (FPN) terms to correct a portion of FPN in infrared images captured by the infrared imaging device.