Method, an apparatus and a computer-readable medium for processing a night vision image dataset

• US 8,139,889 B2
• Filed: 08/16/2007
• Issued: 03/20/2012
• Est. Priority Date: 08/16/2006
• Status: Active Grant

• Current Assignee: Eric Warrant, Henrik Malm, Magnus Oskarsson, Toyota Jidosha Kabushiki Kaisha (Toyota Motor Corporation)
• Original Assignee: Lund University, Toyota Motor Europe SA (Toyota Motor Corporation)
• Inventors: Malm, Henrik, Warrant, Eric, Ambeck-Madsen, Jonas, Yanagihara, Hiromichi, Oskarsson, Magnus
• Primary Examiner(s): Liew, Alex
• Application Number:
  US12/377,793
• Publication Number:
  US 20100284626A1
• Time in Patent Office: 1,678 Days
• Field of Search: 382124-127, 382254-275
• US Class Current: 382/261
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 5/40 using histogram techniques
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/94 based on local image properties, e.g. for local contrast enhancement

Abstract

Method for processing a sequence of at least two image datasets each having at least two pixels, each pixel having an intensity value by calculating a structure tensor for each pixel in an image dataset included in the sequence of image datasets; calculating values in a summation kernel based on the structure tensor for each pixel in the image dataset; calculating a weighted intensity value for each pixel in the first image dataset, using as weights the values in the summation kernel; storing the weighted intensity value for each pixel in the image dataset as a processed intensity value for each corresponding pixel in a processed output image dataset; rotating a local coordinate system in which the summation kernel is described so that the coordinate axes of said local coordinate system coincide with the directions of the eigenvectors of the structure tensor.

Claims

1. A method for processing a sequence of night vision image datasets, wherein said sequence comprises at least two image datasets each having at least two pixels, wherein each pixel has an intensity value, said method comprising:

calculating, based on estimated intensity derivatives in each pixel, a structure tensor for each pixel in an image dataset comprised in said sequence of image datasets, and performing an eigen-decomposition of said structure tensor,

calculating values in a summation kernel based on said structure tensor for each pixel in said image dataset,

calculating a weighted intensity value for each pixel in said image dataset, using as weights the values in said summation kernel,

storing said weighted intensity value for each pixel in said image dataset as a processed intensity value for each corresponding pixel in a processed output image dataset,

the method furthermore comprising;

rotating a local coordinate system in which the summation kernel is described resulting in that the coordinate axes of said local coordinate system coincide with the directions of the eigenvectors of said structure tensor, where said eigenvectors are described in the global coordinate system of the image dataset, and

scaling the coordinate axes of the local coordinate system in which the summation kernel is described by an amount related to the eigenvalues of the structure tensor via a width function w(λ

_i)=σ

_i, and wherein said eigenvalues depend on the amount of intensity variation in the direction of their corresponding eigenvectors,

and in addition the width function being a decreasing function depending on the noise level in each pixel such that w(0)=σ

_max and lim<sub>a→

∞</sub>

w(a)=σ

_min.

2. The method of claim 1 wherein the width function is defined by

3. The method according to claim 1, wherein adapting the width function to the noise level comprises estimating a noise in the image datasets and scaling an estimation of the variance of the noise so that the result is of a magnitude that is suitable for insertion in the width function.

4. The method according to claim 3, wherein scaling an estimation of the variance is performed by means of a fixed scaling factor, or wherein scaling an estimation of the variance is performed by means of a scaling factor which is determined from the image dataset.

5. The method according to claim 1, comprising:

repeating the method steps for subsequent image datasets.

6. The method according to claim 1, wherein said step of calculating a structure tensor comprises:

calculating the structure tensor based on the mean of the corresponding estimated intensity derivatives for each channel in each pixel when said image dataset is a multi-channel image dataset, or

calculating said structure tensor for each channel in said image dataset when said image dataset is a multi-channel image dataset.

7. The method according to claim 1, further comprising:

calculating the elementwise mean structure tensor of said structure tensors calculated for each channel in said image dataset when said image dataset is a multi-channel image dataset, or

calculating said structure tensor using the mean value of the values in each channel in said dataset when said image dataset is a multi-channel image data set.

8. The method according to claim 1, further comprising:

performing low-pass filtering of said image dataset before or during said step of calculating the structure tensor.

9. The method according to claim 1, wherein said step of calculating of the structure tensor further comprises:

calculating the motion vectors for each pixel of each image dataset of said sequence of image datasets.

10. The method according to claim 1, further comprising:

performing an intensity transformation on said image dataset by an intensity transformation function to increase the intensity contrast in at least a part of said first image dataset.

11. The method claim 1, further comprising:

rendering each of said first output image dataset for presentation on a display.

12. The method according to claim 1, further comprising performing a foreground-background segmentation or sharpening of the image.

13. The method according to claim 12, wherein the sharpening is obtained by convolution of the summation kernel with a high-boost filter giving the summation kernel S_H or wherein the sharpening is obtained by using a summation function S_h obtained by approximating a discrete high-boost filter with a continuous function h and calculating a new summation function analytically by the convolution of the original summation kernel S with the function h, and discretizing the summation function S_h, giving the summation kernel S_h.

14. An apparatus for processing a sequence of night vision image datasets, said sequence comprising at least two image datasets each having at least two pixels, wherein each pixel has an intensity value, said apparatus comprising:

a first calculation unit for calculating, based on estimated intensity derivatives in each pixel, a structure tensor for each pixel in an image dataset comprised in said sequence of image datasets, and performing an eigen-decomposition of said structure tensor,

a second calculation unit for calculating values in a summation kernel based on said structure tensor for each pixel in said image dataset,

a third calculation unit for calculating a weighted intensity value for each pixel in said image dataset, using as weights the values in said summation kernel,

a storage unit for storing said weighted intensity value for each pixel in said image dataset as a processed intensity value for each corresponding pixel in a processed output image dataset,

wherein the apparatus further comprises;

a calculation unit for rotating a local coordinate system in which the summation kernel is described resulting in that the coordinate axes of said local coordinate system coincide with the directions of the eigenvectors of said structure tensor, where said eigenvectors are described in the global coordinate system of the image dataset, and

a scaling unit for scaling the coordinate axes of the local coordinate system in which the summation kernel is described by an amount related to the eigenvalues of the structure tensor via a width function w(λ

_i)=σ

_i, and wherein said eigenvalues depend on the amount of intensity variation in the direction of their corresponding eigenvectors,

and in addition the width function being a decreasing function depending on the noise level in each pixel such that w(0)=σ

_max and lim<sub>a→

∞</sub>

w(a)=σ

_min.

15. The apparatus of claim 14, the width function being a decreasing function defined by

16. The apparatus according to claim 15, further comprising a repeating unit for repeating the above steps for subsequent image datasets.

17. A computer-readable medium having embodied thereon a computer program for processing by a processor of a sequence of image datasets, wherein said sequence comprises at least two image datasets, each image dataset having at least two pixels, wherein each pixel has an intensity value, said computer program comprising

a first calculation code segment for calculating, based on estimated intensity derivatives in each pixel, a structure tensor for each pixel in an image dataset comprised in said sequence of image datasets, and for performing an eigen-decomposition of said structure tensor,

a second calculation code segment for calculating values in a summation kernel based on said structure tensor for each pixel in said image dataset,

a third calculation code segment for calculating a weighted intensity value for each pixel in said image dataset, using as weights the values in said summation kernel,

a storage code segment for storing said weighted intensity value for each pixel in said image dataset as a processed intensity value for each corresponding pixel in a processed output image dataset,

a further calculation code segment for rotating a local coordinate system in which the summation kernel is described resulting in that the coordinate axes of said local coordinate system coincide with the directions of the eigenvectors of said structure tensor, where said eigenvectors are described in the global coordinate system of the image dataset, and

a scaling code segment for scaling the coordinate axes of the local coordinate system in which the summation kernel is described by an amount related to the eigenvalues of the structure tensor via a width function w(λ

_i)=σ

_i, and wherein said eigenvalues depend on the amount of intensity variation in the direction of their corresponding eigenvectors,

and in addition the width function being a decreasing function depending on the noise level in each pixel such that w(0)=σ

_max and lim<sub>a→

∞</sub>

w(a)=σ

_min.

18. A computer-readable medium according to claim 17, the width function being a decreasing function defined by

19. The computer-readable medium according to claim 17, further comprising a repeating code segment for repeating the above steps for subsequent image datasets.