Optimized video denoising for heterogeneous multisensor system

• US 10,116,851 B2
• Filed: 03/03/2015
• Issued: 10/30/2018
• Est. Priority Date: 03/03/2014
• Status: Active Grant

• Current Assignee: Sagem Defense Securite SA (Safran SA)
• Original Assignee: Sagem Defense Securite SA (Safran SA)
• Inventors: Bezot, Gregoire, Budin, Joel
• Primary Examiner(s): Tran, Sinh
• Assistant Examiner(s): Gebriel, Selam
• Application Number:
  US15/123,508
• Publication Number:
  US 20170078546A1
• Time in Patent Office: 1,337 Days
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10048 Infrared image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 2207/30252 Vehicle exterior; Vicinity of vehicle
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: H04N 23/11 for generating image signals from visible and infrared light wavelengths
• CPC Class Codes: H04N 23/45 for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
• CPC Class Codes: H04N 23/6811 based on the image signal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 5/265 Mixing

Abstract

The description relates in particular to a method for temporal denoising of image sequences with the aid of at least two sensors of different types. The description also relates to an optronics device, a computer program and a storage medium for the implementation of such a method.

Claims

1. A method for temporal denoising of a sequence of images, said method comprising:

/a/ capturing, by a first sensor, a sequence of first images corresponding to a given scene, each first image being divided into elements each associated with a corresponding area of said first image,

/b/ capturing, by a second sensor of a type different from the type of the first sensor, a sequence of second images corresponding to said given scene, each second image corresponding to a first image, each second image being divided into elements each associated with a corresponding area of said second image, each pair of element and associated area of the second image corresponding to a pair of element and associated area of the corresponding first image, and

/c/ obtaining, by calculation circuitry, a first sequence of images derived from the sequence of first images and a second sequence of images derived from the sequence of second images,

/d/ obtaining, by the calculation circuitry, for each area of each of the images of the first and second sequences of images, an associated weight,

/e/ obtaining, by the calculation circuitry, a first weighted sequence of images, in which each element of each image is equal to the corresponding element of the first sequence of images weighted by the weight associated with the area associated with said corresponding element, and a second weighted sequence of images, in which each element of each image is equal to the corresponding element of the second sequence of images weighted by the weight associated with the area associated with said corresponding element,

/f/ obtaining, by the calculation circuitry, a sequence of enhanced images resulting from combining sequences of images comprising the first weighted sequence of images and the second weighted sequence of images,

/g/ obtaining, by the calculation circuitry, a motion estimation based on the obtained sequence of enhanced images,

/h/ obtaining, by the calculation circuitry, based on the calculated motion estimation, a spatial alignment of the images of a sequence of images to be displayed derived from sequences of images corresponding to the given scene and comprising the sequence of first images and the sequence of second images,

/i/ a temporal denoising, by the calculation circuitry, based on the determined spatial alignment of the sequence of images to be displayed.

2. The method according to claim 1, comprising selecting, by a selection circuit, as the sequence of images to be displayed which is denoised in step /i/, one among the sequence of first images and the sequence of second images.

3. The method according to claim 1, comprising, in step /c/:

obtaining, by the calculation circuitry, for each area of each of the images of the sequence of first images and of the sequence of second images, a local average of said area, and

obtaining, by the calculation circuitry, the first sequence of images, comprising subtracting, from each element of each image of the sequence of first images, the local average of the area corresponding to said element, and obtaining the second sequence of images, comprising subtracting, from each element of each image of the sequence of second images, the local average of the area corresponding to said element.

4. The method according to claim 1, wherein the first sensor and the second sensor are not spatially harmonized, the method comprising, in step /c/, obtaining, by the calculation circuitry, the first sequence of images resulting from pre-smoothing the sequence of first images and the second sequence of images resulting from pre-smoothing the sequence of second images.

5. A non-transitory computer-readable storage medium, said medium storing a computer program comprising instructions which, when executed by a processor, lead the processor to implement steps comprising:

/a/ capturing, by a first sensor, a sequence of first images corresponding to a given scene, each first image being divided into elements each associated with a corresponding area of said first image,

/b/ capturing, by a second sensor of a type different from the type of the first sensor, a sequence of second images corresponding to said given scene, each second image corresponding to a first image, each second image being divided into elements each associated with a corresponding area of said second image, each pair of element and associated area of the second image corresponding to a pair of element and associated area of the corresponding first image, and

/c/ obtaining, by calculation circuitry, a first sequence of images derived from the sequence of first images and a second sequence of images derived from the sequence of second images,

/d/ obtaining, by the calculation circuitry, for each area of each of the images of the first and second sequences of images, an associated weight,

/e/ obtaining, by the calculation circuitry, a first weighted sequence of images, in which each element of each image is equal to the corresponding element of the first sequence of images weighted by the weight associated with the area associated with said corresponding element, and a second weighted sequence of images, in which each element of each image is equal to the corresponding element of the second sequence of images weighted by the weight associated with the area associated with said corresponding element,

/f/ obtaining, by the calculation circuitry, a sequence of enhanced images resulting from combining sequences of images comprising the first weighted sequence of images and the second weighted sequence of images,

/g/ obtaining, by the calculation circuitry, a motion estimation based on the obtained sequence of enhanced images,

/h/ obtaining, by the calculation circuitry, based on the calculated motion estimation, a spatial alignment of the images of a sequence of images to be displayed derived from sequences of images corresponding to the given scene and comprising the sequence of first images and the sequence of second images,

/i/ a temporal denoising, by the calculation circuitry, based on the determined spatial alignment of the sequence of images to be displayed.

6. The non-transitory computer-readable storage medium according to claim 5, the steps further comprising selecting, by a selection circuit, as the sequence of images to be displayed which is denoised in step /i/, one among the sequence of first images and the sequence of second images.

7. The non-transitory computer-readable storage medium according to claim 5, the steps further comprising, in step /c/:

obtaining, by the calculation circuitry, for each area of each of the images of the sequence of first images and of the sequence of second images, a local average of said area, and

obtaining, by the calculation circuitry, the first sequence of images, comprising subtracting, from each element of each image of the sequence of first images, the local average of the area corresponding to said element, and obtaining the second sequence of images, comprising subtracting, from each element of each image of the sequence of second images, the local average of the area corresponding to said element.

8. The non-transitory computer-readable storage medium according to claim 5, wherein the first sensor and the second sensor are not spatially harmonized, the steps further comprising, in step /c/, obtaining, by the calculation circuitry, the first sequence of images resulting from pre-smoothing the sequence of first images and the second sequence of images resulting from pre-smoothing the sequence of second images.

9. An optoelectronic device for temporal denoising of a sequence of images, said optoelectronic device comprising:

a first sensor arranged to capture a sequence of first images corresponding to a given scene, each first image being divided into elements each associated with a corresponding area of said first image,

a second sensor of a type different from the type of the first sensor, arranged to capture a sequence of second images corresponding to said given scene, each second image corresponding to a first image, each second image being divided into elements each associated with a corresponding area of said second image, each pair of element and associated area of the second image corresponding to a pair of element and associated area of the corresponding first image,

calculation circuitry arranged to obtain a first sequence of images derived from the sequence of first images and a second sequence of images derived from the sequence of second images,

the calculation circuitry further arranged to obtain, for each area of each of the images of the first and second sequences of images, an associated weight,

the calculation circuitry further arranged to obtain a first weighted sequence of images, in which each element of each image is equal to the corresponding element of the first sequence of images weighted by the weight associated with the area associated with said corresponding element; and

a second weighted sequence of images, in which each element of each image is equal to the corresponding element of the second sequence of images weighted by the weight associated with the area associated with said corresponding element,

the calculation circuitry further arranged to obtain a sequence of enhanced images resulting from combining sequences of images comprising the first weighted sequence of images and the second weighted sequence of images,

the calculation circuitry further arranged to obtain a motion estimation based on the obtained sequence of enhanced images,

the calculation circuitry further arranged to obtain, based on the calculated motion estimation, a spatial alignment of the images of a sequence of images to be displayed derived from sequences of images corresponding to the given scene and comprising the sequence of first images and the sequence of second images,

the calculation circuitry further arranged to perform a temporal denoising, based on the determined spatial alignment of the sequence of images to be displayed.

10. The optoelectronic device according to claim 9, comprising a selection circuit arranged to select, as the sequence of images to be displayed which are to be denoised, one among the sequence of first images and the sequence of second images.

11. The optoelectronic device according to claim 9, the calculation circuitry arranged to obtain the first and second sequences of images is further arranged to calculate, for each area of each of the images of the sequence of first images and of the sequence of second images, a local average of said area, then to obtain the first sequence of images by subtracting, from each element of each image of the sequence of first images, the local average of the area corresponding to said element, and to obtain the sequence of second images by subtracting, from each element of each image of the sequence of second images, the local average of the area corresponding to said element.

12. The optoelectronic device according to claim 9, wherein the first sensor and the second sensor are not spatially harmonized, the calculation circuitry arranged to obtain the first and second sequences of images is further arranged to obtain the first sequence of images resulting from pre-smoothing the sequence of first images and the second sequence of images resulting from pre-smoothing the sequence of second images.