Automatic digital film and video restoration

• US 20080170801A1
• Filed: 03/04/2008
• Published: 07/17/2008
• Est. Priority Date: 09/05/2005
• Status: Active Grant

• Current Assignee: Algosoft-Tech USA, LLC
• Original Assignee: Algosoft Limited
• Inventors: Petukhov, Alexander, Kozlov, Inna, Zheludev, Valery
• Granted Patent: US 7,769,244 B2
• Time in Patent Office: Days
• US Class Current: 382/275
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/20032 Median filtering
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 5/005 Retouching; Inpainting; Scratch removal detecting, correction, reducing or removing defects, e.g. non-responsive pixels of solid state image sensors H04N25/68, scratch removal for cinematographic films scanned by electronic image sensor H04N5/253
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal

Abstract

A method, system, and computer program for automatic restoration of digital film and video frames, by operation of a sequence of five main interactive computer program blocks, mainly separating moving objects from still objects, separating smooth texture from regions containing edges using the coherency map, and detecting and removing defects to provide restored frames using a pair of frames as reference frames for each processed frame.

Claims

1. A method for the fully-automatically correcting an image having defects thereon, by running a software computer program comprising the steps of:

loading parameters in a procedure lunlo2.

preprocessing and processing frames in sequence by operation of a procedure lproc2,

operating a frame-by-frame processing procedure lmain<sub>—

proc2,</sub>

separating coarsely moving objects from static objects in a frame by operation of a program block named lchofrag10,

separating finely moving objects for differentiation from still background and for detection of areas with a smooth texture by execution of a routine lser6, and

detecting defects inside the smooth texture and background area and removing defects by operating the subroutine lchu6,

saving corrected images and restored frames by use of routine lwrim4,

whereby all defects from the image are removed and an image free of defects is obtained.

16. Software computer programs for the fully-automatically correction of an image having defects thereon, comprising any of the preceding claims 1 to 15.

2. A method for the fully-automatically correcting an image having defects thereon, comprising the steps of:

loading at least one color component, selected from the group of three components consisting of red, green, and blue, from the image to be corrected,

calculating a coherency map for the image,

separating areas of the image having a smooth texture from regions of the image containing edges,

removing defects appearing on the image by application of median filtering,

correcting the image for smoothed edges, and

reconstructing the image from the at least one corrected color component and remaining color components out of the group of three components,

whereby all defects from the image are removed and an image free of defects is obtained.

3. The method according to the preceding claims, wherein:

multi-resolution analysis is applied for camera motion compensation by means of a function lglobshif25.

4. The method according to the preceding claims, wherein:

a coarse separation block and a fine separation block are performed,

the coarse separation block carries out preliminary separation of moving objects inside the images from the still background, by use of the formula

qq=|(Rv1(;

,;

,1)+Rv3(;

,;

,1))*Rv2(;

,;

,1)²,|

in association with two-dimensional arrays Rv1, Rv2, Rv3, to denote the displacements between frames of interest,

the fine separation block performs refined separation of moving objects from the still background and finds fragments of smooth texture inside moving objects that can be treated similarly to the still background, by operating a technique executed by the means of a wandering testing circle and subsequent morphological operations, and checking in association with the inequality

Mcd<

up.ma,

whether the area covered by the testing circle is referred to background or to smooth texture and it is the subject to optional correction.

5. The method according to the preceding claims, wherein:

a defects detection and recovering block is operated to prepares a function to be used for the detection of defects inside the smooth texture and background area, by calculating the absolute differences between the three images, namely;

Q12=|A(;

,;

,1)−

A(;

,;

,2)|,

Q=&

=&

|A(;

,;

,2)−

A(;

,;

,3)|,

Q13&

=&

|A(;

,;

,3)−

A(;

,;

,1)|,

and defining an image QQ123 as

QQ123=|(Q32+Q12)−

Q13|.fb.

eventually calculating a parameter

mu13=min(Q12, Q32)/(max(Q12, Q32)+ε

)

to detect defects inside the smooth texture and background area of the image of interest and for fixing the defects using values of reference images, and constructing a future threshold trel as;

trel=tre.me.

6. The method according to the preceding claims, wherein:

a coherency map of an image is calculated by carrying out median filtering only within the smooth texture areas and applying a correction procedure to avoid smoothing edges.

7. The method according to the preceding claims, wherein:

a coherency map is calculated for at least one color component by finding coefficients of coherency along four directions and calculating coefficients of coherency for each one out of the four directions;

$\begin{array}{c}P<br><br>\left(M_0,{\mathrm{\theta <br><br>}}_j\right)=<br><br>\frac{{\left(\underset{k=-\mathrm{luli}}{\overset{\mathrm{luli}}{\sum <br><br>}}<br><br><br><br>F_{j,k}\right)}^2}{\left(2<br><br><br><br>\mathrm{luli}+1\right)<br><br>\underset{k=-\mathrm{luli}}{\overset{\mathrm{luli}}{\sum <br><br>}}<br><br>F_{j,k}^2},\\ j=<br><br>0,1,2,3.\end{array}$

8. The method according to the preceding claims, wherein:

results are collected in two maps, whose size are equal to the size of the image;

$\mathrm{MAn}<br><br>\left(M<br><br>\left(x,y\right)\right)-\{\begin{array}{cc}0,& \mathrm{if}<br><br><br><br>M<br><br>\left(x,j\right)\in <br><br>\widetilde{\mathrm{\mu <br><br>}};<br><br>\\ {\mathrm{\theta <br><br>}}_{\max }<br><br>\left(M\right)+1,& \mathrm{if}<br><br><br><br>M<br><br>\left(x,y\right)\in <br><br>\mathrm{\mu <br><br>},\end{array}<br><br><br><br>\mathrm{MA}<br><br><br><br>b<br><br>\left(M\right)<br><br>\left(x,y\right)=\{\begin{array}{cc}0,& \mathrm{if}<br><br><br><br>M<br><br>\left(x,y\right)\in <br><br>\widetilde{\mathrm{\mu <br><br>}};<br><br>\\ p_{\max }<br><br>\left(M\right)+1,& \mathrm{if}<br><br><br><br>M<br><br>\left(x,y\right)\in <br><br>\mathrm{\mu <br><br>}.\end{array}$

9. The method according to the preceding claims, wherein:

smooth texture areas are separated from regions containing edges by use of the morphological operations

se=strel(‘

line’

,10,0);

x=imopen(MAn,se);

x=bwareaopen(x,550);

that remove small non-zero patches and apply the procedure

se=strel(‘

line’

,15,90);

x=imclose(x,se);

to fill the gaps in the remaining non-zero regions.

10. The method according to the preceding claims, wherein:

median filtering is performed to remove the defects from the image by use of the MATLAB function medfilt1.

11. The method according to the preceding claims, wherein:

smoothed edges are corrected by the morphological operations;

mam=max(aq(;

));

qa=(aq>

mam/10);

qaq=bwareaopen(qa,Q);

the map qaq of remaining blocks is displayed and the original values are restored to the pixels of the Red, Green, and Blue components located at the nonzero-regions of the map qaq,

the frame is reconstructed from the corrected Red, Green, and Blue components,

whereby all defects are removed.

12. The method according to the preceding claims, wherein:

the coherency map is used to separate smooth texture from regions containing edges.

13. The method according to the preceding claims, wherein:

correction of smoothed edges pertaining to connected blocks corresponding to these edges are singled out from the difference map, by use of the coherency map.

14. The method according to the preceding claims, wherein:

the coherency map is used for camera compensation.

15. The method according to the preceding claims, wherein:

median filtering is performed for removing defects from high definition images.

17. A computer program for correcting an image having defects thereon, comprising:

a computer-readable medium encoded with instructions set forth in the computer program, which is executed by a computer having a memory,

at least one color component, selected from the group of three components consisting of red, green, and blue, being derived from the image and loaded into the computer,

a coherency map being derived for the image for executing calculation on the at least one derived color,

areas of the image having a smooth texture and regions of the image containing edges being separated from each other,

median filtering being applied to the image for removing defects thereon,

smoothed edges being corrected, and

the image being reconstructed from the at least one corrected color component and remaining color components out of the group of three components, whereby all defects from the image are removed and an image free of defects is obtained.