Method of classifying the pixels of an image belonging to a sequence of moving images and method of temporal interpolation of images using the said classification

• US 5,394,196 A
• Filed: 11/19/1992
• Issued: 02/28/1995
• Est. Priority Date: 04/05/1991
• Status: Expired due to Term

• Current Assignee: Thomson-CSF (Thales SA)
• Original Assignee: Thomson-CSF (Thales SA)
• Inventors: Robert, Philippe
• Primary Examiner(s): Britton, Howard W.
• Application Number:
  US07/946,371
• Time in Patent Office: 831 Days
• Field of Search: 348/416, 348/699, 348/413, 348/407
• US Class Current: 348/699
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51

Abstract

The present invention relates to a method of classifying the pixels of an image belonging to a sequence of moving images, as well as to a method of temporal interpolation of images using this classification. The method of classifying the pixels of an image t(j+1) pertains to temporal projection of the field of motion of an image t(j) belonging to a sequence of moving images, with the pixels of the image t(j) being defined by a motion vector and by a tag belonging to a class such as "normal", "occluding" or "appearing". The method is characterized by the fact that, before projection, all pixels of t(j) are tagged "appearing", and then, during the temporal projection, the tagging is modified as a function of the tagging of the pixels of t(j) in such a way as to obtain a tagging according to the same above classes (i.e., normal, occluding, or appearing).

Claims

1. Method for classifying the pixels of a current image t(j+1) obtained by temporal projection of the field of motion of a preceding image t(j) belonging to a sequence of moving images, the pixels of the preceding image being defined by a motion vector and by a tag, the tag belonging to a class defined from at least three classes termed "normal", "occluding" or "appearing", the said method comprising the steps of:

before projection, all the pixels of the current image t(j) are tagged "appearing";

carrying out temporal projection of t(j);

modify tagging during temporal projection by taking into account the tagging of the pixels of the image t(j) in such a way that;

if a pixel tagged "appearing" receives a motion vector, change its tag to "normal";

if a pixel is tagged "normal" receives a motion vector, change its tag to "occluding" and assign it the motion vector with minimum |DFD| (for displaced frame difference);

if a pixel tagged "occluding" receives a motion vector, leave its tag as occluding and assign it the motion vector with minimum |DFD| and;

if a pixel tagged "appearing" receives no motion vector, leave the tag as "appearing".

2. Method according to claim 1, wherein, if during the temporal projection of the field of motion of the preceding image t(j) onto the current image t(j+1), the motion vector projected onto the current image t(j+1) of a pixel of the preceding image t(j) conflicts with another vector and is not retained, then the pixel of the image t(j) is tagged "disappearing".

3. Method of modifying the tags assigned to the pixels of the image t(j+1), each pixel being defined by a motion vector and a tag which are obtained using the method according to claim 1, in such a way that, for the image t(j+1), a temporal motion prediction field is available, wherein, in a first step, the motion is estimated for all the vectors of the image t(j+1) so as to obtain an estimated field of motion, then for each pixel the motion prediction field is compared with the estimated field of motion in such a way as possibly to modify the tags assigned to the pixels of the image t(j+1) according to the following procedure:

if the tag is "normal" it remains "normal";

if the tag is "appearing" it remains "appearing";

if the tag is "filled-in" it is changed to "normal";

if the tag is "occluding", for each of the horizontal and vertical components of the predicted motion vector and of the estimated motion vector, their difference is calculated and if one of the differences is greater than a threshold the tag becomes "normal", otherwise it remains "occluded".

4. Method according to claim 3, wherein the "occluding" tag is affirmed only when the pixel following in the sequence of images has been labelled "occluding" at least N times with N>

1.

5. Method according to claim 1, wherein, during temporal projection, the tags assigned to the pixels of the image t(j) are examined and if the pixel of the image t(j) is tagged "appearing" its motion vector is not assigned to the pixels of the current image t(j+1).

6. Method according to claim 1, wherein, during temporal projection, the tags assigned to the pixels of the image t(j) are examined and the pixels of the current image t(j+1) are tagged according to the table below in which the pixels of the image t(j+1) are assigned a tag chosen from the classes "appearing", "normal", "occluding" and "filled in":

space="preserve" listing-type="tabular">______________________________________ t(j +

     1) t(j) Appearing Normal or occluding tag + tag + D1 D1 vector vector Appearing Filled in Normal D1 D1 Normal Normal Occluding D2 D2 min(|DFD(D1)|, |DFD(D2)|) Occluding Occluding Occluding D2 D2 min(|DFD(D1)|, |DFD(D2)|) Filled in Filled in Normal D2 min(|DFD(D1)| D1 |DFD(D2)|) ______________________________________

7. Method according to either one of claims 5 or 6, wherein the motion vector of a pixel of the current image t(j+1) assigned an "appearing" tag is obtained by spatial interpolation of the motion vectors of its neighbourhood.

8. Method according to claim 7, wherein the motion vector of a pixel of the current image t(j+1) assigned an "appearing" tag is obtained by taking a mean of the motion vectors of the pixels which form part of its spatial neighbourhood without taking account of the pixels tagged "appearing".

9. Method of temporal interpolation of images with motion compensation, including in particular a phase of estimating the field of motion of the field of tags of an image to be interpolated T(j) from at least two known images t(j), t(j+1) enclosing it, the characteristics of each pixel of the image T(j) being determined from the characteristics of the associated pixels of the images t(j), t(j+1) each defined by a motion vector and by a tag, in which the field of motion of the image to be interpolated T(j) is defined by temporal projection of the estimated field of motion at the known image t(j+1) by using the methods according to claim 1, wherein

before projection, all the pixels of the image to be interpolated T(j) are tagged "holes";

then, in the course of the temporal projection, the tags and the motion vectors assigned to the pixels of the image t(j+1) are examined and a tag and a motion vector are given to the pixels of the image to be interpolated T(j) according to the table below;

space="preserve" listing-type="tabular">______________________________________ T(j) t(j +

     1) Normal or Occluding tag + tag + appearing or and |DFD |<

S vector vector occluding and |DFD| >

S D1 (S = threshold) D1 Hole Normal D1 Occluding D1 Normal Normal Occluding D2 min(|DFD(D1)|, D1 |DFD(D2|) Occlud- Occluding Occluding ing D2 D2 min(|DFD(D1)|, |DFD(D2)|) ______________________________________

10. Method according to claim 9, wherein, having carried out the temporal projection of the field of motion of the image t(j+1), the pixels of the image to be interpolated T(j) tagged "holes" receive a motion vector obtained in a known manner by spatial interpolation.

11. Method according to claims 9 or 10, wherein the pixels of the image T(j) tagged "normal" or "occluding" are interpolated from the two images enclosing it t(j) and t(j+1), whereas the pixels of the image T(j) tagged "holes" are interpolated by examining the tag of each of the four pixels enclosing the two ends of the motion vector in the images t(j) and t(j+1), in such a way that:

if, in image t(j+1), at least one of the four pixels enclosing the end of the motion vector is tagged "occluding" and if, in the image t(j) none of the four pixels enclosing the end of the motion vector is tagged "occluding", the luminance of the pixel of the image T(j) is constructed from the image t(j) alone;

if in the image t(j), at least one of the four pixels enclosing the end of the motion vector is tagged "occluding" and if in the image t(j+1) none of the four pixels enclosing the end of the motion vector is tagged "occluding" the luminance of the pixel of the image T(j) is reconstructed from the image t(j+1) alone;

if, at the two ends of the motion vector, at least one of the four pixels is tagged "occluding" or no pixel is tagged "occluding", the luminance of the pixel of the image T(j) is reconstructed from both images.

12. Method according to claims 9 or 10, characterised in that the pixels of the image T(j) tagged "normal" or "occluding" are interpolated from the two images enclosing it t(j) and t(j+1) whereas the pixels of the image T(j) tagged "holes" are interpolated by examining the tag of each of the four pixels enclosing the two ends of the motion vector on the images t(j) and t(j+1), in such a way that:

if, in the image t(j), one of the four pixels enclosing the end of the motion vector is tagged "occluding" and none is "disappearing", and if in the image t(j+1) one of the four pixels enclosing the end of the motion vector is tagged "appearing", the luminance of the pixel of the image T(j) is reconstructed from the image t(j+1) alone;

if, in the image t(j+1), one of the four pixels enclosing the end of the motion vector is tagged "occluding" and none is "appearing" and if in the image t(j) one of the four pixels enclosing the end of the motion vector tagged "disappearing", the luminance of the pixel of the image T(j) is reconstructed from the image t(j) alone;

in the other cases, the luminance of the pixel of the image T(j) is reconstructed from the images t(j+1) and t(j).

13. Method of temporal interpolation according to claim 9, wherein:

when projecting the field of motion of the known image t(j+1) onto the image to be interpolated T(j), the motion vector of the pixels of the image t(j+1) tagged "appearing" is not projected;

next, the field of motion of the image t(j) is projected onto the image to be interpolated T(j) by using the same method as for the projection of the image t(j+1), a "hole" pixel filled by a motion vector being tagged "disappearing";

then, the field of motion of an image t(j+2) following an image t(j+1) is projected onto the image to be interpolated T(j) by using the same method as for the projection of the image t(j+1) but tagging a hole pixel filled by an "appearing" motion vector in such a way as to take into account the "appearing" and "disappearing" pixels.

14. Method according to claim 13, wherein the motion vector of a pixel of the image to be interpolated T(j) assigned a "hole" tag is calculated by spatial interpolation and the pixel is assigned a "normal" tag.

15. Method according to either one of claims 13 or 14, wherein:

the pixels of the image T(j) tagged "normal" are interpolated from the two images enclosing it t(j) and t(j+1);

the pixels of the image T(j) tagged "disappearing" are interpolated from the image t(j), and

the pixels of the image T(j) tagged "appearing" are interpolated from the image t(j+1).