Motion estimator for reduced halos in MC up-conversion

• US 20030206246A1
• Filed: 05/15/2001
• Published: 11/06/2003
• Est. Priority Date: 05/18/2000
• Status: Active Grant

• Current Assignee: MaxLinear Communications, LLC (f/k/a Entropic Communications, LLC) (Maxlinear Incorporated)
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: De Haan, Gerard, Mertens, Mark Jozef Willem
• Granted Patent: US 7,010,039 B2
• Time in Patent Office: Days
• US Class Current: 348/699
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 7/238 using non-full search, e.g. three-step search
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/553 Motion estimation dealing with occlusions

Abstract

An apparatus for detecting motion at a temporal intermediate position between a previous image and a next image has optimizing means (1) for optimizing a criterion function for candidate vectors, whereby the criterion function depends on data from both the previous and the next image. The motion is detected at the temporal intermediate position in non-covering and in non-uncovering areas. The apparatus according to the invention has means for detecting covering and uncovering areas (2) and has its optimizing means being arranged to carry out the optimizing at the temporal position of the next image in covering areas and at the temporal position of the previous image in uncovering areas.

Claims

1. Method for detecting motion at a temporal intermediate position between previous and next images, in which a criterion function for candidate vectors is optimised, said function depending on data from both previous and next images and in which the optimising is carried out at the temporal intermediate position in non-covering and non-uncovering areas, characterised in that the optimising is carried out at the temporal position of the next image in covering areas and at the temporal position of the previous image in uncovering areas.

2. Method according to claim 1, wherein the previous image is shifted over a fraction α

times the candidate vector, the next image is shifted over 1−

α

times the candidate vector and the fraction a may change within the image period.

3. Method according to claim 1, wherein the criterion function is a match error which is minimised.

4. Method according to claim 2, wherein the fraction α

is controlled by a covering/uncovering detector in the matching process.

5. Method according to claim 4, wherein the fraction α

is set to 1 in case of covering and set to 0 in case of uncovering.

6. Method according to claim 4, wherein the covering/uncovering detector decides on data in a previous image to the fraction α

in the current estimation.

7. Method according to claim 1, wherein a velocity edge X_E is determined, an occlusion area is marked around said edge and in said area foreground velocity is replaced by background velocity or reversibly dependent on the occlusion is a covering or uncovering area, the sign of the foreground velocity and on which side of the velocity edge X_E the foreground is.

8. Method according to claim 7, wherein at the position {right arrow over (x)}₁ of a velocity edge

a first position {right arrow over (x)}_α

in the previous (covering) or next (uncovering) image is calculated by shifting {right arrow over (x)}₁ over the first vector at one side of the edge

a second position {right arrow over (x)}_b in the previous (covering) or next (uncovering) image is calculated by shifting {right arrow over (x)}₁ over the second vector at the other side of the edge

and a third intermediate position between {right arrow over (x)}_α

and {right arrow over (x)}_b is calculated

while finally, the vector fetched with v<sub>{right arrow over (α

)}v</sub> at the third position in the previous (covering) or next (uncovering) image is filled in those regions of the image in the environment of the edge, to which no vector is projected, in case the background vector v_{{right arrow over (F)}G} should be filled in and the vector chosen between

$\stackrel{\to <br><br>}{D}<br><br>\left(\stackrel{\to <br><br>}{x}-\left(\begin{array}{c}1\\ 0\end{array}\right),n\right)$

and

$\text{\hspace{1em}<br><br>}<br><br>\stackrel{\to <br><br>}{D}<br><br>\left(\stackrel{\to <br><br>}{x}+\left(\begin{array}{c}1\\ 0\end{array}\right),n\right)$

which is most different from v<sub>{right arrow over (α

)}v</sub> is filled in, in case a foreground vector v_{{right arrow over (F)}G} should be filled in.

9. Method according to claim 8, wherein the intermediate position is ({right arrow over (x)}_α

+{right arrow over (x)}_b)/2.

10. Method according to claim 7, wherein a background velocity is identified as a velocity which crosses the velocity discontinuity and projects to a foreground velocity in the previous picture, whereas a foreground velocity projects to itself.

11. Method according to claim 7, wherein near edges it is tested whether the mentioned edge has moved over the first vector on one side of the edge, or over the second vector on the other side of the edge, in case the edge moves with the first (second) vector, the second (first) vector is filled in those regions of the projected vector field in the environment of the edge, to which no vector is projected, in case a background vector v_{{right arrow over (B)}G} should be filled in, and the other vector is filled in, in case a foreground vector v_{{right arrow over (F)}G} should be filled.

12. Method according to claim 10, wherein the crossing from a background region to a foreground region in the previous image is verified by the match error of the vector in that block.

19. Apparatus according to one of the preceding claims, wherein a velocity edge X_E is determined, an occlusion area is marked around said edge and in said area foreground velocity is replaced by background velocity or reversibly dependent on the occlusion is a covering or uncovering area, the sign of the foreground velocity and on which side of the velocity edge X_E the foreground is.

20. Apparatus according to claim 19, wherein calculation means (5,6,8) are provided for, at the position {right arrow over (x)}₁ of a velocity edge, calculating

a first position {right arrow over (x)}_α

in the previous (covering) or next (uncovering) image by shifting {right arrow over (x)}₁ over the first vector at one side of the edge

a second position {right arrow over (x)}_b in the previous (covering) or next (uncovering) image by shifting {right arrow over (x)}₁ over the second vector at the other side of the edge

and a third intermediate position between {right arrow over (x)}_α

and {right arrow over (x)}_b,

while finally, the vector fetched with v<sub>{right arrow over (α

)}v</sub> at the third position in the previous (covering) or next (uncovering) image (9) is filled in those regions of the image in the environment of the edge, to which no vector is projected, in case the background vector v_{{right arrow over (F)}G} should be filled in and the vector chosen between

$\stackrel{\to <br><br>}{D}<br><br>\left(\stackrel{\to <br><br>}{x}-\left(\begin{array}{c}1\\ 0\end{array}\right),n\right)<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{and}<br><br>\text{\hspace{1em}<br><br>}<br><br>\stackrel{\to <br><br>}{D}<br><br>\left(\stackrel{\to <br><br>}{x}+\left(\begin{array}{c}1\\ 0\end{array}\right),n\right)$

which is most different from v<sub>{right arrow over (α

)}v</sub> is filled in, in case a foreground vector v_{{right arrow over (F)}G} should be filled in.

21. Apparatus according to claim 20, wherein the intermediate position is ({right arrow over (x)}_α

+{right arrow over (x)}_b)/2.

22. Apparatus according to claim 19, wherein means (10,11) are provided for projecting two positions on either side of the edge to the previous (covering) or next (uncovering) image, in which a background velocity is identified (14) as a velocity which crosses the velocity discontinuity and projects to a foreground velocity in the previous picture, whereas a foreground velocity projects to itself.

23. Apparatus according to claim 19, wherein means (20) are provided for testing near edges whether the mentioned edge has moved over the first vector on one side of the edge, or over the second vector on the other side of the edge, in case the edge moves with the first (second) vector, the second (first) vector is filled in those regions of the projected vector field in the environment of the edge, to which no vector is projected, in case a background vector v_{{right arrow over (B)}G} should be filled in, and the other vector is filled in, in case a foreground vector v_{{right arrow over (F)}G} should be filled.

24. Apparatus according to claim 22, wherein verification means are provided for verifying the crossing from a background region to a foreground region in the previous image by the match error of the vector in that block.

13. Apparatus for detecting motion at a temporal intermediate position between previous and next images, comprising means (1) optimising a criterion function for candidate vectors, said function depending on data from both previous and next images in which the optimising is carried out at the temporal intermediate position in non-covering and non-uncovering areas, characterised in that means for detection covering or uncovering areas (2) are provided and that the optimising is carried out at the temporal position of the next image in covering areas and at the temporal position of the previous image in uncovering areas.

14. Apparatus according to claim 13, wherein the previous image is shifted over a fraction α

times the candidate vector, the next image is shifted over 1−

α

times the candidate vector and the fraction α

may change within the image period.

15. Apparatus according to claim 13, wherein the criterion function is a match error which is minimised.

16. Apparatus according to claim 14, wherein the fraction α

is controlled by a covering/uncovering detector (2) in the matching process.

17. Apparatus according to claim 16, wherein the fraction α

is set to 1 in case of covering and set to 0 in case of uncovering.

18. Apparatus according to claim 16, wherein the covering/uncovering detector (2) decides on data in a previous image to the fraction α

in the current estimation.

25. Image display apparatus comprising detection apparatus (21) for detection a motion vector according to claim 13, means (22) for interpolating image parts connected to said detection apparatus (21) and a display device (23) connected to the interpolating means.