Temporal interpolation of a pixel on basis of occlusion detection

• US 20070002058A1
• Filed: 08/23/2004
• Published: 01/04/2007
• Est. Priority Date: 09/02/2003
• Status: Active Grant

• Current Assignee: Dynamic Data Technologies LLC
• Original Assignee: Koninklijke Philips Electronics N.V. (Koninklijke Philips N.V.)
• Inventors: Wittebrood, Rimmert
• Granted Patent: US 7,536,031 B2
• Time in Patent Office: Days
• US Class Current: 345/475
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/553 Motion estimation dealing with occlusions
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence

Abstract

A method of determining a value for a particular pixel of a particular image being temporarily located intermediate a first image and a second image is disclosed. The method comprises: computing a first motion vector difference on basis of a first (D_p) and second (D_pp) motion vector of a first motion vector field (D₃(x, n−1)) corresponding to the first image; computing a second motion vector difference on basis of a third (D_n) and fourth (D_nn) motion vector of a second motion vector field (D₃(x, n)) corresponding to the second image; and establishing the value of the particular pixel on basis of a first value of a first pixel of the first image if the first motion vector difference is smaller than the second motion vector difference and establishing the value of the particular pixel on basis of a second value of a second pixel of the second image if the second motion vector difference is smaller than the first motion vector difference.

Claims

1. A method of determining a value for a particular pixel of a particular image being temporarily located intermediate a first image and a second image, the method comprising:

computing a first motion vector difference on basis of a first ({right arrow over (D)}_p) and second ({right arrow over (D)}_pp) motion vector of a first motion vector field ({right arrow over (D)}₃({right arrow over (x)},n−

1)) corresponding to the first image, the first motion vector ({right arrow over (D)}_p) being selected from the first motion vector field ({right arrow over (D)}₃({right arrow over (x)},n−

1)) on basis of a particular spatial position ({right arrow over (x)}) of the particular pixel and a particular motion vector ({right arrow over (D)}_c) being estimated for the particular pixel;

computing a second motion vector difference on basis of a third ({right arrow over (D)}_n) and fourth ({right arrow over (D)}_nn) motion vector of a second motion vector field ({right arrow over (D)}₃({right arrow over (x)},n)) corresponding to the second image, the third motion vector ({right arrow over (D)}_n) being selected from the second motion vector field ({right arrow over (D)}₃({right arrow over (x)},n)) on basis of the particular spatial position ({right arrow over (x)}) of the particular pixel and the particular motion vector ({right arrow over (D)}_c) being estimated for the particular pixel; and

establishing the value of the particular pixel on basis of a first value of a first pixel of the first image if the first motion vector difference is smaller than the second motion vector difference and establishing the value of the particular pixel on basis of a second value of a second pixel of the second image if the second motion vector difference is smaller than the first motion vector difference.

2. A method of determining a value for a particular pixel as claimed in claim 1, wherein the second motion vector ({right arrow over (D)}_pp) is selected from the first motion vector field ({right arrow over (D)}₃({right arrow over (x)},n−

1)) on basis of the particular spatial position ({right arrow over (x)}) of the particular pixel and the first motion vector ({right arrow over (D)}_p).

3. A method of determining a value for a particular pixel as claimed in claim 2, wherein the fourth motion vector ({right arrow over (D)}_nn) is selected from the second motion vector field ({right arrow over (D)}₃({right arrow over (x)},n)) on basis of the particular spatial position ({right arrow over (x)}) of the particular pixel and the third motion vector ({right arrow over (D)}_n).

4. A method of determining a value for a particular pixel as claimed in claim 1, comprising establishing the value of the particular pixel on basis of the first value of the first pixel and of the second value of the second pixel if the first motion vector difference and the second motion vector difference are smaller than a predetermined threshold.

5. A method of determining a value for a particular pixel as claimed in claim 1, comprising establishing the value of the particular pixel by means of interpolation of the first value of the first pixel and a further value of a further pixel in a spatial neighborhood of the first pixel.

6. A pixel value determining unit (500) for determining a value for a particular pixel of a particular image being temporarily located intermediate a first image and a second image, the pixel value determining unit comprising:

first computing means (508) for computing a first motion vector difference on basis of a first ({right arrow over (D)}_p) and second ({right arrow over (D)}_pp) motion vector of a first motion vector field ({right arrow over (D)}₃({right arrow over (x)},n−

1)) corresponding to the first image, the first motion vector ({right arrow over (D)}_p) being selected from the first motion vector field ({right arrow over (D)}₃({right arrow over (x)},n−

1)) on basis of a particular spatial position ({right arrow over (x)}) of the particular pixel and a particular motion vector ({right arrow over (D)}_c) being estimated for the particular pixel;

second computing means (510) for computing a second motion vector difference on basis of a third ({right arrow over (D)}_n)) and fourth ({right arrow over (D)}_nn) motion vector of a second motion vector field ({right arrow over (D)}₃({right arrow over (x)},n)) corresponding to the second image, the third motion vector ({right arrow over (D)}_n) being selected from the second motion vector field ({right arrow over (D)}₃({right arrow over (x)},n)) on basis of the particular spatial position ({right arrow over (x)}) of the particular pixel and the particular motion vector ({right arrow over (D)}_c) being estimated for the particular pixel; and

establishing means (504) for establishing the value of the particular pixel on basis of a first value of a first pixel of the first image if the first motion vector difference is smaller than the second motion vector difference and establishing the value of the particular pixel on basis of a second value of a second pixel of the second image if the second motion vector difference is smaller than the first motion vector difference.

7. An image processing apparatus (600) comprising:

receiving means (602) for receiving a signal corresponding to a sequence of video images;

motion estimation means (512) for estimating a first motion vector field for a first one of the video images and a second motion vector field for a second one of the video images; and

a pixel value determining unit (500) for determining a value for a particular pixel of a particular image being temporarily located intermediate the first one of the video images and the second one of the video images, the pixel value determining unit comprising;

first computing means (508) for computing a first motion vector difference on basis of a first ({right arrow over (D)}_p) and second ({right arrow over (D)}_pp) motion vector of a first motion vector field ({right arrow over (D)}₃({right arrow over (x)},n−

1)) corresponding to the first image, the first motion vector ({right arrow over (D)}_p) being selected from the first motion vector field ({right arrow over (D)}₃({right arrow over (x)},n−

1)) on basis of a particular spatial position ({right arrow over (x)}) of the particular pixel and a particular motion vector ({right arrow over (D)}_c) being estimated for the particular pixel;

second computing means (510) for computing a second motion vector difference on basis of a third ({right arrow over (D)}_n) and fourth ({right arrow over (D)}_nn) motion vector of a second motion vector field ({right arrow over (D)}₃({right arrow over (x)},n)) corresponding to the second image, the third motion vector ({right arrow over (D)}_n) being selected from the second motion vector field ({right arrow over (D)}₃({right arrow over (x)},n)) on basis of the particular spatial position ({right arrow over (x)}) of the particular pixel and the particular motion vector ({right arrow over (D)}_c) being estimated for the particular pixel;

establishing means (504) for establishing the value of the particular pixel on basis of a first value of a first pixel of the first one of the video images if the first motion vector difference is smaller than the second motion vector difference and establishing the value of the particular pixel on basis of a second value of a second pixel of the second one of the video images if the second motion vector difference is smaller than the first motion vector difference.

8. An image processing apparatus (600) as claimed in claim 7, further comprising a display device (606) for displaying the output images.

9. An image processing apparatus (600) as claimed in claim 8, being a TV.

10. A computer program product to be loaded by a computer arrangement, comprising instructions to determine a value for a particular pixel of a particular image being temporarily located intermediate a first image and a second image, the computer program product, after being loaded providing said processing means with the capability to carry out:

computing a first motion vector difference on basis of a first ({right arrow over (D)}_p) and second ({right arrow over (D)}_pp) motion vector of a first motion vector field ({right arrow over (D)}₃({right arrow over (x)},n−

1)) corresponding to the first image, the first motion vector ({right arrow over (D)}_p) being selected from the first motion vector field ({right arrow over (D)}₃({right arrow over (x)},n−

1)) on basis of a particular spatial position ({right arrow over (x)}) of the particular pixel and a particular motion vector ({right arrow over (D)}_c) being estimated for the particular pixel;

computing a second motion vector difference on basis of a third ({right arrow over (D)}_n) and fourth ({right arrow over (D)}_nn) motion vector of a second motion vector field ({right arrow over (D)}₃({right arrow over (x)},n)) corresponding to the second image, the third motion vector ({right arrow over (D)}_n) being selected from the second motion vector field ({right arrow over (D)}₃({right arrow over (x)},n)) on basis of the particular spatial position ({right arrow over (x)}) of the particular pixel and the particular motion vector ({right arrow over (D)}_c) being estimated for the particular pixel; and

establishing the value of the particular pixel on basis of a first value of a first pixel of the first image if the first motion vector difference is smaller than the second motion vector difference and establishing the value of the particular pixel on basis of a second value of a second pixel of the second image if the second motion vector difference is smaller than the first motion vector difference.