Method and apparatus for motion-vector-aided pixel interpolation

• US 20050024542A1
• Filed: 06/18/2004
• Published: 02/03/2005
• Est. Priority Date: 06/18/2003
• Status: Active Grant

• Current Assignee: Dynamic Data Technologies LLC
• Original Assignee: Micronas Gmbh (TDK Corporation)
• Inventors: Hahn, Marko
• Granted Patent: US 7,274,402 B2
• Time in Patent Office: Days
• US Class Current: 348/700
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

The invention relates to a method for motion-vector-aided interpolation of a pixel in an intermediate image lying temporally or spatially between first and second images, a first pixel and at least one first further pixel in the first image and a second pixel and at least one second further pixel in the second image being selected on the basis of first and second motion vectors, to each of which there are assigned video information values, an interpolated video information value being chosen in such fashion that it lies inside an interval given by the first and second video information values, account being taken of the distance from a third video information value, determined by mixing the first and second further video information values, to the interval boundaries.

Claims

1. Method for motion-vector-aided interpolation of a pixel in an intermediate image lying temporally or spatially between a first image and a second image (A1, B1), comprising:

assigning at least a first and a second motion vector (vec1, vec2) to the pixel (Px) to be interpolated, a pixel (P1, P3) in the first image (A1) and a pixel (P2, P4) in the second image (B1) being assigned to each motion vector (vec1, vec2);

determining a first video information value (L10) on the basis of the video information value of at least one first pixel (P1) assigned to the at least one first displacement vector (vec1) in the first image (A1) and determining a second video information value (L20) on the basis of the video information value of at least one second pixel (P2) assigned to the at least one first displacement vector (vec1) in the second image (B1);

determining a first further video information value (L30) on the basis of the video information value of a third pixel (P3) at the initial point of the second motion vector (vec2) in the first image (A1) and determining a second further video information value (L40) on the basis of the video information value of a fourth pixel (P4) at the terminus of the second motion vector (vec2) in the second image (B1);

generating a third video information value (L3) by mixing the first further video information value (L30) and the second further video information value (L40);

determining an interval given by the first and second video information values (L10, L20);

determining the position of the third video information value (L3) relative to the interval boundaries (L10, L20);

selecting the third video information value (L3) as the video information value (Lx) of the interpolated pixel (Px) if the third video information value (L3) lies inside the interval;

determining the video information value (Lx) of the pixel (Px) to be interpolated if the third video information value (L3) lies outside the interval, by

a) determining the mathematical distance from the third video information value (L3) to the nearest interval boundary (L10;

L20); and

b) determining a video information value that is remote from the nearest interval boundary (L10;

L20) by the value of the distance determined in step a) and lies inside the interval and selecting this video information value as the video information value (Lx) of the interpolated pixel (Px).

2. Method according to claim 1 wherein the average of the first video information value (L10) and the second video information value (L20) is selected as the interpolated video information value (Lx) if the third video information value (L3) lies outside the interval given by the first and second video information values (L10, L20) and the distance from the third video information value (L3) to the nearest interval boundary is larger than half the interval width.

3. Method according to claim 1 wherein the mathematical distance is set equal to zero in step a) if the mathematical distance is smaller than a specified threshold value.

4. Method according to claim 3 wherein the boundary value is smaller than half the interval width.

5. Method according to claim 1 wherein the mathematical distance in step a) is increased (amplified) or diminished (damped).

6. Method according to claim 1 wherein the second displacement vector (vec2) is a zero vector.

7. Method according to claim 2 wherein the third video information value (L3) is formed from the average of the first and second further video information values (L30, L40).

8. Method according to claim 1 comprising the following steps for the determination of the first and second video information values (L10, L20);

selecting a plurality of first motion vectors (vec1) and determining the video information values of pixels assigned to these motion vectors in the first image (A1) and determining the video information values assigned to these motion vectors in the second image (B1),

selecting one of the first video information values as the first video information value and selecting one of the second video information values as the second video information value.

9. Method according to claim 8 comprising the following further steps:

determining the intervals between each of the video information values of pixels assigned to the first motion vectors in the first image (A1) and each of the video information values of pixels assigned to the first motion vectors in the second image (B1) and selecting as the first and second video information values (L10, L20) those two video information values between which the smallest interval is formed or between which the largest interval is formed.

10. Method according to claim 1 comprising the following steps for the determination of the first and second video information values (L10, L20):

selecting a plurality of first motion vectors (vec1) and determining the video information values of pixels (L11, L12, L13, L14) assigned to these motion vectors in the first image (A1) and determining the video information values (L21, L22, L23, L24) assigned to these motion vectors in the second image (B1),

filtering the video information values (L11-L14, L21-L24) determined on the basis of the motion vectors in order to generate an intermediate value,

selecting the video information values assigned to one of the first motion vectors (vec1) and selecting as the first and second video information values (Max, Min) the two values, from a set including these two video information values and the intermediate value, that span the largest interval.

11. Apparatus for providing a video information value of an interpolated pixel, comprising:

a comparator unit (2), which is supplied with video information values (Max, Min) resulting from first and second video information values (L10, L20), provides a first control signal (S1) that indicates whether a third video information value (L3) lies inside an interval given by the first and second video information values (L10, L20), provides a second control signal (S2) that indicates whether the third video information value (L3) is larger than the upper or smaller than the lower interval boundary, and provides as output signal (A2) the first or second video information value nearest to the third video information value (L3) if the third video information value (L3) lies outside the interval,

a subtractor arrangement (3), which forms the difference between the output signal (A2) of the comparator unit and the third video information value (L3) and the absolute value of this difference and provides it as output signal (A3),

a limiter arrangement (5), which is supplied with the output signal (A3) of the subtractor arrangement (3), limits the output signal (A3) of the subtractor arrangement (3) to a specified limit value, and provides the limited signal as output signal (A5),

a second subtractor arrangement (6), which is supplied with the output signal of the comparator arrangement (2), the second control signal (S2) of the comparator arrangement, and the output signal (A5) of the limiter arrangement (5) and, as directed by the second control signal (S2), adds the output signal (A5) of the limiter arrangement (5) to the output signal (A2) of the comparator arrangement (2) or subtracts it from the latter (A2), and provides this sum or difference as output signal (A6),

a selector circuit (7), which is supplied with the output signal (A6) of the second subtractor arrangement (6), the first control signal (S1) of the comparator arrangement (2), and the third video information value (L3) and, as directed by the first control signal (S1), outputs as the interpolated video information value (Lx) the output signal (A6) of the second subtractor arrangement (6) or the third video information value (L3),

a mixer (16), which provides the third video information value (L3) from at least a first and a second further video information value (L30, L40) by mixing these video information values.

12. Apparatus of claim 11 wherein, connected ahead of the comparator arrangement (2), there is a sorting and interval forming unit (11), which is supplied with a plurality of first video information values (L11-L14) and a plurality of second video information values (L21-L24), which are respectively assigned to one another and form video information value pairs, the sorting and interval forming unit selecting one of the value pairs and outputting one of the two video information values of the pair as the first video information value (Max) and the other video information value of the pair as the second video information value (Min).

13. Apparatus of claim 12 wherein the sorting and interval forming unit (11) determines an interval between the first video information value (L11-L14) and the second video information value (L21-L24) of each value pair and selects the value pair between the values of which the smallest interval is formed.

14. Apparatus of claim 11 wherein the limit value to which the limiter arrangement (5) limits the output signal (A3) of the subtractor arrangement (3) equals half the difference between the first and second video information values (L1, L2).

15. Apparatus for providing a video information value of an interpolated pixel, having the following features:

a first subtractor arrangement (21), which is supplied with a first video information value (Max) as the input signal, and a second subtractor unit (22), which is supplied with a second video information value (Min) as the input signal, each of which is supplied with a third video information value (L3), the subtractor arrangements (21, 22) each providing as the output signals (A21, A22) the absolute value of a difference between the respective input signal (Max, Min) and the third video information value (L3),

a first limiter arrangement (51) connected behind the first subtractor arrangement (21) and a second limiter arrangement (52) connected behind the second subtractor arrangement (22), the limiter arrangements (51, 52) each limiting the output signals (A21, A22) of the subtractor arrangements (21, 22) to a specified limit value,

a further subtractor arrangement (31), connected behind the first limiter arrangement (51), which subtracts a signal present at the output of the first limiter arrangement (A51) from the first video information value (Max), and an adder arrangement (32) connected behind the second limiter arrangement (52), which adds a signal (A52) present at the output of the second limiter arrangement (52) to the second video information value (Min),

a comparator arrangement (80), which is supplied with the first and second limiter output signals (A51, A52) and provides a control signal (S80) in dependence on a comparison of these signals (A51, A52),

a first selector circuit (90), which is supplied with the control signal (S80) of the comparator arrangement (80) as well as an output signal (A31) of the further subtractor arrangement (31) and an output signal (A32) of the adder arrangement (32) and, as directed by the control signal (S80), furnishes one of these two output signals at its output as output signal (A90),

a second comparator arrangement (60), which is supplied with the first and second video information values (L10, L20) and a third video information value (L3), and provides a second control signal (S60) in dependence on a comparison of these signals,

a second selector circuit (70), which is supplied with the third video information value (L3), the output signal of the first selector circuit (90) and the second control signal (S60) and, in dependence on the second control signal (S60), outputs as the interpolated video information value (Lx) the third video information value (L3) or the output signal (A90) of the first selector circuit (90),

a mixer (16), which provides the third video information value (L3) from at least a first and a second further video information value (L30, L40) by mixing these video information values.

16. Apparatus of claim 15 wherein, connected ahead of the subtractor arrangements (21, 22), there is a sorting unit (1;

11), which is supplied with at least one video information value pair (L10, L20;

L1, L2, Lf), the sorting arrangement (1) selecting a value pair and outputting the larger value of the value pair as the first video information value (Max) and the smaller value of the value pair as the second video information value (Min).

17. Apparatus of claim 16 wherein the sorting apparatus determines, for each value pair, an interval between the video information values of the value pair and selects the value pair between the values of which the smallest interval is formed.