Segmentation-based method for motion-compensated frame interpolation

• US 6,008,865 A
• Filed: 09/19/1997
• Issued: 12/28/1999
• Est. Priority Date: 02/14/1997
• Status: Expired due to Fees

• Current Assignee: Eastman Kodak Company
• Original Assignee: Eastman Kodak Company
• Inventors: Fogel, Sergei
• Primary Examiner(s): Chin, Tommy P.
• Assistant Examiner(s): Voisinet, John A.
• Application Number:
  US08/934,262
• Time in Patent Office: 830 Days
• Field of Search: 348/700, 348/699, 348/169, 348/170, 348/171, 348/584, 348/586, 382/103, 382/236, 382/291
• US Class Current: 348/700
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures

Abstract

A method of motion compensated frame interpolation wherein an intermediate image frame is generated between two image frames in a motion image sequence. The method includes the steps of: identifying a moving foreground or object and the background for two successive frames in a motion image sequence; estimating a velocity vector field for the foreground, and a velocity vector field for the background; using the estimated velocity vector field for the foreground to identify the location of the foreground and the background in the intermediate image; and obtaining the intermediate image by using the foreground velocity vector field to perform motion compensated interpolation for the foreground in the intermediate image and using the background velocity vector field to perform motion compensated interpolation for the background in the intermediate image.

Claims

1. A method of motion compensated frame interpolation wherein an intermediate image frame is generated between two image frames in a motion image sequence, comprising the steps of:

a) identifying a foreground and a background for two successive frames in a motion image sequence;

b) estimating a velocity vector field for the foreground, and a velocity vector field for the background;

c) using the estimated velocity vector field for the foreground to locate corresponding points in the two frames and, if the points in both frames are in the foreground, to identify the location as foreground and, otherwise, as background in the intermediate image; and

d) obtaining the intermediate image by using the foreground velocity vector field to perform motion compensated interpolation for the foreground in the intermediate image and using the background velocity vector field to perform motion compensated interpolation for the background in the intermediate image.

2. The method claimed in claim 1, further comprising the steps of:

e) identifying a second foreground for the two successive frames;

f) treating the first foreground as the background for the second foreground; and

g) applying steps b) through d) to the second foreground and the first background.

3. The method claimed in claim 1, wherein step a) of identifying a foreground and a background is performed by a user employing a graphical user interface.

4. The method claimed in claim 1, wherein step b) of estimating a velocity vector field for the foreground, and a velocity vector field for the background includes the steps of:

i) computing a velocity vector field for a coarsest resolution level;

ii) project the velocity vector field from the previously computed resolution level onto a next finer resolution level;

iii) computing a velocity vector field at the finer resolution level;

iv) repeating steps b) and c) until a final finest resolution level is reached; and

v) interpolating the velocity vector field at the finest resolution level to the pixel resolution level.

5. The method claimed in claim 1, wherein step d) of obtaining the intermediate image by using the foreground velocity vector field to perform motion compensated interpolation for the foreground in the intermediate image and using the background velocity vector field to perform motion compensated interpolation for the background in the intermediate image, further comprises the steps of:

i) for each pixel in the intermediate image, if it is a foreground pixel, using the foreground velocity vector at that pixel to find the corresponding points in the two frames, and computing the time weighted average of the two points; and

ii) if it is a background pixel, finding the corresponding points in the two frames using the background velocity vector and if they are both background points, computing the time weighted average of the two points;

if only one of the two corresponding points is a background point, using the value of the background point; and

if they are both foreground point determining the value of the pixel from the values of neighboring background pixels in the intermediate image.