Method for temporal interpolation of an image sequence using object-based image analysis

• US 6,625,333 B1
• Filed: 08/06/1999
• Issued: 09/23/2003
• Est. Priority Date: 08/06/1999
• Status: Expired due to Term

• Current Assignee: E.B. Wells Electronics II, LLC
• Original Assignee: Her Majesty The Queen In Right of Canada As Represented By The Minister of National Defence
• Inventors: Wang, Demin, Vincent, André
• Primary Examiner(s): Mehta, Bhavesh M.
• Assistant Examiner(s): PATEL, KANJIBHAI B
• Application Number:
  US09/369,347
• Time in Patent Office: 1,509 Days
• Field of Search: 382/236, 382/300, 382/173, 382/174, 345/606-611, 345/422, 345/582, 345/581, 348/459, 348/441, 348/699, 348/700, 358/525, 358/428, 375/240.16
• US Class Current: 382/300
• CPC Class Codes: G06T 3/4007 based on interpolation, e.g. bilinear interpolation image demosaicing G06T3/4015; edge-driven or edge-based scaling G06T3/403
• CPC Class Codes: G06T 7/215 Motion-based segmentation
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

A method for temporal interpolation of an image sequence using object-based image analysis is disclosed. The method comprises the steps of image segmentation for partitioning a known image into objects, motion estimation, object-based motion field processing, determination of object depth order, detection of covered areas and newly exposed areas, adaptive interpolation using motion projection, and graceful degradation for alleviating remaining visible artifacts. The method produces interpolated images of high quality in real time for every type of image areas including fast moving objects, covered areas, and newly exposed areas.

Claims

1. A method for generating an image from at least two known images of an image sequence comprising the steps of:

a) segmenting at least one of the at least two known images into objects, the objects having a homogeneous interior with respect to luminance, colour, and/or texture;

b) estimating motion from the at least one segmented known image towards another of the at least two known images and assigning a motion vector to each pixel of the at least one segmented known image generating an estimated motion field;

c) processing the estimated motion field to obtain a smooth motion field within each object;

d) calculating pixel values of the image to be interpolated using object-based motion projection;

e) determining object-depth order and detecting covered areas;

f) projecting every object of the segmented known image to another known image and calculating a projection error for each pixel of the segmented known image;

g) calculating a total projection error for each of two overlapping areas; and

,

h) comparing the total projection errors of overlapping areas and assigning a depth-order to each of the objects which contain the overlapping areas.

2. A method for generating an image from at least two known images of an image sequence as defined in claim 1, comprising the steps of:

determining covered areas in dependence upon the depth-order; and

,

generating a mask containing shape information of the covered areas.

3. A method for generating an image from at least two known images of an image sequence as defined in claim 1, comprising the steps of:

modifying the motion field within objects comprising a large covered area by replacing the motion vectors within the object with motion vectors interpolated from motion vectors of adjacent objects.

4. A method for generating an image from at least two known images of an image sequence as defined in claim 1, wherein step d) comprises the step of:

calculating pixel values of the image to be interpolated as a weighted average of pixel values of the segmented known image and the at least other known image.

5. A method for generating an image from at least two known images of an image sequence as defined in claim 1, comprising the step of:

retaining one projection in areas where projections overlap each other according to the depth-order of objects.

6. A method for generating an image from at least two known images of an image sequence as defined in claim 5, wherein the pixel values within projected areas corresponding to covered areas are equal to the pixel values of the segmented known image.

7. A method for generating an image from at least two known images of an image sequence as defined in claim 5, wherein the pixel values within projected areas corresponding to newly exposed areas are determined using only the pixel values of the other known image.

8. A method for generating an image from at least two known images of an image sequence as defined in claim 7, comprising the step of:

determining pixel values for pixels within a band surrounding newly exposed areas using only the pixel values of the other known image if an interpolation error of pixel values of the pixels within the band obtained as a weighted average of pixel values of the segmented known image and the at least other known image is greater than a predetermined threshold.

9. A method for temporal interpolating an image from two known images of an image sequence comprising the steps of:

a) segmenting one of the two known images which is closer to the image to be interpolated into objects, the objects having a homogeneous interior with respect to luminance and colour;

b) estimating motion from the segmented known image towards the other known image and assigning a motion vector to each pixel of the segmented known image generating an estimated motion field;

c) processing the estimated motion field to obtain a smooth motion field within each object comprising the steps of;

approximating the estimated motion field within each object using an affine motion model describing a smooth motion field within each object;

comparing the estimated motion field with the approximated motion field to determine an approximation error for each motion vector of the estimated motion field;

classifying the motion vectors of the estimated motion field into correctly estimated motion vectors and erroneous motion vectors by comparing the approximation error of each estimated motion vector to a threshold; and

,

replacing each erroneous motion vector with a motion vector obtained by interpolating correct motion vectors closest to the erroneous motion vector using an object-adaptive interpolator, the correct motion vectors belonging to a same object as the erroneous motion vector;

d) determining object-depth order and detecting covered areas comprising the steps of;

projecting every object of the segmented known image to another known image and calculating a projection error for each pixel of the segmented known image;

calculating a total projection error for each of two overlapping areas;

comparing the total projection errors of overlapping areas and assigning a depth-order to each of the objects which contain the overlapping areas;

determining covered areas in dependence upon the depth-order; and

,

generating a mask containing shape information of the covered areas;

e) calculating pixel values of the image to be interpolated using object-based motion projection; and

,

f) alleviating remaining visible artifacts by eliminating false sharp edges by replacing the value of a pixel within the interpolated image with an average value within a window centered on the pixel if the motion vector of the pixel is greater than a threshold.

10. A method for temporal interpolating an image from two known images of an image sequence comprising the steps of:

a) segmenting the first and the second known image into objects, the objects having a homogeneous interior with respect to luminance and colour;

b) estimating motion from the first segmented known image towards the second segmented known image and assigning a motion vector to each pixel of the first segmented known image generating a first estimated motion field;

c) estimating motion from the second segmented known image towards the first segmented known image and assigning a motion vector to each pixel of the second segmented known image generating a second estimated motion field;

c) processing the first and second estimated motion field to obtain a smooth first and second motion field within each object;

d) determining object-depth order and detecting covered areas for the first and second motion field;

e) calculating pixel values of a first image to be interpolated based on the first motion field using object-based motion projection;

f) calculating pixel values of a second image to be interpolated second motion field using object-based motion projection;

g) alleviating remaining visible artifacts in the first interpolated image by eliminating false sharp edges through graceful degradation;

h) alleviating remaining visible artifacts in the second interpolated image by eliminating false sharp edges through graceful degradation; and

,

i) determining the image to be interpolated as a weighted average of the first interpolated image and the second interpolated image.

11. A method for temporal interpolating an image from two known images of an image sequence as defined in claim 10, wherein weighting coefficients in step i) are determined in dependence upon interpolation error and time distance between the two known images and the interpolated images.