Motion compensated interpolation system using combination of full and intermediate frame occlusion

• US 8,542,322 B2
• Filed: 03/01/2010
• Issued: 09/24/2013
• Est. Priority Date: 03/01/2010
• Status: Active Grant

• Current Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Original Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Inventors: Petrides, Gordon
• Primary Examiner(s): Harold, Jefferey
• Assistant Examiner(s): HAIEM, SEAN N
• Application Number:
  US12/715,235
• Publication Number:
  US 20110211125A1
• Time in Patent Office: 1,303 Days
• Field of Search: None
• US Class Current: 348/616
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/17 the unit being an image region, e.g. an object
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/553 Motion estimation dealing with occlusions
• CPC Class Codes: H04N 19/82 involving filtering within a prediction loop

Abstract

In one embodiment of the present invention, motion compensated interpolation is performed by locating full frame conceal and reveal areas, determining intermediate frame occlusion areas of an interpolated frame of the displayable output by locating intermediate frame conceal areas based on projected locations of pixels within the full frame conceal areas using primary forward motion vectors and information about a time slot for the interpolated frame, and by locating intermediate frame reveal areas based on projected locations of pixels within the full frame reveal areas using primary backward motion vectors and information about the time slot for the interpolated frame. The intermediate frame conceal areas are then modified by projecting locations of pixels using secondary forward motion vectors and the intermediate frame reveal areas are modified by projecting locations of pixels using secondary backward motion vectors.

Claims

1. A method for performing motion compensated interpolation using a previous frame and a current frame of a displayable output, the method comprising:

determining full frame occlusion areas of the displayable output by locating full frame conceal areas where pixels of the previous frame have no match in the current frame and full frame reveal areas where pixels of the current frame have no match in the previous frame;

determining intermediate frame occlusion areas of an interpolated frame of the displayable output by locating intermediate frame conceal areas based on projected locations of pixels within the full frame conceal areas using primary forward motion vectors and information about a time slot for the interpolated frame, and by locating intermediate frame reveal areas based on projected locations of pixels within the full frame reveal areas using primary backward motion vectors and information about the time slot for the interpolated frame;

modifying the intermediate frame conceal areas by projecting locations of pixels using secondary forward motion vectors, wherein the secondary forward motion vectors are associated with a frame subsequent to the current frame; and

modifying the intermediate frame reveal areas by projecting locations of pixels using secondary backward motion vectors, wherein the secondary backward motion vectors are associated with a frame prior to the previous frame.

2. The method of claim 1, further comprising:

for any pixels in the interpolated frame to which there is neither a forward vector nor a backward vector projecting;

including the pixel in an intermediate frame conceal area if it is not located within the frame reveal area;

including the pixel in an intermediate frame reveal area if it is not located within the full frame conceal area; and

using the intermediate frame conceal and reveal areas to interpolate values for pixels within an area of the interpolated frame.

3. The method of claim 1, further comprising:

determining if a pixel location is outside a letterbox or pillarbox border of the displayable output; and

if the pixel location is outside the letterbox or pillarbox border of the displayable output, inserting the temporal average of the pixel values from the previous and current frames.

4. The method of claim 1, further comprising:

determining if a pixel location is outside a letterbox or pillarbox border of the displayable output; and

if the pixel location is outside the letterbox or pillarbox border of the displayable output, inserting a repeat of the pixel value from the previous frame.

5. The method of claim 1, further comprising:

determining if a pixel location is outside a letterbox or pillarbox border of the displayable output; and

if the pixel location is outside the letterbox or pillarbox border of the displayable output, inserting a repeat of the pixel value from the current frame.

6. The method of claim 1, further comprising setting a system to a full motion compensated mode, a temporal filtered motion compensated mode, or a motion compensated bypass mode on a frame by frame basis according to analysis of the picture content.

7. The method of claim 2, wherein the using the intermediate frame conceal and reveal areas to interpolate values for pixels within an area of the interpolated frame includes:

when the area is neither an intermediate frame reveal area nor an intermediate frame conceal area, using both forward vectors applied to the current frame and backward vectors applied to the previous frame to calculate values for pixels within the area;

when the area is an intermediate frame reveal area, using forward vectors applied to the current frame to calculate values for pixels within the area, without using backward vectors; and

when the area is an intermediate frame conceal area, using backward vectors applied to the previous frame to calculate values for pixels within the area, without using forward vectors.

8. The method of claim 1, wherein the determining intermediate frame occlusion areas includes weighting backward and forward motion vectors based upon the proximity of the time slot of the interpolated frame to the current and previous frames, respectively.

9. The method of claim 1, wherein the time slot indicates the temporal distance between the interpolated frame and the previous frame and between the interpolated frame and the current frame.

10. The method of claim 1, further comprising initializing a forward frame store and a backward frame store for the interpolated frame, wherein the forward and backward frame store each contain a single bit storage for each pixel of the interpolated frame, and wherein the initialization includes setting each of the single bit storages to zero.

11. The method of claim 10, further comprising:

setting corresponding pixels in the forward frame store to non-zero based on the projected locations of pixels using forward motion vectors and the previous frame and the information about a time slot for the interpolated frame; and

setting corresponding pixels in the backward frame store non-zero based on the projected locations of pixels using backward motion vectors and the information about a time slot for the interpolated frame.

12. The method of claim 10, wherein the determining intermediate frame occlusion areas of an interpolated frame includes identifying a pixel as part of an intermediate frame conceal region if the single bit storage corresponding to the pixel in the backward frame store is zero while the single bit storage corresponding to the pixel in the forward frame store is non-zero.

13. The method of claim 10, wherein the determining intermediate frame occlusion areas of an interpolated frame includes identifying a pixel as part of an intermediate frame reveal region if the single bit storage corresponding to the pixel in the forward frame store is zero while the single bit storage corresponding to the pixel in the backward frame store is non-zero.

14. The method of claim 10, wherein a pixel in the interpolated frame to which there is neither a forward vector nor a backward vector is identified based upon whether both the single bit storage corresponding to the pixel in the forward frame store and the single bit storage corresponding to the pixel in the backward frame store are zero.

15. The method of claim 1, further comprising using a histogram filter to aid in determining whether a region of pixels in the intermediate frame is an occlusion area.

16. The method of claim 15, wherein the histogram filter has an adjustable threshold, wherein the higher the threshold is set, the less likely that a region of pixels will be determined to be an occlusion area.

17. An interpolator comprising:

a forward data store containing an entry for each pixel in an intermediate frame to be interpolated from a current and previous frame of displayable output;

a backward data store containing an entry for each pixel in the intermediate frame;

a full frame occlusion generator configured to determine full frame occlusion areas of the displayable output by locating full frame conceal areas where pixels of the previous frame have no match in the current frame and full frame reveal areas where pixels of the current frame have no match in the previous frame;

an intermediate frame occlusion generator configured to determine intermediate frame occlusion areas of an interpolated frame of the displayable output by locating intermediate frame conceal areas based on projected locations of pixels within the full frame conceal areas using primary forward motion vectors and information about a time slot for the interpolated frame, and by locating intermediate frame reveal areas based on projected locations of pixels within the full frame reveal areas using primary backward motion vectors and information about the time slot for the interpolated frame; and

an intermediate frame occlusion modifier configured to;

modify the intermediate frame conceal areas by projecting locations of pixels using secondary forward motion vectors, wherein the secondary forward motion vectors are associated with a frame subsequent to the current frame; and

modify the intermediate frame reveal areas by projecting locations of pixels using secondary backward motion vectors, wherein the secondary backward motion vectors are associated with a frame prior to the previous frame.

18. The interpolator of claim 17, further comprising a histogram filterer coupled to the intermediate frame occlusion generator, the forward data store and the backward data store.

19. The interpolator of claim 17, wherein the full frame occlusion generator is a halo reducing interpolator.

20. An apparatus for performing motion compensated interpolation using a previous frame and a current frame of a displayable output, the apparatus comprising:

means for determining full frame occlusion areas of the displayable output by locating full frame conceal areas where pixels of the previous frame have no match in the current frame and full frame reveal areas where pixels of the current frame have no match in the previous frame;

means for determining intermediate frame occlusion areas of an interpolated frame of the displayable output by location intermediate frame conceal areas based on projected locations of pixels within the full frame conceal areas using primary forward motion vectors and information about a time slot for the interpolated frame, and by locating intermediate frame reveal areas based on projected locations of pixels within the full frame reveal areas using primary backward motion vectors and information about the time slot for the interpolated frame;

means for modifying the intermediate frame conceal areas by projecting locations of pixels using secondary forward motion vectors, wherein the secondary forward motion vectors are associated with a frame subsequent to the current frame; and

means for modifying the intermediate frame reveal areas by projecting locations of pixels using secondary backward motion vectors, wherein the secondary backward motion vectors are associated with a frame prior to the previous frame.

21. The apparatus of claim 20, further comprising:

means for, for any pixels in the interpolated frame to which there is neither a forward vector nor a backward vector projecting;

including the pixel in an intermediate frame conceal area if it is not located within the full frame reveal area; and

including the pixel in an intermediate frame reveal area if it is not located within the full frame conceal area; and

means for using the intermediate frame conceal and reveal areas to interpolate values for pixels within an area of the interpolated frame.

22. A computer chip configured to:

determine full frame occlusion areas of the displayable output by locating full frame conceal areas where pixels of the previous frame have no match in the current frame and full frame reveal areas where pixels of the current frame have no match in the previous frame;

determine intermediate frame occlusion areas of an interpolated frame of the displayable output by locating intermediate frame conceal areas based on projected locations of pixels within the full frame conceal areas using primary forward motion vectors and information about a time slot for the interpolated frame, and by locating intermediate frame reveal areas based on projected locations of pixels within the full frame reveal areas using primary backward motion vectors and information about the time slot for the interpolated frame;

modify the intermediate frame conceal areas by projecting locations of pixels using secondary forward motion vectors, wherein the secondary forward motion vectors are associated with a frame subsequent to the current frame; and

modify the intermediate frame reveal areas by projecting locations of pixels using secondary backward motion vectors, wherein the secondary backward motion vectors are associated with a frame prior to the previous frame.