Intermediate frame occlusion estimation system using projected vectors

• US 8,289,446 B2
• Filed: 03/01/2010
• Issued: 10/16/2012
• Est. Priority Date: 03/01/2010
• Status: Active Grant

• Current Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Original Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Inventors: Petrides, Gordon
• Primary Examiner(s): Tran, Trang U
• Application Number:
  US12/715,067
• Publication Number:
  US 20110211119A1
• Time in Patent Office: 960 Days
• Field of Search: 348/441, 348451-452, 348/448, 348458-459, 348/576, 348/616, 348/620, 348699-700, 348415-416, 348/410, 348/409, 348/413, 375/240.01, 375/240.16, 375/240.12, 375/240.14, 375/240.24, 382/260, 382/215, 382/265, 382/238, 382/236
• US Class Current: 348/441
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter

Abstract

In one embodiment, a method is provided for performing motion compensated interpolation wherein, for any pixels in the interpolated frame to which there is neither a forward vector nor a backward vector projecting: including the pixel in an intermediate frame conceal area if it is not located within the full frame reveal area; including the pixel in an intermediate frame reveal area if it is not located within the full frame conceal area; and using the intermediate frame conceal and reveal areas to interpolate values for pixels within an area of the interpolated frame.

Claims

1. A method for performing motion compensated interpolation using a previous frame and a current frame of a displayable output, the method comprising:

determining full frame occlusion areas of the displayable output by locating full frame conceal areas where pixels of the previous frame have no match in the current frame and full frame reveal areas where pixels of the current frame have no match in the previous frame;

determining intermediate frame occlusion areas of an interpolated frame of the displayable output by locating intermediate frame conceal areas based on projected locations of pixels within the full frame conceal areas using forward motion vectors and information about a time slot for the interpolated frame, and by locating intermediate frame reveal areas based on projected locations of pixels within the full frame reveal areas using backward motion vectors and information about the time slot for the interpolated frame;

for any pixels in the interpolated frame to which there is neither a forward vector nor a backward vector projecting;

including the pixel in an intermediate frame conceal area if it is not located within the full frame reveal area;

including the pixel in an intermediate frame reveal area if it is not located within the full frame conceal area; and

using the intermediate frame conceal and reveal areas to interpolate values for pixels within an area of the interpolated frame.

2. The method of claim 1, wherein the using the intermediate frame conceal and reveal areas to interpolate values for pixels within an area of the interpolated frame includes:

when the area is neither an intermediate frame reveal area nor an intermediate frame conceal area, using both forward vectors applied to the current frame and backward vectors applied to the previous frame to calculate values for pixels within the area;

when the area is an intermediate frame reveal area, using forward vectors applied to the current frame to calculate values for pixels within the area, without using backward vectors; and

when the area is an intermediate frame conceal area, using backward vectors applied to the previous frame to calculate values for pixels within the area, without using forward vectors.

3. The method of claim 1, wherein the determining intermediate frame occlusion areas includes weighting backward and forward motion vectors based upon the proximity of the time slot of the interpolated frame to the current and previous frames, respectively.

4. The method of claim 1, wherein the time slot indicates the temporal distance between the interpolated frame and the previous frame and between the interpolated frame and the current frame.

5. The method of claim 1, further comprising initializing a forward frame store and a backward frame store for the interpolated frame, wherein the forward and backward frame store each contain a single bit storage for each pixel of the interpolated frame, and wherein the initialization includes setting each of the single bit storages to zero.

6. The method of claim 5, further comprising:

setting corresponding pixels in the forward frame store to non-zero based on the projected locations of pixels using forward motion vectors and the previous frame and the information about a time slot for the interpolated frame; and

setting corresponding pixels in the backward frame store non-zero based on the projected locations of pixels using backward motion vectors and the information about a time slot for the interpolated frame.

7. The method of claim 6, wherein the determining intermediate frame occlusion areas of an interpolated frame includes identifying a pixel as part of an intermediate frame conceal area if the single bit storage corresponding to the pixel in the backward frame store is zero while the single bit storage corresponding to the pixel in the forward frame store is non-zero.

8. The method of claim 6, wherein the determining intermediate frame occlusion areas of an interpolated frame includes identifying a pixel as part of an intermediate frame reveal area if the single bit storage corresponding to the pixel in the forward frame store is zero while the single big storage corresponding to the pixel in the backward frame store is non-zero.

9. The method of claim 6, wherein a pixel in the interpolated frame to which there is neither a forward vector nor a backward vector projecting is identified based upon whether both the single bit storage corresponding to the pixel in the forward frame store and the single bit storage corresponding to the pixel in the backward frame store are zero.

10. The method of claim 1, further comprising using a histogram filter to aid in determining whether a region of pixels in the intermediate frame is an occlusion area.

11. The method of claim 10, wherein the histogram filter has an adjustable threshold, wherein the higher the threshold is set, the less likely that a region of pixels will be determined to be an occlusion area.

12. An intermediate frame occlusion estimation system comprising:

a forward data store containing an entry for each pixel in an intermediate frame to be interpolated from a current and previous frame of displayable output;

a backward data store containing an entry for each pixel in the intermediate frame;

a full frame occlusion generator configured to;

determine full frame occlusion areas of the displayable output by locating full frame conceal areas where pixels of the previous frame have no match in the current frame and full frame reveal areas where pixels of the current frame have no match in the previous frame;

an intermediate frame occlusion generator configured to;

determine intermediate frame occlusion areas of the interpolated frame by locating intermediate frame conceal areas based on projected locations of pixels within the full frame conceal areas using motion vectors and information about a time slot for the interpolated frame, and by locating intermediate frame reveal areas based on projected locations of pixels within the full frame reveal areas using backward motion vectors and information about the time slot for the interpolated frame;

for any pixels in the interpolated frame to which there is neither a forward vector nor a backward vector projecting;

include the pixel in an intermediate frame conceal area if it is not located within the full frame reveal area;

include the pixel in an intermediate frame reveal area if it is not located within the full frame conceal area, and use the intermediate frame conceal and reveal areas to interpolate values for pixels within an area of the interpolated frame.

13. The intermediate frame occlusion estimation system of claim 12, further comprising a histogram filter coupled to the intermediate frame occlusion generator, the forward data store and the backward data store.

14. The intermediate frame occlusion estimation system of claim 12, wherein the full frame occlusion generator is a halo reducing interpolator.

15. The intermediate frame occlusion estimation system of claim 12, further comprising an interpolator coupled to the intermediate frame occlusion generator.

16. An apparatus for performing motion compensated interpolation using a previous frame and a current frame of a displayable output, the apparatus comprising:

means for determining full frame occlusion areas of the displayable output by locating full frame conceal areas where pixels of the previous frame have no match in the current frame and full frame reveal areas where pixels of the current frame have no match in the previous frame;

means for determining intermediate frame occlusion areas of an interpolated frame of the displayable output by locating intermediate frame conceal areas based on projected locations of pixels within the full frame conceal areas using motion vectors and information about a time slot for the interpolated frame, and by locating intermediate frame reveal areas based on projected locations of pixels within the full frame reveal areas using backward motion vectors and information about the time slot for the interpolated frame;

means for, for any pixels in the interpolated frame to which there is neither a forward vector nor a backward vector projecting;

including the pixel in an intermediate frame conceal area if it is not located within the full frame reveal area;

including the pixel in an intermediate frame reveal area if it is not located within the full frame conceal area; and

means for using the intermediate frame conceal and reveal areas to interpolate values for pixels within an area of the interpolated frame.

17. The apparatus of claim 16, wherein the means for using the intermediate frame conceal and reveal areas to interpolate values for pixels within an area of the interpolated frame includes:

means for, when the area is neither an intermediate frame reveal area nor an intermediate frame conceal area, using both forward vectors applied to the current frame and backward vectors applied to the previous frame to calculate values for pixels within the area;

means for, when the area is an intermediate frame reveal area, using forward vectors applied to the current frame to calculate values for pixels within the area, without using backward vectors; and

means for, when the area is an intermediate frame conceal area, using backward vectors applied to the previous frame to calculate values for pixels within the area, without using forward vectors.