BORDER HANDLING FOR MOTION COMPENSATED TEMPORAL INTERPOLATOR USING CAMERA MODEL

• US 20110211083A1
• Filed: 03/01/2010
• Published: 09/01/2011
• Est. Priority Date: 03/01/2010
• Status: Active Grant

• Current Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Original Assignee: STMicroelectronics Incorporated (STMicroelectronics NV)
• Inventors: Petrides, Gordon
• Granted Patent: US 9,013,584 B2
• Time in Patent Office: Days
• US Class Current: 348/208.4
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20212 Image combination
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/005 Retouching; Inpainting; Scratch removal detecting, correction, reducing or removing defects, e.g. non-responsive pixels of solid state image sensors H04N25/68, scratch removal for cinematographic films scanned by electronic image sensor H04N5/253
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/77 Retouching; Inpainting; Scratch removal
• CPC Class Codes: H04N 23/683 performed by a processor, e.g. controlling the readout of an image memory
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

In one embodiment of the present invention, a method is provided for providing border handling in motion compensated interpolation, wherein a camera model is used to detect areas of predominant movements in a displayable output, and for each reveal or conceal area: it is determined whether the reveal or conceal areas is due to a camera movement or an object movement. if the reveal or conceal areas is due to a camera movement. The camera model is used to arrive at interpolated pixel values for pixels within the corresponding area. If the reveal or conceal areas is not due to a camera movement, arriving at interpolated pixel values for pixels within the corresponding area without using the camera model.

Claims

1. A method for providing border handling in motion compensated interpolation, the method comprising:

using a camera model to detect areas of predominant movements in a displayable output, wherein the camera model identifies probable values for pixels in border areas of the displayable output based on common camera movements;

locating reveal and conceal areas near the borders of the displayable output, wherein a reveal area is an area visible in a current frame that was not visible in a previous frame and a conceal area is an area visible in a previous frame that is not visible in a current frame;

for each reveal or conceal area;

determining whether the reveal or conceal areas is due to a camera movement or an object movement;

if the reveal or conceal areas is due to a camera movement, using the camera model to arrive at interpolated pixel values for pixels within the corresponding area; and

if the reveal or conceal areas is not due to a camera movement, arriving at interpolated pixel values for pixels within the corresponding area without using the camera model.

2. The method of claim 1, wherein the common camera movements include pans, zooms, and rotations.

3. The method of claim 1, wherein the arriving at interpolated pixel values for pixels within the corresponding area without using the camera model includes using a halo reducing interpolation process.

4. The method of claim 3, wherein the halo-reducing interpolation process includes:

determining if motion of an object within the displayable output with respect to motion of a background within the displayable output is such that a particular area of an interpolated frame is an object reveal region, an object conceal region, or neither;

when the area is neither an object reveal region nor an object conceal region, using both forward vectors applied to the current frame and backward vectors applied to the previous frame to calculate values for pixels within the area;

when the area is an object reveal region, using forward vectors applied to the current frame to calculate values for pixels within the area, without using backward vectors; and

when the area is an object conceal region, using backward vectors applied to the previous frame to calculate values for pixels within the area, without using forward vectors.

5. The method of claim 1, wherein the borders are the top and bottom edges of the displayable output for letterbox output, and the left and right edges of the displayable output for pillarbox output.

6. A border handling motion compensation interpolator comprising:

a camera modeler configured to model movements in a displayable output and identify probable reveal and conceal areas for the displayable output;

a border occlusion generator coupled to the camera model and configured to generate probable values for pixels based upon output from the camera modeler;

an interpolator configured to identify values a border occlusion generator configured to generate probable values for pixels not based upon output from the camera modeler; and

a selector configured to select between the probable values from the border occlusion generator and the probable values from the interpolator based upon whether a particular reveal or conceal area is caused by camera movement or by object movement.

7. The border handling motion compensation interpolator of claim 6, wherein the interpolator is a conventional motion compensated pixel interpolator.

8. The border handling motion compensation interpolator of claim 6, wherein the interpolator is a halo reducing interpolator.

9. The border handling motion compensation interpolator of claim 6, further comprising a letterbox detector configured to indicate whether the displayable output is in letterbox or pillarbox format.

10. The border handling motion compensation interpolator of claim 9, wherein the border occlusion generator acts only on reveal or conceal areas near the top and bottom borders of a letterbox displayable output, or near the left and right borders of a pillarbox displayable output.

11. The border handling motion compensation interpolator of claim 6, further comprising an edge region vector/camera vector correlation accumulator.

12. The border handling motion compensation interpolator of claim 11, further comprising a vector field generator coupled to the camera modeler and to the edge region vector/camera vector correlation accumulator.

13. An interpolation system comprising:

means for using a camera model to detect areas of predominant movements in a displayable output, wherein the camera model identifies probable values for pixels in border areas of the displayable output based on common camera movements;

means for locating reveal and conceal areas near the borders of the displayable output, wherein a reveal area is an area visible in a current frame that was not visible in a previous frame and a conceal area is an area visible in a previous frame that is not visible in a current frame;

means for, for each reveal or conceal area;

determining whether the reveal or conceal areas is due to a camera movement or an object movement;

if the reveal or conceal areas is due to a camera movement, using the camera model to arrive at interpolated pixel values for pixels within the corresponding area; and

if the reveal or conceal areas is not due to a camera movement, arriving at interpolated pixel values for pixels within the corresponding area without using the camera model.

14. The interpolation system of claim 13, wherein the common camera movements include pans, zooms, and rotations.

15. The interpolation system of claim 13, wherein the arriving at interpolated pixel values for pixels within the corresponding area without using the camera model includes using a halo reducing interpolation process.

16. The interpolation system of claim 15, wherein the halo-reducing interpolation process includes:

determining if motion of an object within the displayable output with respect to motion of a background within the displayable output is such that a particular area of an interpolated frame is an object reveal region, an object conceal region, or neither;

when the area is neither an object reveal region nor an object conceal region, using both forward vectors applied to the current frame and backward vectors applied to the previous frame to calculate values for pixels within the area;

when the area is an object reveal region, using forward vectors applied to the current frame to calculate values for pixels within the area, without using backward vectors; and

when the area is an object conceal region, using backward vectors applied to the previous frame to calculate values for pixels within the area, without using forward vectors.

17. The interpolation system of claim 13, wherein the borders are the top and bottom edges of the displayable output for letterbox output, and the left and right edges of the displayable output for pillarbox output.

18. A computer chip configured to:

use a camera model to detect areas of predominant movements in a displayable output, wherein the camera model identifies probable values for pixels in border areas of the displayable output based on common camera movements;

locate reveal and conceal areas near the borders of the displayable output, wherein a reveal area is an area visible in a current frame that was not visible in a previous frame and a conceal area is an area visible in a previous frame that is not visible in a current frame;

for each reveal or conceal area;

determine whether the reveal or conceal areas is due to a camera movement or an object movement;

if the reveal or conceal areas is due to a camera movement, use the camera model to arrive at interpolated pixel values for pixels within the corresponding area; and

if the reveal or conceal areas is not due to a camera movement, arrive at interpolated pixel values for pixels within the corresponding area without using the camera model.