System and method for frame interpolation for a compressed video bitstream

• US 20100201870A1
• Filed: 02/09/2010
• Published: 08/12/2010
• Est. Priority Date: 02/11/2009
• Status: Abandoned Application

• Current Assignee: Aggelos Katsaggelos, Dusan Veselinovic, James J. Kosmach, Krisda Lengwehasatit, Martin Luessi
• Original Assignee: Aggelos Katsaggelos, Dusan Veselinovic, James J. Kosmach, Krisda Lengwehasatit, Martin Luessi
• Inventors: Lengwehasatit, Krisda, Luessi, Martin, Katsaggelos, Aggelos, Kosmach, James J., Veselinovic, Dusan
• Application Number:
  US12/658,470
• Publication Number:
  US 20100201870A1
• Time in Patent Office: Days
• US Class Current: 348/452
• CPC Class Codes: G06T 1/00 General purpose image data processing
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

A system and a method perform frame interpolation for a compressed video bitstream. The system and the method may combine candidate pictures to generate an interpolated video picture inserted between two original video pictures. The system and the method may generate the candidate pictures from different motion fields. The candidate pictures may be generated partially or wholly from motion vectors extracted from the compressed video bitstream. The system and the method may reduce computation required for interpolation of video frames without a negative impact on visual quality of a video sequence.

Claims

1. A method for frame interpolation for a bitstream encoding a first source image and a second source image which is encoded subsequent to the first source image wherein a device receives the bitstream, the method comprising the steps of:

decoding the first source image and the second source image from the bitstream;

performing a first motion estimation which uses the first source image and the second source image to create a first motion field wherein the first source image is a reference grid for the first motion estimation;

performing a first motion compensation which uses the first motion field to create a forward candidate interpolation picture;

performing a second motion estimation which uses the first source image and the second source image to create a second motion field which is a different motion field than the first motion field wherein the second source image is a reference grid for the second motion estimation;

performing a second motion compensation which uses the second motion field to create a backward candidate interpolation picture;

performing a third motion estimation which uses the first source image and the second source image to create a third motion field which is a different motion field than the first motion field and the second motion field wherein a bidirectional candidate interpolation picture is a reference grid for the third motion estimation;

performing a third motion compensation which uses the third motion field to create the bidirectional candidate interpolation picture;

determining an estimated visual quality of a final interpolated picture formed by a combination of the forward candidate interpolation picture, the backward candidate interpolation picture and the bidirectional candidate interpolation picture; and

displaying the final interpolated picture if the estimated visual quality exceeds a threshold.

2. The method of claim 1 further comprising the step of:

applying a first sum of absolute difference operation to the forward candidate interpolation picture and the backward candidate interpolation picture, a second sum of absolute difference operation to the forward candidate interpolation picture and the bidirectional candidate interpolation picture, and a third sum of absolute difference operation to the backward candidate interpolation picture and the bidirectional candidate interpolation picture wherein results of the first sum of absolute difference operation, the second sum of absolute difference operation and the third sum of absolute difference operation are used to determine the estimated visual quality of the final interpolated picture.

3. The method of claim 1 further comprising the step of:

performing a median filtering operation for the forward candidate interpolation picture, the backward candidate interpolation picture and the bidirectional candidate interpolation picture wherein the median filtering operation combines the forward candidate interpolation picture, the backward candidate interpolation picture and the bidirectional candidate interpolation picture to produce the final interpolated picture.

4. The method of claim 1 further comprising the step of:

determining an estimated number of blocks in the final interpolated picture which are likely to have motion artifacts wherein the estimated number of blocks in the final interpolated picture which are likely to have motion artifacts is determined without combining the forward candidate interpolation picture, the backward candidate interpolation picture and the bidirectional candidate interpolation picture to produce the final interpolated picture and further wherein the estimated visual quality of the final interpolated picture is based on the estimated number of blocks in the final interpolated picture which are likely to have motion artifacts.

5. The method of claim 1 wherein at least one of the first motion estimation, the second motion estimation and the third motion estimation use enhanced predictive zonal search motion estimation.

6. The method of claim 1 further comprising the step of:

performing overlapped block motion compensation to at least one of the forward candidate interpolation picture, the backward candidate interpolation picture and the bidirectional candidate interpolation picture wherein the overlapped block motion compensation is performed in a corresponding one of the first motion compensation, the second motion compensation and the third motion compensation.

7. The method of claim 1 further comprising the step of:

using parameters encoded by the bitstream to determine whether to use motion vectors encoded by the bitstream in the first motion estimation and the second motion estimation for a block of one of the first source image and the second source image.

8. The method of claim 1 further comprising the step of:

using information encoded by the bitstream to determine whether to split a 16×

16 block of one of the first source image and the second source image into smaller blocks for at least one of the first motion estimation, the second motion estimation and the third motion estimation wherein each of the smaller blocks is associated with a motion vector.

9. The method of claim 1 further comprising the step of:

using an estimate of a number of blocks of the final interpolated picture which are likely to have motion artifacts to determine a presence of a scene change wherein the forward candidate interpolation picture, the backward candidate interpolation picture and the bidirectional candidate interpolation picture are not combined to form the final interpolated picture if the presence of the scene change is determined.

10. The method of claim 1 further comprising the step of:

using frame repetition to extend display of the first source image before displaying the second source image if the estimated visual quality is below the threshold wherein the forward candidate interpolation picture, the backward candidate interpolation picture and the bidirectional candidate interpolation picture are not combined to form the final interpolated picture if the estimated visual quality is below the threshold.

11. The method of claim 1 further comprising the step of:

resetting at least one of the first motion field, the second motion field and the third motion field with zero motion vectors if an estimated number of blocks in the final interpolated picture which are likely to have motion artifacts does not meet a predetermined value.

12. The method of claim 1 further comprising the step of:

rotating at least one of the first motion field, the second motion field and the third motion field wherein rotating the at least one of the first motion field, the second motion field and the third motion field causes a current motion field to become a previous motion field and further wherein the first motion estimation, the first motion compensation, the second motion estimation, the second motion compensation, the third motion estimation and the third motion compensation are repeated using the motion fields which are rotated, the second source image and a third source image which is encoded subsequent to the second source image in the bitstream.

13. The method of claim 1 further comprising the step of:

performing chroma channel motion compensation on the final interpolated picture using the first motion field, the second motion field and the third motion field.

14. A method for frame interpolation for a bitstream encoding a first source image and a second source image subsequent to the first source image wherein the first source image and the second source image are formed by macroblocks and further wherein motion vectors are encoded by the bitstream wherein each of the macroblocks is associated with at least one of the motion vectors and further wherein the bitstream encodes block mode information wherein a device receives the bitstream, the method comprising the steps of:

determining reliable motion vectors of the motion vectors encoded by the bitstream wherein the motion vectors and the block mode information are used to determine the reliable motion vectors;

performing a first motion estimation which uses the first source image and the second source image to create a first motion field wherein the first source image is a reference grid for the first motion estimation and further wherein the first motion estimation uses the reliable motion vectors;

performing a first motion compensation which uses the first motion field to create a forward candidate interpolation picture;

performing a second motion estimation which uses the first source image and the second source image to create a second motion field which is a different motion field than the first motion field wherein the second source image is a reference grid for the second motion estimation and further wherein the second motion estimation uses the reliable motion vectors;

performing a second motion compensation which uses the second motion field to create a backward candidate interpolation picture;

performing a third motion estimation which uses the first source image and the second source image to create a third motion field which is a different motion field than the first motion field and the second motion field wherein a bidirectional candidate interpolation picture is a reference grid for the third motion estimation;

performing a third motion compensation which uses the third motion field to create the bidirectional candidate interpolation picture; and

displaying the first source image, the second source image and an interim image wherein the interim image is displayed after the first source image and before the second source image.

15. The method of claim 14 further comprising the steps of:

determining an estimated number of blocks in a final interpolated picture which are likely to have motion artifacts wherein the final interpolated picture is a combination of the forward candidate interpolation picture, the backward candidate interpolation picture, and the bidirectional candidate interpolation picture and further wherein the estimated number of blocks which are likely to have motion artifacts is determined without combining the forward candidate interpolation picture, the backward candidate interpolation picture, and the bidirectional candidate interpolation picture to produce the final interpolated picture;

identifying one of the final interpolated picture and a frame repetition of the first source image to use as the interim image wherein identification is based on the estimated number of blocks in the final interpolated picture which are likely to have the motion artifacts; and

forming the interim image wherein the interim image is formed using median filtering to combine the forward candidate interpolation picture, the backward candidate interpolation picture and the bidirectional candidate interpolation picture if the final interpolated picture is identified for use as the interim image and further wherein the interim image is formed using the frame repetition of the first source image if the frame repetition of the first source image is identified for use as the interim image.

16. The method of claim 14 further comprising:

determining whether to split blocks used in the first motion estimation and the second motion estimation into smaller blocks based on the block mode information encoded by the bitstream wherein each of the smaller blocks is associated with at least one of the motion vectors and further wherein the smaller blocks correspond to areas of increased density of the first motion field and the second motion field.

17. The method of claim 14 wherein the bitstream is a H.264 compressed video bitstream.

18. A system for frame interpolation for a bitstream encoding a first source image and a second source image, the system comprising:

a mobile device which receives the bitstream;

a processor connected to the mobile device which decodes the first source image and the second source image from the bitstream; and

an application executed by the mobile device which directs the processor to use the first source image and the second source image to generate at least three candidate interpolation pictures wherein the processor applies a sum of absolute difference operation to the at least three candidate interpolation pictures to estimate a number of blocks which are likely to have motion artifacts in a final interpolated picture formed by the at least three candidate interpolation pictures.

19. The system of claim 18 wherein the processor uses the number of blocks which are likely to have motion artifacts to determine a presence of a scene change between the first source image and the second source image and further wherein the processor does not form the final interpolated picture if the processor determines the presence of the scene change wherein the mobile device uses frame repetition in displaying the first source image before the second source image if the processor determines the presence of the scene change.

20. The system of claim 18 wherein the processor uses the number of blocks which are likely to have motion artifacts to estimate a visual quality of the final interpolated picture and further wherein the processor forms the final interpolated picture from the at least three candidate interpolation pictures if the visual quality estimated meets a threshold wherein the mobile device displays the first source image, the final interpolated picture and the second source image.

21. The system of claim 18 wherein the processor uses the number of blocks which are likely to have motion artifacts to estimate a visual quality of the final interpolated picture and further wherein the processor does not form the final interpolated picture if the visual quality estimated does not meet a threshold wherein the mobile device uses frame repetition to extend display of the first source image before displaying the second source image if the visual quality estimated does not meet the threshold.