Quality control in frame interpolation with motion analysis

• US 7,558,320 B2
• Filed: 06/13/2003
• Issued: 07/07/2009
• Est. Priority Date: 06/13/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Winder, Simon Anthony John, Ribas-Corbera, Jordi
• Primary Examiner(s): AN, SHAWN S
• Application Number:
  US10/461,730
• Publication Number:
  US 20040252759A1
• Time in Patent Office: 2,216 Days
• Field of Search: 375/240.12, 375/240.29, 375/240.16, 375/240.26, 348/699, 348/700, 382/260
• US Class Current: 375/240.12
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/537 Motion estimation other than block-based
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 21/440281 by altering the temporal resolution, e.g. by frame skipping television signal recording using magnetic recording on tape for reproducing at a rate different from the recording rate H04N5/783
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 5/147 Scene change detection
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

Techniques and tools for quality control in frame interpolation and motion analysis are described. The techniques and tools may be implemented separately or in combination in software and/or hardware devices for various applications. For example, a media playback device uses quality control in frame interpolation with motion analysis to increase the frame rate of streamed video for playback. The device selectively uses frame synthesis to increase frame rate and quality under normal circumstances, but avoids using frame synthesis when it would not provide suitable quality improvement. The device uses selective ghosting reduction and selective feathering to reduce artifacts in synthesized frames.

Claims

1. A method comprising:

receiving a pair of source video frames in a video sequence;

determining whether frame synthesis with motion analysis is on, wherein scene change detection indicates whether the frame synthesis with motion analysis is on;

if the frame synthesis with motion analysis is on, synthesizing pixel values for an intermediate video frame between the source video frames using frame interpolation with motion analysis, wherein frame interpolation with motion analysis includes global motion estimation,

otherwise, synthesizing pixel values for the intermediate video frame using a fallback mode or skipping synthesis of the intermediate video frame; and

providing the source video frames and the intermediate video frame, if any, for display.

2. The method of claim 1 wherein the fallback mode comprises simple averaging of pixel values of the source video frames.

3. The method of claim 1 wherein the fallback mode comprises using only pixel values of a selected one of the source video frames.

4. The method of claim 1 wherein one or more bits in a bitstream signal whether the frame synthesis with motion analysis is on or off.

5. The method of claim 4 wherein the one or more bits are at frame level.

6. The method of claim 4 wherein the one or more bits are at sequence level.

7. The method of claim 4 wherein the one or more bits also signal a count of intermediate video frames between the source video frames.

8. The method of claim 1 wherein quality control measurement indicates whether the frame synthesis with motion analysis is on or off.

9. The method of claim 8 wherein a playback device performs the quality control measurement.

10. The method of claim 8 wherein the quality control measurement is based at least in part on results of the global motion estimation.

11. The method of claim 8 wherein the quality control measurement is based at least in part on results of local motion estimation.

12. The method of claim 1 wherein, for the frame synthesis with motion analysis, the synthesizing includes on a pixel-by-pixel basis determining valid pixels in motion compensated versions of the source video frames.

13. The method of claim 1 wherein, for the frame synthesis with motion analysis, the synthesizing includes feathering.

14. The method of claim 1 wherein, for the frame synthesis with motion analysis, the synthesizing includes ghosting reduction.

15. The method of claim 1 wherein, for the frame synthesis with motion analysis, the synthesizing includes:

detecting an active window;

using the frame interpolation with motion analysis for pixels in the active window; and

using another technique for pixels outside the active window.

16. The method of claim 1 wherein the determining whether frame synthesis with motion analysis is on occurs before the receiving the pair of source video frames.

17. The method of claim 1 wherein the determining whether frame synthesis with motion analysis is on occurs after the receiving the pair of source video frames.

18. The method of claim 1 wherein the frame interpolation with motion analysis includes producing a global warping matrix.

19. The method of claim 18 wherein the global warping matrix is an affine matrix.

20. A computer-readable medium storing computer-executable instructions comprising code for:

receiving one or more bits in a bitstream, wherein the one or more bits signal whether frame synthesis is on or off;

if frame synthesis is on, determining whether frame synthesis with motion analysis is on, wherein scene change detection indicates whether the frame synthesis with motion analysis is on; and

if the frame synthesis with motion analysis is on, synthesizing one or more output frames between source frames using a normal mode, wherein the normal mode comprises frame interpolation with motion analysis including global motion estimation,

otherwise, synthesizing the one or more output frames using a fallback mode or skipping synthesis of the one or more output frames.

21. The computer-readable medium of claim 20 wherein the one or more bits are at frame level in the bitstream.

22. The computer-readable medium of claim 20 wherein the one or more bits are at sequence level in the bitstream.

23. The computer-readable medium of claim 20 wherein the one or more bits also signal a count of output frames between the source frames.

24. The computer-readable medium of claim 20 wherein the frame interpolation with motion analysis further includes local motion estimation to correct the global motion estimation.

25. A computer-readable medium storing computer-executable instructions comprising code for:

receiving a pair of source video frames in a video sequence;

determining whether frame synthesis with motion analysis is on, wherein scene change detection indicates whether the frame synthesis with motion analysis is on; and

synthesizing pixel values for an intermediate video frame between the source video frames using frame interpolation with motion analysis if the frame synthesis with motion analysis is on, wherein the frame interpolation with motion analysis includes global motion estimation,

otherwise, synthesizing pixel values for the intermediate video frame using a fallback mode or skipping synthesis of the intermediate video frame.

26. The computer-readable medium of claim 25 wherein the fallback mode comprises simple averaging of pixel values of the source video frames.

27. The computer-readable medium of claim 25 wherein the fallback mode comprises using only pixel values of a selected one of the source video frames.

28. The computer-readable medium of claim 25 wherein one or more bits in a bitstream signal whether the frame synthesis with motion analysis is on.

29. The computer-readable medium of claim 28 wherein the one or more bits are at frame level.

30. The computer-readable medium of claim 28 wherein the one or more bits are at sequence level.

31. The computer-readable medium of claim 28 wherein the one or more bits also signal a count of intermediate video frames between the source video frames.

32. The computer-readable medium of claim 25 wherein quality control measurement indicates whether the frame synthesis with motion analysis is on.

33. The computer-readable medium of claim 32 wherein a playback device performs the quality control measurement.

34. The computer-readable medium of claim 32 wherein the quality control measurement is based at least in part on results of the global motion estimation.

35. The computer-readable medium of claim 32 wherein the quality control measurement is based at least in part on results of local motion estimation.

36. The computer-readable medium of claim 25 wherein, for the frame synthesis with motion analysis, the synthesizing includes on a pixel-by-pixel basis determining valid pixels in motion compensated versions of the source video frames.

37. The computer-readable medium of claim 25 wherein, for the frame synthesis with motion analysis, the synthesizing includes feathering.

38. The computer-readable medium of claim 25 wherein, for the frame synthesis with motion analysis, the synthesizing includes ghosting reduction.

39. The computer-readable medium of claim 25 wherein, for the frame synthesis with motion analysis, the synthesizing includes:

detecting an active window;

using the frame interpolation with motion analysis for pixels in the active window; and

using another technique for pixels outside the active window.

40. The computer-readable medium of claim 25 wherein the frame interpolation with motion analysis includes producing a global warping matrix.

41. The computer-readable medium of claim 40 wherein the global warping matrix is an affine matrix.

42. The computer-readable medium of claim 25 wherein the frame interpolation with motion analysis further includes correcting the global motion estimation with local motion estimation.

43. A computer-readable medium storing computer-executable instructions comprising code for:

receiving a pair of source video frames in a video sequence;

determining whether frame synthesis with motion analysis is on, wherein scene change detection indicates whether the frame synthesis with motion analysis is on; and

synthesizing pixel values for an intermediate video frame between the source video frames using frame interpolation with motion analysis if the frame synthesis with motion analysis is on,

otherwise, synthesizing pixel values for the intermediate video frame using a fallback mode or skipping synthesis of the intermediate video frame.

44. The computer-readable medium of claim 43 wherein the fallback mode comprises simple averaging of pixel values of the source video frames.

45. The computer-readable medium of claim 43 wherein the fallback mode comprises using only pixel values of a selected one of the source video frames.

46. The computer-readable medium of claim 43 wherein one or more bits in a bitstream signal whether the frame synthesis with motion analysis is on.

47. The computer-readable medium of claim 43 wherein, for the frame synthesis with motion analysis, the synthesizing includes on a pixel-by-pixel basis determining valid pixels in motion compensated versions of the source video frames.

48. The computer-readable medium of claim 43 wherein, for the frame synthesis with motion analysis, the synthesizing includes feathering.

49. The computer-readable medium of claim 43 wherein, for the frame synthesis with motion analysis, the synthesizing includes ghosting reduction.

50. The computer-readable medium of claim 43 wherein, for the frame synthesis with motion analysis, the synthesizing includes:

detecting an active window;

using the frame interpolation with motion analysis for pixels in the active window; and

using another technique for pixels outside the active window.