Increasing motion smoothness using frame interpolation with motion analysis

• US 20040252230A1
• Filed: 06/13/2003
• Published: 12/16/2004
• Est. Priority Date: 06/13/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Winder, Simon Anthony John
• Granted Patent: US 7,408,986 B2
• Time in Patent Office: Days
• US Class Current: 348/402.1
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20032 Median filtering
• CPC Class Codes: G06T 2207/20182 Noise reduction or smoothing in the temporal domain; Spatio-temporal filtering
• CPC Class Codes: G06T 5/20 using local operators
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/70 Denoising; Smoothing
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: G06T 7/254 involving subtraction of images
• CPC Class Codes: H04N 21/4143 embedded in a Personal Computer [PC]
• CPC Class Codes: H04N 21/440281 by altering the temporal resolution, e.g. by frame skipping television signal recording using magnetic recording on tape for reproducing at a rate different from the recording rate H04N5/783
• CPC Class Codes: H04N 21/4622 Retrieving content or additional data from different sources, e.g. from a broadcast channel and the Internet web site content organization and management for information retrieval from the Internet G06F16/958; transmission by internet of broadcast information H04H60/82; stock exchange data over packet-switching network H04L12/1804; push services including data channel over packet-switching network H04L12/1859
• CPC Class Codes: H04N 21/4782 Web browsing , e.g. WebTV
• CPC Class Codes: H04N 21/6125 involving transmission via Internet transmission by internet of broadcast information H04H60/82
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51
• CPC Class Codes: H04N 7/17318 Direct or substantially direct transmission and handling of requests

Abstract

Techniques and tools for video frame interpolation and motion analysis are described. The techniques and tools may be implemented separately or in combination in software and/or hardware devices for various applications. For example, a media playback device uses frame interpolation and motion analysis in real time to increase the frame rate of streamed video for playback. The device uses feature/region selection in global motion estimation, local motion estimation to correct the global motion estimation at an intermediate timestamp for a synthesized frame, and vector switching in the local motion estimation.

Claims

1. In a computer system, a computer-implemented method of video frame synthesis, the method comprising:

estimating global motion between first video data with a first timestamp and second video data with a second timestamp; and

estimating local motion at an intermediate timestamp between the first and second timestamps to correct the global motion at the intermediate timestamp.

2. The method of claim 1 wherein the global motion comprises a forward warping or a reverse warping.

3. The method of claim 1 wherein an affine warping matrix parameterizes the global motion.

4. The method of claim 1 further comprising:

before the estimating local motion, warping the first video data and the second data to the intermediate timestamp using the global motion.

5. The method of claim 4 wherein the warping comprises warping the first video data forward to the intermediate timestamp and warping the second video data backward to the intermediate timestamp.

6. The method of claim 1 wherein a streaming video client performs the estimating global motion and the estimating local motion in real time.

7. The method of claim 1 wherein delta vectors parameterize the local motion.

8. The method of claim 1 further comprising using the global motion and the local motion in motion compensation for synthesis of a new video frame at the intermediate timestamp.

9. The method of claim 1 wherein the first video data are a first video frame and the second video data are a second video frame.

10. The method of claim 1 further comprising decomposing each of first and second video frames into plural levels, wherein the first video data are a level for the first video frame, and wherein the second video data are a corresponding level for the second video frame.

11. The method of claim 10 wherein the decomposing includes performing Laplacian pyramid decomposition.

12. The method of claim 10 wherein the estimating global motion includes iteratively refining the global motion through the plural levels.

13. The method of claim 10 wherein the estimating local motion includes iteratively refining the local motion through the plural levels.

14. The method of claim 1 wherein the estimating global motion includes using emphasis selection to improve accuracy and/or increase speed.

15. A computer-readable medium storing computer-executable instructions for causing the computer system to perform the method of claim 1.

16. A computer-readable medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method of processing source data samples in a sequence, the method comprising:

estimating global motion between source data samples in a sequence; and

estimating local motion to correct the global motion at a position of an intermediate data sample in the sequence, wherein the intermediate data sample is between the source data samples in the sequence, and wherein the estimating local motion includes estimating motion at the position of the intermediate data sample.

17. The computer-readable medium of claim 16 wherein the estimating local motion occurs after warping of the source data samples to the position of the intermediate data sample using the global motion

18. The computer-readable medium of claim 16 wherein the method further comprises de-emphasizing center region points in the estimating global motion.

19. The computer-readable medium of claim 16 wherein the method further comprises emphasizing points with little or no motion in the estimating global motion.

20. The computer-readable medium of claim 16 wherein the method further comprises emphasizing spatial feature points in the estimating global motion.

21. The computer-readable medium of claim 16 wherein the method further comprises checking for a scene change.

22. The computer-readable medium of claim 16 wherein the method further comprises checking for a window around an active area.

23. The computer-readable medium of claim 16 wherein the method further comprises synthesizing frames based at least in part on the global and local motion.

24. The computer-readable medium of claim 23 wherein the synthesizing includes selectively feathering to reduce artifacts in the synthesized frames.

25. The computer-readable medium of claim 23 wherein the synthesizing includes reducing ghosting artifacts in the synthesized frames.

26. The computer-readable medium of claim 16 wherein the source data samples are source video frames.

27. The computer-readable medium of claim 16 wherein the source data samples are levels of pyramid decompositions of source video frames.

28. The computer-readable medium of claim 16 wherein the estimating global motion includes iteratively refining the global motion at progressively finer levels of pyramid decompositions of source frames.

29. The computer-readable medium of claim 16 wherein the estimating global motion includes iteratively refining the global motion at progressively finer levels of pyramid decompositions of source frames.

30. The computer-readable medium of claim 16 wherein an affine warping matrix parameterizes the global motion, and wherein the estimating global motion includes:

initializing translation parameters for the affine warping matrix;

refining the translation parameters;

initializing affine parameters for the affine warping matrix based at least in part on the refined translation parameters; and

refining the affine parameters.

31. The computer-readable medium of claim 16 wherein the estimating local motion includes iteratively refining the local motion at progressively finer levels of pyramid decompositions of source frames.

32. The computer-readable medium of claim 16 wherein the estimating local motion includes computing plural candidate vector sets and switching between the plural candidate vector sets.

33. The computer-readable medium of claim 32 wherein the switching occurs on a point-by-point basis.

34. The computer-readable medium of claim 32 wherein the plural candidate vector sets include a first set computed after warping according to the global motion and a second set computed after warping according to the global motion and previously estimated local motion.

35. In a computer system, a computer-implemented method of increasing the playback frame rate of streamed video on a client computer, the method comprising:

receiving a stream by the client computer from a server computer, wherein the stream is for a video sequence having a streaming frame rate;

decompressing plural source video frames in real time;

performing motion analysis between the plural source video frames in real time;

synthesizing a video frame in real time using interpolation of the plural source video frames after warping; and

playing back the source and synthesized video frames in real time at the client computer at a playback frame rate greater than the streaming frame rate.

36. The method of claim 35 wherein motion analysis includes:

estimating global motion; and

estimating local motion at an intermediate time to correct the global motion at the intermediate time.

37. The method of claim 36 wherein the synthesizing includes:

warping the plural source video frames to the intermediate time using the global motion and the local motion; and

combining pixel data from the warped source video frames to generate the synthesized video frame.

38. The method of claim 37 wherein the combining includes non-linear filtering for at least some points in the synthesized video frame and linear filtering for other points in the synthesized video frame.

39. The method of claim 36 wherein the estimating local motion includes vector switching.

40. The method of claim 35 wherein the motion analysis includes selectively emphasizing regions and/or features.

41. The method of claim 40 wherein the selectively emphasizing includes de-emphasizing center region points and emphasizing feature points.

42. The method of claim 35 wherein the synthesizing includes selective non-linear filtering to reduce ghosting artifacts.

43. The method of claim 42 wherein the synthesizing further includes selective feathering to reduce seam artifacts.

44. The method of claim 35 further comprising skipping the synthesizing upon detection of a scene change.

45. The method of claim 35 further comprising skipping the synthesizing for border regions outside active areas.

46. The method of claim 35 further comprising skipping the synthesizing upon detection of poor quality for the synthesized video frame.

47. A computer-readable medium storing computer-executable instructions for causing the computer system to perform the method of claim 35.

48. A computer-readable medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method of processing video data in a streaming media playback device, the method comprising:

receiving at least part of a stream of data from a streaming media provider, wherein the stream is for a media sequence;

temporarily buffering the received stream;

reconstructing first and second source frames in the media sequence based at least in part on the buffered stream;

temporarily buffering the first and second reconstructed source frames;

estimating motion between the first and second reconstructed source frames;

synthesizing one or more new frames between the first and second reconstructed source frames using interpolation to increase motion smoothness for the media sequence;

temporarily buffering the one or more synthesized frames;

playing back the first reconstructed source frame, the one or more synthesized frames, and the second reconstructed source frame, wherein the receiving, the reconstructing, the estimating motion, the synthesizing, and the playing back occur in real time, thereby providing unbroken playback of the media sequence by the streaming media playback device as the receiving continues.

49. The computer-readable medium of claim 48 wherein the reconstructing including decompressing the first and second source frames.

50. The computer-readable medium of claim 48 wherein the estimating motion includes:

estimating global motion; and

estimating local motion at an intermediate time to correct the global motion at the intermediate time.

51. The computer-readable medium of claim 50 wherein the synthesizing includes:

warping the first and second source frames to the intermediate time using the global motion and the local motion; and

combining pixel data from the warped source frames to generate the one or more synthesized frames.

52. The computer-readable medium of claim 50 wherein the estimating motion includes selectively emphasizing regions and/or features.

53. The computer-readable medium of claim 52 wherein the selectively emphasizing includes de-emphasizing center region points.

54. The computer-readable medium of claim 52 wherein the selectively emphasizing includes emphasizing temporal feature points.

55. The computer-readable medium of claim 52 wherein the selectively emphasizing includes emphasizing spatial feature points.

56. The computer-readable medium of claim 50 wherein the estimating motion includes vector switching.

57. The computer-readable medium of claim 50 wherein the synthesizing includes non-linear filtering to reduce ghosting artifacts.

58. The computer-readable medium of claim 48 wherein the synthesizing includes feathering to reduce seam artifacts.

59. The computer-readable medium of claim 48 wherein the method further comprises skipping the synthesizing upon detection of a scene change.

60. The computer-readable medium of claim 48 wherein the method further comprises skipping the synthesizing for inactive areas.

61. The computer-readable medium of claim 48 wherein the method further comprises skipping the synthesizing upon detection of poor quality for the one or more synthesized frames.

62. In a computer system, a computer-implemented method of video frame synthesis, the method comprising:

selecting features and/or regions of source video data for emphasis;

estimating motion between the source video data, wherein the selecting increases accuracy and/or speed of the estimating; and

synthesizing a video frame.

63. The method of claim 62 wherein the selecting includes de-emphasizing center region points.

64. The method of claim 63 wherein the selecting comprises:

defining a center region boundary; and

determining whether points of the source video data are center region points with reference to the center region boundary.

65. The method of claim 62 wherein the selecting includes emphasizing temporal feature points.

66. The method of claim 65 wherein the selecting includes:

constructing a histogram; and

identifying the temporal features points based at least in part upon the histogram, wherein the temporal feature points indicate motion below a threshold value.

67. The method of claim 62 wherein the selecting includes emphasizing spatial feature points.

68. The method of claim 67 wherein the selecting includes:

constructing a histogram; and

identifying the spatial feature points based at least in part upon the histogram, wherein the spatial feature points indicate intensity change above a threshold value.

69. The method of claim 62 wherein the estimating motion includes estimating global motion, and wherein the selecting reduces weight given to points for the selected features and/or regions in the estimating global motion.

70. The method of claim 62 wherein the source video data include a source video frame.

71. The method of claim 62 wherein the source video data include a level of a pyramid decomposition of a source video frame.

72. A computer-readable medium storing computer-executable instructions for causing the computer system to perform the method of claim 62.

73. A computer-readable medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method comprising:

identifying one or more points in a region of an active area in source data with reference to a boundary around the region; and

de-emphasizing the one or more points in motion analysis for frame synthesis.

74. The computer-readable medium of claim 73 wherein the region is a center region in the source data.

75. The computer-readable medium of claim 73 wherein the source data include a source video frame, and wherein the one or more points are pixels.

76. The computer-readable medium of claim 73 wherein the source data include a level of a pyramid decomposition of a source video frame.

77. The computer-readable medium of claim 73 wherein the motion analysis includes global motion estimation, and wherein the de-emphasizing reduces weight given to the one or more points in the global motion estimation.

78. The computer-readable medium of claim 73 wherein the method further comprises:

distinguishing an inactive area from the active area in the source data; and

de-emphasizing points in the inactive area in the motion analysis.

79. A computer-readable medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method comprising:

constructing a histogram for source data;

identifying one or more feature points based at least in part upon the histogram;

emphasizing the one or more feature points in motion analysis for frame synthesis.

80. The computer-readable medium of claim 79 wherein the histogram records temporal derivatives, and wherein the feature points are temporal feature points.

81. The computer-readable medium of claim 79 wherein the histogram records spatial derivatives, and wherein the feature points are spatial feature points.

82. The computer-readable medium of claim 79 wherein the constructing includes setting a threshold value used to identify the one or more feature points.

83. The computer-readable medium of claim 82 wherein the source data include a level of a pyramid decomposition each of one or more frames, and wherein the constructing and the setting occur at a coarse resolution level for identification at a finer resolution level.

84. The computer-readable medium of claim 79 wherein the source data include one or more source video frames.

85. The computer-readable medium of claim 79 wherein the motion analysis includes global motion estimation, and wherein the emphasizing increases weight given to the one or more feature points in the global motion estimation.

86. In a computer system, a computer-implemented method of video frame synthesis comprising:

estimating motion between source video data, wherein the estimating includes vector switching; and

synthesizing a new video frame using the estimated motion along with interpolation.

87. The method of claim 86 wherein the interpolation includes linear interpolation for some points in the synthesized video frame and non-linear interpolation for other points in the synthesized video frame.

88. The method of claim 86 wherein the vector switching includes computing plural candidate sets of vectors and selecting between the plural candidate sets.

89. The method of claim 88 wherein the selecting occurs on a point-by-point basis.

90. The method of claim 88 wherein the computing includes applying a different motion model before deriving each of the plural candidate sets of vectors.

91. The method of claim 88 wherein the plural candidate sets include a first set indicating motion after previous warping by global motion, a second set indicating motion after previous warping by global and local motion, and a third set indicating motion without previous warping.

92. The method of claim 88 wherein the plural candidate sets include a first set indicating motion after previous warping and a second set indicating motion without previous warping.

93. The method of claim 86 wherein the vector switching includes selecting between differential motion vectors.

94. The method of claim 93 wherein the differential motion vectors indicate corrections to global motion.

95. The method of claim 86 wherein the source video data include levels of pyramid decompositions of video frames.

96. The method of claim 86 wherein the source video data include video frames.

97. A computer-readable medium storing computer-executable instructions for causing the computer system to perform the method of claim 86.

98. A computer-readable medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method comprising:

estimating global motion between first video data with a first timestamp and second video data with a second timestamp;

computing plural candidate sets of differential vectors to correct the global motion at an intermediate timestamp between the first timestamp and the second timestamp; and

selecting between the plural candidate sets of differential vectors.

99. The computer-readable medium of claim 98 wherein the plural candidate sets include a first set indicating motion after previous warping by the global motion and a second set indicating motion without previous warping.

100. The computer-readable medium of claim 98 wherein the plural candidate sets further include a third set indicating motion after previous warping by the global motion and previously computed local motion.

101. The computer-readable medium of claim 98 wherein the first video data is a first video frame and the second video data is a second video frame.

102. The computer-readable medium of claim 98 further comprising decomposing each of first and second video frames into plural levels, wherein the first video data are a level for the first video frame, and wherein the second video data are a corresponding level for the second video frame.

103. The computer-readable medium of claim 98 wherein the selecting occurs on a point-by-point basis.

104. A computer-readable medium storing computer-executable instructions for causing a computer system programmed thereby to perform a method of frame synthesis, the method comprising:

computing plural candidate sets of motion information, wherein the computing uses a different motion model before deriving each of the plural candidate sets;

selecting between the plural candidate sets of motion information; and

performing motion compensation on plural source frames based at least in part on the selected motion information; and

generating a new frame based at least in part upon interpolation between the motion compensated source frames.

105. The computer-readable medium of claim 104 wherein the selecting occurs on a point-by-point basis.

106. The computer-readable medium of claim 104 wherein the plural candidate sets include a first set derived after applying global motion to the plural source frames, a second set derived after applying global and local motion to the plural source frames, and a third set derived after applying no motion to the plural source frames.

107. The computer-readable medium of claim 104 wherein the motion information comprises differential motion vectors.

108. A system comprising:

a pyramid decomposition module for decomposing source frames into levels;

a global motion estimation module for estimating global motion between the levels, wherein the global motion estimation module checks quality of the global motion before frame synthesis;

a local motion estimation module for correcting the global motion at an intermediate timestamp, wherein the local motion estimation module checks quality of the corrected global motion before the frame synthesis;

a frame synthesis module for synthesizing a new frame at the intermediate timestamp using the corrected global motion.

109. The system of claim 108 further comprising:

a scene change detection module.

110. The system of claim 108 further comprising:

a window detection module.