Method and apparatus for increasing video frame rate
First Claim
1. A method of increasing a video frame rate of a sequence of video frames comprising:
- interpolating at least one frame between a first frame and a second frame of the sequence by applying a spatio-temporal median filter to pixel signal values of the first and second frames, wherein output signals of the spatio-temporal median filter comprise a median signal value of the pixels of spatially corresponding regions in the first and second frames and a pixel having the median signal value.
7 Assignments
0 Petitions
Accused Products
Abstract
Synthesizing of video frames that have been dropped by a video encoder is achieved by interpolating between decoded frames at a decoder. The method consists of successive refinement stages that increase in computational complexity. Starting with a spatio-temporal median filtering approach, each stage uses information that improves the quality of the interpolated frames, such as bit stream motion information, decoder-based motion estimation and motion-based state segmentation of regions. By using more computational resources, each of these stages results in an improved quality of interpolated video. The motion compensation techniques are based on block-based motion estimation of the kind used by block-transform based video encoders. More accurate motion estimates are obtained by using a combination of forward and backward block motion estimation. The method is further extended by incorporating global/local motion estimation based on the segmentation information, and employing image warping techniques to compensate for motion resulting from deformations.
-
Citations
34 Claims
-
1. A method of increasing a video frame rate of a sequence of video frames comprising:
-
interpolating at least one frame between a first frame and a second frame of the sequence by applying a spatio-temporal median filter to pixel signal values of the first and second frames, wherein output signals of the spatio-temporal median filter comprise a median signal value of the pixels of spatially corresponding regions in the first and second frames and a pixel having the median signal value. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 27, 28)
determining a first set of regions in the first and second frames that have changed due to motion of objects represented in the first and second frames and a second set of regions in the first and second frames that have not changed; and
applying the spatio-temporal median filter to the first set of regions.
-
-
9. The method of claim 8, wherein determining the first set of regions comprises applying a threshold operation to pixel-wise differences between corresponding pixels of the first and second frames.
-
27. The method of claim 1, further comprising selecting pixels of the first and second frames for generation of the interpolated frame based at least in part on segmentation states of regions of the frames, the regions including the pixels.
-
28. The method of claim 27, wherein the segmentation states comprise at least stationary, moving, covered, and uncovered states.
-
10. A method of increasing a video frame rate of a sequence of video frames comprising:
-
determining motion of objects in blocks of the sequence of video frames by applying motion compensation using motion vectors received as part of the sequence;
creating at least one block of an interpolated frame between a first frame and a second frame by applying a spatio-temporal median filter to pixels of best matching blocks of the first and second frames as determined by the motion compensation; and
averaging pixels at the same spatial location in the first and second frames for regions of pixels in the interpolated frame through which no motion vectors pass. - View Dependent Claims (11, 29, 30)
-
-
12. In a decoder, a method of increasing a video frame rate of a sequence of video frames comprising:
-
determining motion of objects in the sequence of video frames by applying motion compensation with at least one of forward motion estimation and backward motion estimation to the sequence of video frames received by the decoder;
selecting motion vectors from blocks of a first frame to blocks of a second frame based on the motion compensation;
creating at least one block of at least one interpolated frame between the first and second frames by applying a spatio-temporal median filter to best matching blocks of the first and second frames as determined by the selected motion vectors;
wherein pixels of the first and second frames are selected for creation of the at least one interpolated block based at least in part on segmentation states of pixels of the blocks, the segmentation states resulting from generation of a pixel state classification map by applying binary threshold operations over the frames. - View Dependent Claims (13, 14, 15, 16, 17, 18, 31)
determining a first set of regions in the first and second frames that have changed due to motion of objects represented in the first and second frames and a second set of regions in the first and second frames that have not changed; and
applying the spatio-temporal median filter to the first set of regions.
-
-
18. The method of claim 17, wherein determining the first set of regions comprises applying a threshold operation to pixel-wise differences between corresponding pixels of the first and second frames.
-
31. The method of claim 12, wherein the segmentation states comprise at least stationary, moving, covered, and uncovered states.
-
19. An apparatus for increasing a video frame rate of a sequence of video frames comprising:
-
a frame rate upsampling device to interpolate at least one frame between a first frame and a second frame of the sequence by applying a spatio-temporal median filter to pixels of the first and second frames, wherein output signals of the spatio-temporal median filter comprise a median signal value of the pixels of spatially corresponding regions in the first and second frames and a pixel having the median signal value. - View Dependent Claims (20, 21, 22, 32, 33)
means for determining a first set of regions in the first and second frames that have changed due to motion of objects represented in the first and second frames and a second set of regions in the first and second frames that have not changed; and
means for applying the spatio-temporal median filter to the first set of regions.
-
-
22. The apparatus of claim 19, wherein the determining means comprises means for applying a threshold operation to pixel-wise differences between corresponding pixels of the first and second frames to determine the first set of regions.
-
32. The apparatus of claim 19, wherein the frame rate upsampling device selects pixels of the first and second frames for generation of the interpolated frame based at least in part on segmentation states of regions of the frames, the regions including the pixels.
-
33. The apparatus of claim 32, wherein the segmentation states comprise at least stationary, moving, covered, and uncovered states.
-
23. An apparatus for increasing a video frame rate of a sequence of video frames comprising:
-
a motion compensator to determine motion of objects in blocks of the sequence of video frames by applying motion compensation operations using motion vectors received as part of the sequence;
a spatio-temporal median filter to create blocks of at least one interpolated frame between a first frame and a second frame by applying a spatio-temporal median filter operation to pixels of best matching blocks of the first and second frames as determined by the motion compensator; and
a state segmentation unit coupled to the spatio-temporal median filter to select pixels of the first and second frames for generation of the interpolated blocks based at least in part on segmentation states of the blocks, the segmentation states resulting from generation of a pixel state classification map by applying binary threshold operations over the frames. - View Dependent Claims (34)
-
-
24. In a decoder, an apparatus for increasing a video frame rate of a sequence of video frames comprising:
-
a motion compensator to determine motion of objects in the sequence of video frames by applying motion compensation with at least one of forward motion estimation and backward motion estimation to the sequence of video frames received by the decoder;
a motion vector selector to select motion vectors from blocks of a first frame to blocks of a second frame based on the motion compensation;
a spatio-temporal median filter to create blocks of at least one interpolated frame between the first and second frame by applying a spatio-temporal median filter operation to best matching blocks of the first and second frames as determined by the selected motion vectors;
a state segmentation unit coupled to the spatio-temporal median filter to select pixels of the first and second frames for generation of the interpolated blocks based at least in part on segmentation states of the blocks, the segmentation states resulting from generation of a pixel state classification map by applying binary threshold operations over the frames.
-
-
25. An apparatus for increasing a video frame rate of a sequence of video frames comprising:
-
means for determining motion of objects in blocks of the sequence of video frames by applying motion compensation using motion vectors received as part of the sequence;
means for creating blocks of at least one interpolated frame between a first frame and a second frame by applying a spatio-temporal median filter to pixels of best matching blocks of the first and second frames as determined by the motion compensation; and
means for selecting pixels of the first and second frames for generation of the interpolated blocks based at least in part on segmentation states of the blocks, the segmentation states resulting from generation of a pixel state classification map by applying binary threshold operations over the frames.
-
-
26. In a decoder, an apparatus for increasing a video frame rate of a sequence of video frames comprising:
-
means for determining motion of objects in the sequence of video frames by applying motion compensation with at least one of forward motion estimation and backward motion estimation to the sequence of video frames received by the decoder;
means for selecting motion vectors from blocks of a first frame to blocks of a second frame based on the motion compensation;
means for creating blocks of at least one interpolated frame between the first and second frame by applying a spatio-temporal median filter operation to best matching blocks of the first and second frames as determined by the selected motion vectors; and
means for selecting pixels of the first and second frames for generation of the interpolated blocks based at least in part on segmentation states of the blocks, the segmentation states resulting from generation of a pixel state classification map by applying binary threshold operations over the frames.
-
Specification