System and method to process motion vectors of video data

• US 9,060,177 B2
• Filed: 07/13/2012
• Issued: 06/16/2015
• Est. Priority Date: 03/05/2009
• Status: Expired due to Fees

• Current Assignee: Qualcomm, Inc.
• Original Assignee: Qualcomm, Inc.
• Inventors: Dane, Gokce, Motta, Giovanni
• Primary Examiner(s): BAILEY, FREDERICK D
• Application Number:
  US13/548,570
• Publication Number:
  US 20120281762A1
• Time in Patent Office: 1,068 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: H04N 19/109 among a plurality of temporal predictive coding modes
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/154 Measured or subjectively estimated visual quality after decoding, e.g. measurement of distortion use of rate-distortion criteria H04N19/147
• CPC Class Codes: H04N 19/176 the region being a block, e.g. a macroblock
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/513 Processing of motion vectors
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness

Abstract

Systems and methods to process motion vectors of video data are disclosed. According to an embodiment, an intra-block in a video frame of the video data is identified. At least a first set and a second set of inter-blocks that neighbor the identified intra-block are identified, where each inter-block in a set of inter-blocks has a motion vector associated therewith. Data of the first and second set of inter-blocks is evaluated to determine an error associated with each of the first and second sets. A motion vector associated with motion vectors of the inter-blocks of the set with the least error is determined. The determined motion vector may be associated with the identified intra-block.

Claims

1. A method comprising:

assessing a spatial continuity relative to a first block of a video frame of video data;

assessing a temporal continuity relative to the first block of the video frame of video data, wherein assessing the temporal continuity comprises determining to utilize an estimated motion vector based on motion vectors of neighboring blocks, and wherein assessing the temporal continuity further comprises;

calculating a weighted difference between a processed motion vector of the first block of the video frame and a motion vector of a corresponding block of a previous frame of the video data,

comparing the weighted difference to a threshold, and

in response to the weighted difference exceeding the threshold, determining an estimated motion vector that is a median value of motion vectors of neighboring blocks in a three block by three block array that includes the first block corresponding to the processed motion vector; and

assigning a motion vector to the first block based on the spatial continuity and the temporal continuity.

2. The method of claim 1, wherein the motion vector is a processed motion vector or an estimated motion vector.

3. The method of claim 1, wherein the motion vector is provided in a received video bit stream corresponding to the first block or is based on motion compensated prediction.

4. The method of claim 1, wherein assessing the spatial continuity comprises comparing the motion vector of the first block to motion vectors of adjacent blocks.

5. The method of claim 1, wherein assessing the spatial continuity comprises:

comparing a first difference between motion vectors of two blocks that are adjacent to the first block to a first threshold;

in response to the first difference being less than the first threshold, determining whether a second difference between a motion vector of the first block and an average motion vector of motion vectors of the two blocks exceeds a second threshold; and

in response to the second difference exceeding the second threshold, determining a processed motion vector for the first block that is an average of the motion vectors of the two blocks.

6. The method of claim 1, wherein assessing the temporal continuity comprises determining a difference between the motion vector of the first block and a motion vector of a corresponding block in a previous frame.

7. The method of claim 1, further comprising:

detecting a block adjacent to a boundary of the video frame;

determining an average value of absolute values of differences between a motion vector of the block adjacent to the boundary and motion vectors of blocks that neighbor the block adjacent to the boundary;

comparing the average value to a threshold; and

setting the motion vector of the block adjacent to the boundary equal to a motion vector of a block that neighbors the block adjacent to the boundary in response to the average value exceeding the threshold.

8. The method of claim 1, wherein the first block is a center block.

9. The method of claim 1, further comprising:

comparing a first difference between motion vectors of two blocks that are adjacent to the first block to a first threshold;

in response to the first difference being less than the first threshold, determining whether a second difference between a motion vector of the first block and an average motion vector of motion vectors of the two blocks exceeds a second threshold;

in response to the second difference exceeding the second threshold, assigning a motion vector for the first block that is an average of the motion vectors of the two blocks;

calculating a weighted difference between a processed motion vector of the first block of the video frame and a motion vector of a corresponding block of a previous frame of the video data;

comparing the weighted difference to a third threshold; and

in response to the weighted difference exceeding the third threshold, replacing the motion vector for the first block with a median value of motion vectors of neighboring blocks in a three block by three block array that includes the first block corresponding to the processed motion vector.

10. The method of claim 9, further comprising:

detecting a block adjacent to a boundary of the video frame;

determining an average value of absolute values of differences between a motion vector of the block adjacent to the boundary and motion vectors of blocks that neighbor the block adjacent to the boundary;

comparing the average value to a fourth threshold; and

in response to the average value exceeding the fourth threshold, setting the motion vector of the block adjacent to the boundary equal to a motion vector of a block that neighbors the block adjacent to the boundary.

11. An apparatus comprising:

a spatial continuity module for assessing a spatial continuity relative to a first block of a video frame of video data;

a temporal continuity module for assessing a temporal continuity relative to the first block of the video frame of video data, wherein assessing the temporal continuity comprises determining to utilize an estimated motion vector based on motion vectors of neighboring blocks, wherein the temporal continuity assessment module calculates a weighted difference between a processed motion vector of the first block of the video frame and a motion vector of a corresponding block of a previous frame of the video data, compares the weighted difference to a threshold, and determines an estimated motion vector that is a median value of motion vectors of neighboring blocks in a three block by three block array that includes the first block corresponding to the processed motion vector when the weighted difference exceeds the threshold; and

a decision module for assigning a motion vector to the first block based on the spatial continuity and the temporal continuity.

12. The apparatus of claim 11, wherein the motion vector is a processed motion vector or an estimated motion vector.

13. The apparatus of claim 11, wherein the first motion vector is provided in a received video bit stream corresponding to the first block or is based on motion compensated prediction.

14. The apparatus of claim 11, wherein the spatial continuity assessment module compares a first difference between motion vectors of two blocks that are adjacent to the first block to a first threshold, determines whether a second difference between a motion vector of the first block and an average motion vector of motion vectors of the two blocks exceeds a second threshold when the first difference is less than the first threshold, and determines a processed motion vector for the first block that is an average of the motion vectors of the two blocks when the second difference exceeds the second threshold.

15. The apparatus of claim 11, further comprising a module for:

detecting a block adjacent to a boundary of the video frame;

determining an average value of absolute values of differences between a motion vector of the block adjacent to the boundary and motion vectors of blocks that neighbor the block adjacent to the boundary;

comparing the average value to a threshold; and

setting the motion vector of the block adjacent to the boundary equal to a motion vector of a block that neighbors the block adjacent to the boundary in response to the average value exceeding the threshold.

16. A non-transitory computer-readable medium storing computer code, the computer code comprising instructions that cause a computer to:

assess a spatial continuity relative to a first block of a video frame of video data;

assess a temporal continuity relative to the first block of the video frame of video data, wherein assessing the temporal continuity comprises determining to utilize an estimated motion vector based on motion vectors of neighboring blocks, wherein the instructions for assessing the temporal continuity further comprise instructions that cause the processor to;

calculate a weighted difference between a processed motion vector of the first block of the video frame and a motion vector of a corresponding block of a previous frame of the video data;

compare the weighted difference to a threshold; and

in response to the weighted difference exceeding the threshold, determine an estimated motion vector that is a median value of motion vectors of neighboring blocks in a three block by three block array that includes the first block corresponding to the processed motion vector; and

assign a motion vector to the first block based on the spatial continuity and the temporal continuity.

17. The computer-readable medium of claim 16, wherein the motion vector is a processed motion vector or an estimated motion vector.

18. The computer-readable medium of claim 16, wherein the motion vector is provided in a received video bit stream corresponding to the first block or is based on motion compensated prediction.

19. The computer-readable medium of claim 16, further comprising instructions that cause the computer to:

compare a first difference between motion vectors of two blocks that are adjacent to the first block to a first threshold;

in response to the first difference being less than the first threshold, determine whether a second difference between a motion vector of the first block and an average motion vector of motion vectors of the two blocks exceeds a second threshold; and

in response to the second difference exceeding the second threshold, determine a processed motion vector for the first block that is an average of the motion vectors of the two blocks.

20. The computer-readable medium of claim 16, further comprising instructions that cause the computer to:

detect a block adjacent to a boundary of the video frame;

determine an average value of absolute values of differences between a motion vector of the block adjacent to the boundary and motion vectors of blocks that neighbor the block adjacent to the boundary;

compare the average value to a threshold; and

set the motion vector of the block adjacent to the boundary equal to a motion vector of a block that neighbors the block adjacent to the boundary in response to the average value exceeding the threshold.

21. The computer-readable medium of claim 16, further comprising instructions that cause the computer to:

compare a first difference between motion vectors of two blocks that are adjacent to the first block to a first threshold;

in response to the first difference being less than the first threshold, determine whether a second difference between a motion vector of the first block and an average motion vector of motion vectors of the two blocks exceeds a second threshold;

in response to the second difference exceeding the second threshold, assign a motion vector for the first block that is an average of the motion vectors of the two blocks;

calculate a weighted difference between a processed motion vector of the first block of the video frame and a motion vector of a corresponding block of a previous frame of the video data;

compare the weighted difference to a third threshold; and

in response to the weighted difference exceeding the third threshold, replace the motion vector for the first block with a median value of motion vectors of neighboring blocks in a three block by three block array that includes the first block corresponding to the processed motion vector.

22. The computer-readable medium of claim 21, further comprising instructions that cause the computer to:

detect a block adjacent to a boundary of the video frame;

determine an average value of absolute values of differences between a motion vector of the block adjacent to the boundary and motion vectors of blocks that neighbor the block adjacent to the boundary;

compare the average value to a fourth threshold; and

in response to the average value exceeding the fourth threshold, set the motion vector of the block adjacent to the boundary equal to a motion vector of a block that neighbors the block adjacent to the boundary.

23. A device comprising:

means for assessing a spatial continuity relative to a first block of a video frame of video data;

means for assessing a temporal continuity relative to the first block of the video frame of video data, wherein assessing the temporal continuity comprises determining to utilize an estimated motion vector based on motion vectors of neighboring blocks, wherein the means for assessing the temporal continuity further comprise;

means for calculating a weighted difference between a processed motion vector of the first block of the video frame and a motion vector of a corresponding block of a previous frame of the video data;

means for comparing the weighted difference to a threshold; and

means for determining, in response to the weighted difference exceeding the threshold, an estimated motion vector that is a median value of motion vectors of neighboring blocks in a three block by three block array that includes the first block corresponding to the processed motion vector; and

means for assigning a motion vector to the first block based on the spatial continuity and the temporal continuity.

24. The device of claim 23, wherein the motion vector is a processed motion vector or an estimated motion vector.

25. The device of claim 23, wherein the motion vector is provided in a received video bit stream corresponding to the first block or is based on motion compensated prediction.

26. The device of claim 23, further comprising:

means for comparing a first difference between motion vectors of two blocks that are adjacent to the first block to a first threshold;

means for determining, in response to the first difference being less than the first threshold, whether a second difference between a motion vector of the first block and an average motion vector of motion vectors of the two blocks exceeds a second threshold; and

means for determining, in response to the second difference exceeding the second threshold, a processed motion vector for the first block that is an average of the motion vectors of the two blocks.

27. The device of claim 23, further comprising:

means for detecting a block adjacent to a boundary of the video frame;

means for determining an average value of absolute values of differences between a motion vector of the block adjacent to the boundary and motion vectors of blocks that neighbor the block adjacent to the boundary;

means for comparing the average value to a threshold; and

means for setting the motion vector of the block adjacent to the boundary equal to a motion vector of a block that neighbors the block adjacent to the boundary in response to the average value exceeding the threshold.

28. The device of claim 23, further comprising:

means for comparing a first difference between motion vectors of two blocks that are adjacent to the first block to a first threshold;

means for determining, in response to the first difference being less than the first threshold, whether a second difference between a motion vector of the first block and an average motion vector of motion vectors of the two blocks exceeds a second threshold;

means for assigning, in response to the second difference exceeding the second threshold, a motion vector for the first block that is an average of the motion vectors of the two blocks;

means for calculating a weighted difference between a processed motion vector of the first block of the video frame and a motion vector of a corresponding block of a previous frame of the video data;

means for comparing the weighted difference to a third threshold; and

means for replacing, in response to the weighted difference exceeding the third threshold, the motion vector for the first block with a median value of motion vectors of neighboring blocks in a three block by three block array that includes the first block corresponding to the processed motion vector.

29. The device of claim 28, further comprising:

means for detecting a block adjacent to a boundary of the video frame;

means for determining an average value of absolute values of differences between a motion vector of the block adjacent to the boundary and motion vectors of blocks that neighbor the block adjacent to the boundary;

means for comparing the average value to a fourth threshold; and

means for setting, in response to the average value exceeding the fourth threshold, the motion vector of the block adjacent to the boundary equal to a motion vector of a block that neighbors the block adjacent to the boundary.