Method for Determining Small-Object Region, and Method and Apparatus for Interpolating Frame Between Video Frames

• US 20160127678A1
• Filed: 12/04/2015
• Published: 05/05/2016
• Est. Priority Date: 06/05/2013
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Zhang, Lei, Yao, Zhigang, Xie, Qingpeng
• Granted Patent: US 9,781,382 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/567 Motion estimation based on rate distortion criteria
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 7/0117 involving conversion of the spatial resolution of the incoming video signal for graphics images G09G2340/0407
• CPC Class Codes: H04N 7/0125 one of the standards being a high definition standard
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

A method and an apparatus for determining a small-object region in a video frame. The method includes dividing a current video frame into at least two regions, and determining a global motion vector corresponding to each region; determining an interframe motion vector of each group of adjacent frames in multiple video frames that include the current video frame and a reference frame of the current video frame; determining a candidate small-object region in the current video frame according to the interframe motion vector of the each group of adjacent frames and the determined global motion vector corresponding to each region; and performing filtering on the candidate small-object region in the current video frame, and determining a region obtained after the filtering as a small-object region in the current video frame.

Claims

1. A method for determining a small-object region in a video frame, comprising:

dividing a current video frame into at least two regions;

determining a global motion vector corresponding to each region;

determining an interframe motion vector of each group of two adjacent frames in multiple video frames that comprise the current video frame and a reference frame of the current video frame;

determining a candidate small-object region in the current video frame according to the determined interframe motion vector of the each group of two adjacent frames and the determined global motion vector corresponding to each region;

performing filtering on the candidate small-object region in the current video frame; and

determining a region obtained after the filtering as a small-object region in the current video frame,

wherein the reference frame of the current video frame comprises one or more of preceding continuous video frames of the current video frame and following continuous video frames of the current video frame.

2. The method according to claim 1, wherein before determining the interframe motion vector of each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame, the method further comprises, for a to-be-processed picture block comprised in each video frame in the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame, executing the following:

selecting at least one video frame from preceding N video frames of the current video frame, wherein N is a positive integer;

determining, according to a small-object region determined in the preceding N video frames of the current video frame, whether a reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is a picture block comprised in the small-object region;

determining that the to-be-processed picture block is a first-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is the picture block comprised in the small-object region; and

determining that the to-be-processed picture block is a second-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is not the picture block comprised in the small-object region,

wherein determining the interframe motion vector of each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame comprises;

separately determining an interframe motion vector of each first-type to-be-processed picture block and an interframe motion vector of each second-type to-be-processed picture block; and

using the determined interframe motion vector of each first-type to-be-processed picture block comprised in each video frame in the each group of adjacent frames and the determined interframe motion vector of each second-type to-be-processed picture block comprised in each video frame in the each group of adjacent frames as the interframe motion vector of the each group of adjacent frames.

3. The method according to claim 2, wherein determining the interframe motion vector of each first-type to-be-processed picture block comprises:

determining a value of a dissimilarity between each candidate motion vector corresponding to the first-type to-be-processed picture block and a background motion vector of a video frame in which the first-type to-be-processed picture block is located;

assigning a corresponding weight to each candidate motion vector according to the determined value of the dissimilarity corresponding to each candidate motion vector by using a rule that a smaller weight is assigned to a larger value of the dissimilarity; and

determining the interframe motion vector of the first-type to-be-processed picture block according to the weight assigned to each candidate motion vector and a Sum of Absolute Difference (SAD) value of pixels of a picture block pointed to by each candidate motion vector and pixels of the first-type to-be-processed picture block.

4. The method according to claim 2, wherein selecting the at least one video frame from the preceding N video frames of the current video frame comprises selecting at least one preceding continuous video frame of the current video frame.

5. The method according to claim 3, wherein determining the interframe motion vector of the first-type to-be-processed picture block comprises, for each candidate motion vector:

determining a product of the weight assigned to the candidate motion vector and the SAD value of the pixels of the picture block pointed to by the candidate motion vector and the pixels of the first-type to-be-processed picture block; and

using a candidate motion vector with a smallest product as the interframe motion vector of the first-type to-be-processed picture block.

6. The method according to claim 1, wherein determining the candidate small-object region in the current video frame according to the determined interframe motion vector of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame and the determined global motion vector corresponding to each region comprises:

determining, in each reference frame of the current video frame and according to the interframe motion vector of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame, a matching block corresponding to each picture block comprised in the current video frame;

determining, in each reference frame, a nearby block near the matching block;

determining an interframe motion vector of each nearby block determined in each reference frame;

determining, for each picture block comprised in the current video frame, a value of a similarity between the interframe motion vector of each nearby block determined for the picture block and an interframe motion vector of the picture block;

determining, for each picture block comprised in the current video frame, a value of a dissimilarity between the interframe motion vector and a global motion vector that are of each nearby block; and

determining, according to the determined value of the similarity and the determined value of the dissimilarity, a picture block comprised in the candidate small-object region in the current video frame, wherein each picture block comprised in the candidate small-object region meets the following;

in multiple nearby blocks that are determined for the picture block and that are comprised in each reference frame corresponding to the current video frame, there are a first set quantity of nearby blocks whose values of similarities are all greater than or equal to a first threshold and there are a second set quantity of nearby blocks whose values of dissimilarities are all greater than or equal to a second threshold.

7. The method according to claim 1, wherein before performing filtering on the candidate small-object region in the current video frame, the method further comprises marking a specific marker on each picture block comprised in the candidate small-object region, and wherein performing filtering on the candidate small-object region in the current video frame, and determining the region obtained after the filtering as the small-object region in the current video frame comprises:

determining, for each picture block comprised in the candidate small-object region in the current video frame, a value of a first quantity of picture blocks that are marked with the specific marker and that are in a first set range in a horizontal direction of the picture block, and a value of a second quantity of picture blocks that are marked with the specific marker and that are in a second set range in a vertical direction of the picture block;

removing the specific marker of the picture block when the determined value of the first quantity or the determined value of the second quantity is greater than a third threshold;

determining a value of a third quantity of picture blocks that are marked with the specific marker and that are in a third set range around the picture block;

removing the specific marker of the picture block when the determined value of the third quantity is less than a fourth threshold; and

determining the picture block that is marked with the specific marker and that is in the current video frame as the small-object region in the current video frame,

wherein the third set range is smaller than the first set range and the second set range, and

wherein the fourth threshold is less than the third threshold.

8. A method for interpolating a frame between two adjacent video frames, comprising:

determining a small-object region in a former video frame and a small-object region in a latter video frame, wherein the former video frame and the latter video frame are two consecutively adjacent video frames;

performing smooth filtering on an interframe motion vector corresponding to a region except the small-object region in the former video frame and the small-object region in the latter video frame; and

interpolating a frame between the two consecutively adjacent video frames according to an interframe motion vector corresponding to the small-object region in the former video frame, an interframe motion vector corresponding to the small-object region in the latter video frame, and an interframe motion vector obtained after the smooth filtering.

9. A method for interpolating a frame between two adjacent video frames, comprising:

classifying, according to a to-be-processed picture block comprised in a small-object region in preceding N video frames of two consecutively adjacent video frames, to-be-processed picture blocks comprised in each video frame of the two consecutively adjacent video frames into a first-type to-be-processed picture block and a second-type to-be-processed picture block;

separately determining an interframe motion vector of each first-type to-be-processed picture block and an interframe motion vector of each second-type to-be-processed picture block;

using the determined interframe motion vector of each first-type to-be-processed picture block comprised in each video frame of the two consecutively adjacent video frames and the determined interframe motion vector of each second-type to-be-processed picture block comprised in each video frame of the two consecutively adjacent video frames as an interframe motion vector between the two consecutively adjacent video frames; and

interpolating a frame between the two consecutively adjacent video frames according to the obtained interframe motion vector between the two consecutively adjacent video frames, wherein N is a positive integer.

10. The method according to claim 9, wherein classifying, according to the to-be-processed picture block comprised in the small-object region in the preceding N video frames of the two consecutively adjacent video frames, the to-be-processed picture blocks comprised in each video frame of the two consecutively adjacent video frames into the first-type to-be-processed picture block and the second-type to-be-processed picture block comprises, for the to-be-processed picture block comprised in each video frame of the two consecutively adjacent video frames, executing the following:

selecting at least one video frame from the preceding N video frames of the two consecutively adjacent video frames;

determining, according to the small-object region in the preceding N video frames of the two consecutively adjacent video frames, whether a reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is a picture block comprised in the small-object region; and

determining that the to-be-processed picture block is a first-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is the picture block comprised in the small-object region; and

determining that the to-be-processed picture block is a second-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is not the picture block comprised in the small-object region.

11. The method according to claim 10, wherein determining the interframe motion vector of each first-type to-be-processed picture block comprises:

determining a value of a dissimilarity between each candidate motion vector corresponding to the first-type to-be-processed picture block and a background motion vector of a video frame in which the first-type to-be-processed picture block is located;

assigning a corresponding weight to each candidate motion vector according to the determined value of the dissimilarity corresponding to each candidate motion vector by using a rule that a smaller weight is assigned to a larger value of the dissimilarity; and

determining the interframe motion vector of the first-type to-be-processed picture block according to the weight assigned to each candidate motion vector and a Sum of Absolute Difference (SAD) value of pixels of a picture block pointed to by each candidate motion vector and pixels of the first-type to-be-processed picture block.

12. The method according to claim 10, wherein selecting the at least one video frame from the preceding N video frames comprises selecting at least one preceding continuous video frame of the two consecutively adjacent video frames.

13. The method according to claim 11, wherein determining the interframe motion vector of the first-type to-be-processed picture block comprises, for each candidate motion vector:

determining a product of the weight assigned to the candidate motion vector and the SAD value of the pixels of the picture block pointed to by the candidate motion vector and the pixels of the first-type to-be-processed picture block; and

using a candidate motion vector with a smallest product as the interframe motion vector of the first-type to-be-processed picture block.

14. An apparatus for determining a small-object region in a video frame, comprising:

a computer processor configured to;

divide a current video frame into at least two regions;

determine a global motion vector corresponding to each region;

transmit the determined global motion vector;

determine an interframe motion vector of each group of adjacent frames in multiple video frames that comprise the current video frame and a reference frame of the current video frame;

transmit the determined interframe motion vector;

determine information about a candidate small-object region in the current video frame according to the interframe motion vector that is of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame and that is from the global motion vector that is corresponding to each region;

transmit the information about the determined candidate small-object region in the current video frame;

determine the candidate small-object region in the current video frame according to the information that is about the candidate small-object region in the current video frame;

perform filtering on the candidate small-object region in the current video frame; and

determine a region obtained after the filtering as a small-object region in the current video frame,

wherein the reference frame of the current video frame comprises one or more of preceding continuous video frames of the current video frame and following continuous video frames of the current video frame.

15. The apparatus according to claim 14, wherein the computer processor is further configured to execute, before the interframe motion vector of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame is determined, the following for a to-be-processed picture block comprised in each video frame in the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame:

selecting at least one video frame from preceding N video frames of the current video frame;

determining, according to a small-object region determined in the preceding N video frames of the current video frame, whether a reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is a picture block comprised in the small-object region;

determining that the to-be-processed picture block is a first-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is the picture block comprised in the small-object region;

determining that the to-be-processed picture block is a second-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is not the picture block comprised in the small-object region;

separately determine an interframe motion vector of each first-type to-be-processed picture block and an interframe motion vector of each second-type to-be-processed picture block; and

use the determined interframe motion vector of each first-type to-be-processed picture block comprised in each video frame in the each group of adjacent frames and the determined interframe motion vector of each second-type to-be-processed picture block comprised in each video frame in the each group of adjacent frames as the interframe motion vector of the each group of adjacent frames.

16. The apparatus according to claim 15, wherein the computer processor is further configured to:

determine a value of a dissimilarity between each candidate motion vector corresponding to the first-type to-be-processed picture block and a background motion vector of a video frame in which the first-type to-be-processed picture block is located;

assign a corresponding weight to each candidate motion vector according to the determined value of the dissimilarity corresponding to each candidate motion vector by using a rule that a smaller weight is assigned to a larger value of the dissimilarity; and

determine the interframe motion vector of the first-type to-be-processed picture block according to the weight assigned to each candidate motion vector and a Sum of Absolute Difference (SAD) value of pixels of a picture block pointed to by each candidate motion vector and pixels of the first-type to-be-processed picture block.

17. The apparatus according to claim 14, wherein the computer processor is further configured to:

determine, in each reference frame of the current video frame and according to the interframe motion vector of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame, a matching block corresponding to each picture block comprised in the current video frame;

determine, in each reference frame, a nearby block near the matching block, and determine an interframe motion vector of each nearby block determined in each reference frame;

determine, for each picture block comprised in the current video frame, a value of a similarity between the interframe motion vector of each nearby block determined for the picture block and an interframe motion vector of the picture block;

determine, for each picture block comprised in the current video frame, a value of a dissimilarity between the interframe motion vector and a global motion vector that are of each nearby block; and

determine, according to the determined value of the similarity and the determined value of the dissimilarity, a picture block comprised in the candidate small-object region in the current video frame, wherein each picture block comprised in the candidate small-object region meets the following;

in multiple nearby blocks that are determined for the picture block and that are comprised in each reference frame corresponding to the current video frame, there are a first set quantity of nearby blocks whose values of similarities are all greater than or equal to a first threshold and there are a second set quantity of nearby blocks whose values of dissimilarities are all greater than or equal to a second threshold.

18. The apparatus according to claim 14, wherein the computer processor is further configured to:

mark, before filtering is performed on the candidate small-object region in the current video frame, a specific marker on each picture block comprised in the candidate small-object region;

determine, for each picture block comprised in the candidate small-object region in the current video frame, a value of a first quantity of picture blocks that are marked with the specific marker and that are in a first set range in a horizontal direction of the picture block, and a value of a second quantity of picture blocks that are marked with the specific marker and that are in a second set range in a vertical direction of the picture block;

remove the specific marker of the picture block when the determined value of the first quantity or the determined value of the second quantity is greater than a third threshold;

determine a value of a third quantity of picture blocks that are marked with the specific marker and that are in a third set range around the picture block;

remove the specific marker of the picture block when the determined value of the third quantity is less than a fourth threshold; and

determine the picture block that is marked with the specific marker and that is in the current video frame as the small-object region in the current video frame, wherein the third set range is smaller than the first set range and the second set range, and the fourth threshold is less than the third threshold.

19. An apparatus for interpolating a frame between two adjacent video frames, comprising:

a computer processor configured to;

determine a small-object region in a former video frame and a small-object region in a latter video frame, wherein the former video frame and the latter video frame are two consecutively adjacent video frames;

transmit information about the determined small-object region in the former video frame, and information about the determined small-object region in the latter video frame;

determine, according to the information that is about the small-object region in the former video frame and the information that is about the small-object region in the latter video frame, the small-object region in the former video frame and the small-object region in the latter video frame;

perform smooth filtering on an interframe motion vector corresponding to a region except the small-object region in the former video frame and the small-object region in the latter video frame;

transmit an interframe motion vector obtained after the smooth filtering; and

interpolate a frame between the two consecutively adjacent video frames according to the interframe motion vector that is obtained after the smooth filtering, an interframe motion vector corresponding to the small-object region in the former video frame, and an interframe motion vector corresponding to the small-object region in the latter video frame.

20. An apparatus for interpolating a frame between two adjacent video frames, comprising:

a computer processor configured to;

classify, according to a to-be-processed picture block comprised in a small-object region in preceding N video frames of two consecutively adjacent video frames, to-be-processed picture blocks comprised in each video frame of the two consecutively adjacent video frames into a first-type to-be-processed picture block and a second-type to-be-processed picture block;

transmit classification information corresponding to the first-type to-be-processed picture block and the second-type to-be-processed picture block;

determine, according to the classification information, the first-type to-be-processed picture block and the second-type to-be-processed picture block that are in the to-be-processed picture blocks comprised in each video frame in the two consecutively adjacent video frames;

separately determine an interframe motion vector of each first-type to-be-processed picture block and an interframe motion vector of each second-type to-be-processed picture block;

use the determined interframe motion vector of each first-type to-be-processed picture block comprised in each video frame of the two consecutively adjacent video frames and the determined interframe motion vector of each second-type to-be-processed picture block comprised in each video frame of the two consecutively adjacent video frames as an interframe motion vector between the two consecutively adjacent video frames;

transmit the interframe motion vector between the two consecutively adjacent video frames; and

interpolate a frame between the two consecutively adjacent video frames according to the interframe motion vector that is of the two consecutively adjacent video frames, wherein N is a positive integer.

21. The apparatus according to claim 20, wherein the computer processor is configured to execute the following for the to-be-processed picture block comprised in each video frame of the two consecutively adjacent video frames:

selecting at least one video frame from the preceding N video frames of the two consecutively adjacent video frames;

determining, according to the small-object region in the preceding N video frames of the two consecutively adjacent video frames, whether a reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is a picture block comprised in the small-object region;

determining that the to-be-processed picture block is a first-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is the picture block comprised in the small-object region; and

determining that the to-be-processed picture block is a second-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is not the picture block comprised in the small-object region.

22. The apparatus according to claim 21, wherein the computer processor is configured to:

determine a value of a dissimilarity between each candidate motion vector corresponding to the first-type to-be-processed picture block and a background motion vector of a video frame in which the first-type to-be-processed picture block is located;

assign a corresponding weight to each candidate motion vector according to the determined value of the dissimilarity corresponding to each candidate motion vector by using a rule that a smaller weight is assigned to a larger value of the dissimilarity; and

determine the interframe motion vector of the first-type to-be-processed picture block according to the weight assigned to each candidate motion vector and a Sum of Absolute Difference (SAD) value of pixels of a picture block pointed to by each candidate motion vector and pixels of the first-type to-be-processed picture block.