Method for determining small-object region, and method and apparatus for interpolating frame between video frames

• US 9,781,382 B2
• Filed: 12/04/2015
• Issued: 10/03/2017
• Est. Priority Date: 06/05/2013
• Status: Active Grant

• Current Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Original Assignee: Huawei Technologies Co., Ltd. (Huawei Investment & Holding Co., Ltd.)
• Inventors: Zhang, Lei, Yao, Zhigang, Xie, Qingpeng
• Primary Examiner(s): Lee, Michael
• Assistant Examiner(s): Khalid, Omer
• Application Number:
  US14/959,341
• Publication Number:
  US 20160127678A1
• Time in Patent Office: 669 Days
• Field of Search: 348441
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/567 Motion estimation based on rate distortion criteria
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 5/145 Movement estimation for video coding H04N19/51
• CPC Class Codes: H04N 7/0117 involving conversion of the spatial resolution of the incoming video signal for graphics images G09G2340/0407
• CPC Class Codes: H04N 7/0125 one of the standards being a high definition standard
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

A method and an apparatus for determining a small-object region in a video frame. The method includes dividing a current video frame into at least two regions, and determining a global motion vector corresponding to each region; determining an interframe motion vector of each group of adjacent frames in multiple video frames that include the current video frame and a reference frame of the current video frame; determining a candidate small-object region in the current video frame according to the interframe motion vector of the each group of adjacent frames and the determined global motion vector corresponding to each region; and performing filtering on the candidate small-object region in the current video frame, and determining a region obtained after the filtering as a small-object region in the current video frame.

Claims

1. A method for determining a small-object region in a video frame, comprising:

dividing a current video frame into at least two regions;

determining a motion vector corresponding to each region;

determining an interframe motion vector of each group of two adjacent frames that comprise the current video frame and reference frames of the current video frame;

determining a candidate small-object region in the current video frame according to the determined interframe motion vector of the each group of two adjacent frames and the determined motion vector corresponding to each region; and

performing filtering on the candidate small-object region in the current video frame to determine a region obtained after the filtering as a small-object region in the current video frame,

wherein the reference frames of the current video frame comprises one or more of preceding continuous video frames of the current video frame and following continuous video frames of the current video frame,

wherein before determining the interframe motion vector of each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame, the method further comprises, for a to-be-processed picture block comprised in each video frame in the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame, executing the following;

selecting at least one video frame from preceding N video frames of the current video frame, wherein N is a positive integer;

determining, according to a small-object region determined in the preceding N video frames of the current video frame, whether a reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is a picture block comprised in the small-object region;

determining that the to-be-processed picture block is a first-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is the picture block comprised in the small-object region; and

determining that the to-be-processed picture block is a second-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is not the picture block comprised in the small-object region,

wherein determining the interframe motion vector of each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame comprises;

determining an interframe motion vector of each first-type to-be-processed picture block and an interframe motion vector of each second-type to-be-processed picture block; and

using the determined interframe motion vector of each first-type to-be-processed picture block comprised in each video frame in the each group of adjacent frames and the determined interframe motion vector of each second-type to-be-processed picture block comprised in each video frame in the each group of adjacent frames as the interframe motion vector of the each group of adjacent frames, and

wherein determining the interframe motion vector of each first-type to-be-processed picture block comprises;

determining a value of a dissimilarity between each candidate motion vector corresponding to the first-type to-be-processed picture block and a motion vector of a video frame in which the first-type to-be-processed picture block is located;

assigning a corresponding weight to each candidate motion vector according to the determined value of the dissimilarity corresponding to each candidate motion vector by using a rule that a smaller weight is assigned to a larger value of the dissimilarity; and

determining the interframe motion vector of the first-type to-be-processed picture block according to the weight assigned to each candidate motion vector and a Sum of Absolute Difference (SAD) value of pixels of a picture block pointed to by each candidate motion vector and pixels of the first-type to-be-processed picture block.

2. The method according to claim 1, wherein selecting the at least one video frame from the preceding N video frames of the current video frame comprises selecting at least one preceding continuous video frame of the current video frame.

3. The method according to claim 1, wherein determining the interframe motion vector of the first-type to-be-processed picture block comprises, for each candidate motion vector:

determining a product of the weight assigned to the candidate motion vector and the SAD value of the pixels of the picture block pointed to by the candidate motion vector and the pixels of the first-type to-be-processed picture block; and

using a candidate motion vector with a smallest product as the interframe motion vector of the first-type to-be-processed picture block.

4. A method for determining a small-object region in a video frame, comprising:

dividing a current video frame into at least two regions;

determining a motion vector corresponding to each region;

determining an interframe motion vector of each group of two adjacent frames that comprise the current video frame and reference frames of the current video frame;

determining a candidate small-object region in the current video frame according to the determined interframe motion vector of the each group of two adjacent frames and the determined motion vector corresponding to each region; and

performing filtering on the candidate small-object region in the current video frame to determine a region obtained after the filtering as a small-object region in the current video frame,

wherein the reference frames of the current video frame comprises one or more of preceding continuous video frames of the current video frame and following continuous video frames of the current video frame,

wherein determining the candidate small-object region in the current video frame according to the determined interframe motion vector of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame and the determined motion vector corresponding to each region comprises;

determining, in each reference frame of the current video frame and according to the interframe motion vector of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame, a matching block corresponding to each picture block comprised in the current video frame;

determining, in each reference frame, a nearby block near the matching block;

determining an interframe motion vector of each nearby block determined in each reference frame;

determining, for each picture block comprised in the current video frame, a value of a similarity between the interframe motion vector of each nearby block determined for the picture block and an interframe motion vector of the picture block;

determining, for each picture block comprised in the current video frame, a value of a dissimilarity between the interframe motion vector and a motion vector that are of each nearby block; and

determining, according to the determined value of the similarity and the determined value of the dissimilarity, a picture block comprised in the candidate small-object region in the current video frame, wherein each picture block comprised in the candidate small-object region meets the following;

in multiple nearby blocks that are determined for the picture block and that are comprised in each reference frame corresponding to the current video frame, there are a first set quantity of nearby blocks whose values of similarities are all greater than or equal to a first threshold and there are a second set quantity of nearby blocks whose values of dissimilarities are all greater than or equal to a second threshold.

5. A method for determining a small-object region in a video frame, comprising:

dividing a current video frame into at least two regions;

determining a motion vector corresponding to each region;

determining an interframe motion vector of each group of two adjacent frames that comprise the current video frame and reference frames of the current video frame;

determining a candidate small-object region in the current video frame according to the determined interframe motion vector of the each group of two adjacent frames and the determined motion vector corresponding to each region; and

performing filtering on the candidate small-object region in the current video frame to determine a region obtained after the filtering as a small-object region in the current video frame,

wherein the reference frames of the current video frame comprises one or more of preceding continuous video frames of the current video frame and following continuous video frames of the current video frame,

wherein before performing filtering on the candidate small-object region in the current video frame, the method further comprises marking a specific marker on each picture block comprised in the candidate small-object region, and

wherein performing filtering on the candidate small-object region in the current video frame and determining the region obtained after the filtering as the small-object region in the current video frame comprises;

determining, for each picture block comprised in the candidate small-object region in the current video frame, a value of a first quantity of picture blocks that are marked with the specific marker and that are in a first set range in a horizontal direction of the picture block, and a value of a second quantity of picture blocks that are marked with the specific marker and that are in a second set range in a vertical direction of the picture block;

removing the specific marker of the picture block when the determined value of the first quantity or the determined value of the second quantity is greater than a third threshold;

determining a value of a third quantity of picture blocks that are marked with the specific marker and that are in a third set range around the picture block;

removing the specific marker of the picture block when the determined value of the third quantity is less than a fourth threshold; and

determining the picture block that is marked with the specific marker and that is in the current video frame as the small-object region in the current video frame,

wherein the third set range is smaller than the first set range and the second set range, and

wherein the fourth threshold is less than the third threshold.

6. An apparatus for determining a small-object region in a video frame, comprising:

a computer processor configured to;

divide a current video frame into at least two regions;

determine a global motion vector corresponding to each region;

transmit the determined global motion vector;

determine an interframe motion vector of each group of adjacent frames in multiple video frames that comprise the current video frame and a reference frame of the current video frame;

transmit the determined interframe motion vector;

determine information about a candidate small-object region in the current video frame according to the interframe motion vector that is of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame and that is from the global motion vector that is corresponding to each region;

transmit the information about the determined candidate small-object region in the current video frame;

determine the candidate small-object region in the current video frame according to the information that is about the candidate small-object region in the current video frame;

perform filtering on the candidate small-object region in the current video frame; and

determine a region obtained after the filtering as a small-object region in the current video frame,

wherein the reference frame of the current video frame comprises one or more of preceding continuous video frames of the current video frame and following continuous video frames of the current video frame,

wherein the computer processor is further configured to execute, before the interframe motion vector of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame is determined, the following for a to-be-processed picture block comprised in each video frame in the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame;

selecting at least one video frame from preceding N video frames of the current video frame;

determining, according to a small-object region determined in the preceding N video frames of the current video frame, whether a reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is a picture block comprised in the small-object region;

determining that the to-be-processed picture block is a first-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is the picture block comprised in the small-object region;

determining that the to-be-processed picture block is a second-type to-be-processed picture block when the reference picture block that is in the selected at least one video frame and that is corresponding to the to-be-processed picture block is not the picture block comprised in the small-object region;

separately determine an interframe motion vector of each first-type to-be-processed picture block and an interframe motion vector of each second-type to-be-processed picture block; and

use the determined interframe motion vector of each first-type to-be-processed picture block comprised in each video frame in the each group of adjacent frames and the determined interframe motion vector of each second-type to-be-processed picture block comprised in each video frame in the each group of adjacent frames as the interframe motion vector of the each group of adjacent frames, and

wherein the computer processor is further configured to;

determine a value of a dissimilarity between each candidate motion vector corresponding to the first-type to-be-processed picture block and a background motion vector of a video frame in which the first-type to-be-processed picture block is located;

assign a corresponding weight to each candidate motion vector according to the determined value of the dissimilarity corresponding to each candidate motion vector by using a rule that a smaller weight is assigned to a larger value of the dissimilarity; and

determine the interframe motion vector of the first-type to-be-processed picture block according to the weight assigned to each candidate motion vector and a Sum of Absolute Difference (SAD) value of pixels of a picture block pointed to by each candidate motion vector and pixels of the first-type to-be-processed picture block.

7. An apparatus for determining a small-object region in a video frame, comprising:

a computer processor configured to;

divide a current video frame into at least two regions;

determine a global motion vector corresponding to each region;

transmit the determined global motion vector;

determine an interframe motion vector of each group of adjacent frames in multiple video frames that comprise the current video frame and a reference frame of the current video frame;

transmit the determined interframe motion vector;

determine information about a candidate small-object region in the current video frame according to the interframe motion vector that is of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame and that is from the global motion vector that is corresponding to each region;

transmit the information about the determined candidate small-object region in the current video frame;

determine the candidate small-object region in the current video frame according to the information that is about the candidate small-object region in the current video frame;

perform filtering on the candidate small-object region in the current video frame; and

determine a region obtained after the filtering as a small-object region in the current video frame,

wherein the reference frame of the current video frame comprises one or more of preceding continuous video frames of the current video frame and following continuous video frames of the current video frame, and

wherein the computer processor is further configured to;

determine, in each reference frame of the current video frame and according to the interframe motion vector of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame, a matching block corresponding to each picture block comprised in the current video frame;

determine, in each reference frame, a nearby block near the matching block, and determine an interframe motion vector of each nearby block determined in each reference frame;

determine, for each picture block comprised in the current video frame, a value of a similarity between the interframe motion vector of each nearby block determined for the picture block and an interframe motion vector of the picture block;

determine, for each picture block comprised in the current video frame, a value of a dissimilarity between the interframe motion vector and a global motion vector that are of each nearby block; and

determine, according to the determined value of the similarity and the determined value of the dissimilarity, a picture block comprised in the candidate small-object region in the current video frame, wherein each picture block comprised in the candidate small-object region meets the following;

in multiple nearby blocks that are determined for the picture block and that are comprised in each reference frame corresponding to the current video frame, there are a first set quantity of nearby blocks whose values of similarities are all greater than or equal to a first threshold and there are a second set quantity of nearby blocks whose values of dissimilarities are all greater than or equal to a second threshold.

8. An apparatus for determining a small-object region in a video frame, comprising:

a computer processor configured to;

divide a current video frame into at least two regions;

determine a global motion vector corresponding to each region;

transmit the determined global motion vector;

determine an interframe motion vector of each group of adjacent frames in multiple video frames that comprise the current video frame and a reference frame of the current video frame;

transmit the determined interframe motion vector,

determine information about a candidate small-object region in the current video frame according to the interframe motion vector that is of the each group of adjacent frames in the multiple video frames that comprise the current video frame and the reference frame of the current video frame and that is from the global motion vector that is corresponding to each region;

transmit the information about the determined candidate small-object region in the current video frame;

determine the candidate small-object region in the current video frame according to the information that is about the candidate small-object region in the current video frame;

perform filtering on the candidate small-object region in the current video frame; and

determine a region obtained after the filtering as a small-object region in the current video frame,

wherein the reference frame of the current video frame comprises one or more of preceding continuous video frames of the current video frame and following continuous video frames of the current video frame, and

wherein the computer processor is further configured to;

mark, before filtering is performed on the candidate small-object region in the current video frame, a specific marker on each picture block comprised in the candidate small-object region;

determine, for each picture block comprised in the candidate small-object region in the current video frame, a value of a first quantity of picture blocks that are marked with the specific marker and that are in a first set range in a horizontal direction of the picture block, and a value of a second quantity of picture blocks that are marked with the specific marker and that are in a second set range in a vertical direction of the picture block;

remove the specific marker of the picture block when the determined value of the first quantity or the determined value of the second quantity is greater than a third threshold;

determine a value of a third quantity of picture blocks that are marked with the specific marker and that are in a third set range around the picture block;

remove the specific marker of the picture block when the determined value of the third quantity is less than a fourth threshold; and

determine the picture block that is marked with the specific marker and that is in the current video frame as the small-object region in the current video frame, wherein the third set range is smaller than the first set range and the second set range, and the fourth threshold is less than the third threshold.