Optical flow based motion vector estimation systems and methods

• US 8,718,143 B2
• Filed: 05/15/2012
• Issued: 05/06/2014
• Est. Priority Date: 10/31/2007
• Status: Active Grant

• Current Assignee: Avago Technologies International Sales Pte Limited (Broadcom, Inc.)
• Original Assignee: Broadcom Corporation (Broadcom, Inc.)
• Inventors: Chen, Xuemin
• Primary Examiner(s): Chriss, Andrew
• Assistant Examiner(s): Rivas, Salvador E
• Application Number:
  US13/471,677
• Publication Number:
  US 20120314772A1
• Time in Patent Office: 721 Days
• US Class Current: 375/240.16
• CPC Class Codes: G06T 7/20 Analysis of motion motion estimation for coding, decoding, compressing or decompressing digital video signals H04N19/43, H04N19/51
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/196 being specially adapted for the computation of encoding parameters, e.g. by averaging previously computed encoding parameters processing of motion vectors H04N19/513
• CPC Class Codes: H04N 19/44 Decoders specially adapted therefor, e.g. video decoders which are asymmetric with respect to the encoder
• CPC Class Codes: H04N 19/51 Motion estimation or motion compensation
• CPC Class Codes: H04N 19/521 for estimating the reliability of the determined motion vectors or motion vector field, e.g. for smoothing the motion vector field or for correcting motion vectors
• CPC Class Codes: H04N 19/527 Global motion vector estimation
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/85 using pre-processing or post-processing specially adapted for video compression
• CPC Class Codes: H04N 19/86 involving reduction of coding artifacts, e.g. of blockiness
• CPC Class Codes: H04N 7/0127 by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
• CPC Class Codes: H04N 7/014 involving the use of motion vectors motion estimation and compensation in video coding H04N19/51

Abstract

Certain aspects of a method and system for optical flow based motion vector estimation for picture rate up-conversion (PRUC) may include generating one or more motion vectors based on extracted picture rate up-conversion (PRUC) data by minimizing a cost function. The cost function may be constrained by any combination of a block matching constraint, a smoothness constraint and a bias constraint. The PRUC data may be extracted from a compressed video data stream while the compressed video data stream is being decompressed by a video decompression engine. The PRUC data may comprise local block motion vectors, block coding modes, quantization levels, quantized residual data and decoded pictures. A plurality of interpolated pictures may be generated based on extracting the PRUC data.

Claims

1. A method, comprising:

receiving a compressed video stream comprising adjacent picture data; and

generating optical flow motion vectors based on the adjacent picture data by minimizing a cost function corresponding to optical flow distortion, the cost function constrained by a block matching constraint, a smoothness constraint, and a bias constraint, the cost function corresponding to a measurement of joint errors with respect to the block matching constraint, the smoothness constraint, and the bias constraint for horizontal and vertical image velocity components, an adaptive smoothness parameter w_S, and an adaptive bias parameter w_B, where;

the block matching constraint is based on a brightness distribution of the optical flow motion vectors over an image space and time;

the smoothness constraint is based on a variation in an optical flow field across the image space; and

the bias constraint is based on measuring a deviation of the optical flow motion vectors from a reference motion vector.

2. The method of claim 1, further comprising extracting the adjacent picture data from the compressed video stream while the compressed video stream is being decompressed.

3. The method of claim 1, further comprising generating one or more interpolated pictures based on the optical flow motion vectors.

4. The method of claim 1, wherein the block matching constraint corresponds to a squared displacement of image brightness with horizontal and vertical image velocity components at a point over a change in time.

5. The method of claim 1, wherein the smoothness constraint corresponds to a measurement of the optical flow field across the image space.

6. The method of claim 1, wherein the bias constraint corresponds to a measurement of closeness between the optical flow motion vectors and the reference motion vector.

7. The method of claim 1, wherein the reference motion vector comprises a global motion vector or a regionally dominant motion vector.

8. The method of claim 1, wherein the adaptive smoothness parameter w_S and the adaptive bias parameter w_B are either fixed or variable.

9. The method of claim 1, wherein the adjacent picture data corresponds to one or more received pictures of the compressed video stream.

10. The method of claim 9, wherein the one or more received pictures comprises one or more adjacent pictures.

11. The method of claim 9, wherein the adjacent picture data comprises motion vector data from the one or more received pictures.

12. A system, comprising:

motion estimation circuitry configured to;

generate optical flow motion vectors based on adjacent picture data, received in a compressed video stream, by minimizing a cost function corresponding to optical flow distortion, the cost function constrained by a block matching constraint, a smoothness constraint, and a bias constraint, the cost function corresponding to a measurement of joint errors with respect to the block matching constraint, the smoothness constraint, and the bias constraint for horizontal and vertical image velocity components, an adaptive smoothness parameter w_S, and an adaptive bias parameter w_B, where;

the block matching constraint is based on a brightness distribution of the optical flow motion vectors over an image space and time;

the smoothness constraint is based on a variation in an optical flow field across the image space; and

the bias constraint is based on measuring a deviation of the optical flow motion vectors from a reference motion vector.

13. The system of claim 12, further comprising decompression circuitry coupled to the motion estimation circuitry, the decompression circuitry configured to:

extract the adjacent picture data from the compressed video stream while the compressed video stream is being decompressed by the decompression circuitry; and

provide the adjacent picture data to the motion estimation circuitry.

14. The system of claim 12, further comprising motion compensated interpolation circuitry coupled to the motion estimation circuitry, wherein the motion compensated interpolation circuitry is configured to receive the optical flow motion vectors from the motion estimation circuitry and generate one or more interpolated pictures based on the optical flow motion vectors.

15. The system of claim 12, wherein:

the block matching constraint corresponds to a squared displacement of image brightness with horizontal and vertical image velocity components at a point over a change in time;

the smoothness constraint corresponds to a measurement of the optical flow field across the image space; and

the bias constraint corresponds to a measurement of closeness between the optical flow motion vectors and the reference motion vector.

16. The system of claim 12, wherein the reference motion vector comprises a global motion vector or a regionally dominant motion vector.

17. The system of claim 12, wherein the adjacent picture data corresponds to one or more received pictures of the compressed video stream, the one or more received pictures comprising one or more adjacent pictures.

18. The system of claim 12, wherein the adaptive smoothness parameter w_S and the adaptive bias parameter w_B are either fixed or variable.

19. A non-transitory, computer readable storage medium comprising executable code executed by a machine to cause the machine to:

receive a compressed video stream comprising adjacent picture data; and

generate optical flow motion vectors based on the adjacent picture data by minimizing a cost function corresponding to optical flow distortion, the cost function constrained by a block matching constraint, a smoothness constraint, and a bias constraint, the cost function corresponding to a measurement of joint errors with respect to the block matching constraint, the smoothness constraint, and the bias constraint for horizontal and vertical image velocity components, an adaptive smoothness parameter w_S, and an adaptive bias parameter w_B, where;

the block matching constraint is based on a brightness distribution of the optical flow motion vectors over an image space and time;

the smoothness constraint is based on a variation in an optical flow field across the image space; and

the bias constraint is based on measuring a deviation of the optical flow motion vectors from a reference motion vector.

20. The non-transitory, computer readable storage medium of claim 19, wherein the adaptive smoothness parameter w_S and the adaptive bias parameter w_B are either fixed or variable.