Motion based dynamic resolution multiple bit rate video encoding

• US 8,270,473 B2
• Filed: 06/12/2009
• Issued: 09/18/2012
• Est. Priority Date: 06/12/2009
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Chen, William, Chan, Chun-Wei, Spears, Stacey, He, Yaming, Folta, Florin, Gu, Chuang, Hor, King Wei
• Primary Examiner(s): AN, SHAWN S
• Application Number:
  US12/483,577
• Publication Number:
  US 20100316126A1
• Time in Patent Office: 1,194 Days
• Field of Search: 375/240.03, 375/240.05, 375/240.16, 375/240.15, 375/240.12, 375/240.24, 382/251, 382/236, 382/238, 382/239
• US Class Current: 375/240.03
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/139 Analysis of motion vectors, e.g. their magnitude, direction, variance or reliability
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/149 by estimating the code amount by means of a model, e.g. mathematical model or statistical model
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/194 involving only two passes
• CPC Class Codes: H04N 19/57 Motion estimation characterised by a search window with variable size or shape
• CPC Class Codes: H04N 19/587 involving temporal sub-sampling or interpolation, e.g. decimation or subsequent interpolation of pictures in a video sequence
• CPC Class Codes: H04N 19/61 in combination with predictive coding

Abstract

A video encoding system encodes video streams for multiple bit rate video streaming using an approach that permits the encoded resolution to vary based, at least in part, on motion complexity. The video encoding system dynamically decides an encoding resolution for segments of the multiple bit rate video streams that varies with video complexity so as to achieve a better visual experience for multiple bit rate streaming. Motion complexity may be considered separately, or along with spatial complexity, in making the resolution decision.

Claims

1. A method of processing input video for motion based dynamic resolution encoding of segmented compressed video streams for multi bit rate video streaming using a computing device, the computing device comprising a processing unit and memory, the method comprising:

with the computing device;

encoding input video content in one or more segments, each segment comprising one or more groups of pictures, wherein the encoding comprises;

for a group of pictures of the one or more groups of pictures, obtaining at least three encoding samples for varying resolution and quantization step size;

establishing a model relating resolution, quantization step size and resulting encoded size for the group of pictures;

calculating an average median motion vector value for the groups of pictures;

deciding a video resolution and quantization step size based, at least in part, on the average median motion vector value, a motion threshold, and the model; and

encoding the group of pictures with the decided video resolution and quantization step size.

2. The method of claim 1 further comprising:

for each P picture in the group of pictures;

calculating a median value of motion vector magnitudes from the motion vectors of the P picture;

wherein the average median motion vector value is an average of the median values for each of the P pictures in the group of pictures.

3. The method of claim 1, wherein the deciding the video resolution and quantization step size comprises:

comparing the average median motion vector value with the motion threshold;

when the average median motion vector value is above the motion threshold, determining a motion resize scale value for resizing resolution of the group of pictures; and

using the motion resize scale value in deciding the video resolution and quantization step size.

4. The method of claim 3, wherein the motion resize scale value is determined using a piecewise linear motion resize scale mapping function.

5. The method of claim 1, wherein the deciding the video resolution and quantization step size comprises:

comparing the average median motion vector value with the motion threshold;

if the comparison indicates that the average median motion vector values is greater than the motion threshold, deciding to reduce resolution of the group of pictures based on motion complexity; and

otherwise, deciding not to reduce resolution of the group of pictures based on the motion complexity.

6. The method of claim 1, wherein the deciding the video resolution and quantization step size is further based on a texture measurement performed on the group of pictures.

7. The method of claim 1 further comprising:

performing a texture measurement on the group of pictures;

determining a quantization threshold for resizing resolution based on the texture measurement;

determining a quantization step size that yields an encoded size corresponding to a desired bit rate for the group of pictures;

comparing the determined quantization step size to the quantization threshold for resizing resolution; and

if the comparison is indicative of resizing resolution, calculating a texture scale factor for the group of pictures, wherein the deciding a video resolution and quantization step size is further based on the texture scale factor.

8. The method of claim 7 further comprising:

comparing the average median motion vector value with the motion threshold; and

based on the comparison of the average median motion vector value with the motion threshold, determining a motion resize scale value;

wherein the determining the video resolution comprises multiplying the motion resize scale value by the texture scale factor.

9. A method of processing input video for motion based dynamic resolution encoding of segmented compressed video streams for multi bit rate video streaming using a computing device, the computing device comprising a processing unit and memory, the method comprising:

with the computing device;

encoding input video content in segments, each segment comprising one or more groups of pictures, as a plurality of compressed video streams ranging in video quality from a top video stream to a bottom video stream, wherein the encoding comprises;

for a group of pictures of a video stream to be encoded, obtaining at least three encoding samples for varying resolution and quantization step size;

establishing a model relating resolution, quantization step size and resulting encoded size for the group of pictures;

performing a texture complexity measurement on the group of pictures;

performing a motion complexity measurement on the group of pictures;

determining a quantization threshold for resizing resolution based, at least in part, on the texture complexity measurement;

dynamically deciding a video resolution and quantization step size for encoding the group of pictures based, at least in part, on the model, the quantization threshold for resizing, the motion complexity measurement, and a motion resize threshold; and

encoding the group of pictures with the dynamically decided video resolution and quantization step size.

10. The method of claim 9 further comprises:

detecting scene changes in the input video content; and

wherein said dynamically deciding video resolution comprises applying dynamic video resolution changes at scene change boundaries between groups of pictures.

11. The method of claim 9 wherein said establishing the model comprises:

establishing a linear model for a relation of quantization step size to encoded size for a given video resolution based on at least two out of the at least three encoding samples of the group of pictures where the encoded size is sampled at the given video resolution and varying quantization step size.

12. The method of claim 11 wherein said performing the motion complexity measurement on the group of pictures comprises calculating an average median motion vector value for the groups of pictures, and wherein said dynamically deciding the video resolution comprises:

determining a quantization step size that yields an encoded size corresponding to a desired bit rate for the group of pictures when encoding at the given video resolution according to the established linear model relating quantization step size to encoded size for the given video resolution;

comparing the determined quantization step size to the quantization threshold for resizing resolution;

if the comparison is not indicative of resizing resolution, deciding to encode the group of pictures using the given video resolution and determined quantization step size; and

otherwise;

when the average median motion vector value is greater than the motion resize threshold, deciding to resize the video resolution for encoding the group of pictures based on texture complexity and motion complexity scaling values; and

otherwise, deciding to resize the video resolution for encoding the group of pictures based on only on the texture complexity scaling value.

13. The method of claim 12 wherein said establishing the model further comprises:

establishing a linear model for a relation of video resolution to encoded size for a given quantization step size based on at least two out of the at least three encoding samples of the group of pictures where the encoded size is sampled at the given quantization step size and varying video resolution.

14. The method of claim 13 wherein said dynamically deciding the video resolution further comprises:

in the case where the video resolution for encoding the group of pictures is to be resized, determining a resized video resolution that yields an encoded size corresponding to a desired bit rate for the group of pictures according to the established linear model relating video resolution to encoded size for the given quantization step size.

15. The method of claim 14 wherein, in the case where the video resolution for encoding the group of pictures is to be resized, said dynamically deciding the quantization step size comprises:

establishing a linear model relating quantization step size to resolution at the desired bit rate for the group of pictures based on the at least three encoding samples; and

deciding the quantization step size for the resized video resolution according to the linear model relating quantization step size to resolution.

16. The method of claim 9, wherein the performing the motion complexity measurement on the group of pictures comprises:

for each P picture in the group of pictures;

calculating a median value of motion vector magnitudes from the motion vectors of the P picture; and

calculating an average median motion vector value by averaging the median values for each of the P pictures in the group of pictures.

17. A video processing system for motion based dynamic resolution encoding of segmented compressed video streams for multi bit rate video streaming, the system comprising:

a memory for storing raw video content to be encoded;

a computer processing means for encoding the raw video content using a variable bit rate and motion based dynamic resolution approach in segments comprising groups of pictures as a plurality of compressed video streams, wherein said encoding by the computer processing means comprises;

for a group of pictures of the one or more groups of pictures, obtaining at least three encoding samples for varying resolution and quantization step size;

establishing a model relating resolution, quantization step size and resulting encoded size for the group of pictures;

calculating an average median motion vector value for the groups of pictures;

deciding a video resolution and quantization step size based, at least in part, on the average median motion vector value, a motion threshold, and the model; and

encoding the group of pictures with the decided video resolution and quantization step size.

18. The video processing system of claim 17 further comprising:

for each P picture in the group of pictures;

calculating a median value of motion vector magnitudes from the motion vectors of the P picture;

wherein the average median motion vector value is an average of the median values for each of the P pictures in the group of pictures.

19. The video processing system of claim 17, wherein the deciding the video resolution and quantization step size comprises:

comparing the average median motion vector value with the motion threshold;

when the average median motion vector value is above the motion threshold, determining a motion resize scale value for resizing resolution of the group of pictures; and

using the motion resize scale value in deciding the video resolution and quantization step size.

20. The video processing system of claim 17, wherein the deciding the video resolution and quantization step size comprises:

comparing the average median motion vector value with the motion threshold;

if the comparison indicates that the average median motion vector values is greater than the motion threshold, deciding to reduce resolution of the group of pictures based on motion complexity; and

otherwise, deciding not to reduce resolution of the group of pictures based on the motion complexity.