Joint rate control for stereoscopic video coding

• US 6,151,362 A
• Filed: 08/06/1999
• Issued: 11/21/2000
• Est. Priority Date: 10/30/1998
• Status: Expired due to Term

• Current Assignee: Google Technology Holdings LLC (Alphabet Inc.), ARRIS Enterprises LLC (CommScope Holding Company, Inc.)
• Original Assignee: General Instrument Corporation (CommScope Holding Company, Inc.), Motorola, Inc. (Motorola Solutions, Inc.)
• Inventors: Wang, Limin
• Primary Examiner(s): Kelley, Chris S.
• Assistant Examiner(s): WONG, ALLEN C
• Application Number:
  US09/369,624
• Time in Patent Office: 473 Days
• Field of Search: 348/43, 348/419, 348/405, 348/584, 348/409, 375/240.12
• US Class Current: 375/240.12
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/194 Transmission of image signals
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/117 Filters, e.g. for pre-processing or post-processing sub-band filter banks H04N19/635
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/149 by estimating the code amount by means of a model, e.g. mathematical model or statistical model
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/597 specially adapted for multi-view video sequence encoding
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals

Abstract

A method and apparatus are provided for controlling and balancing a quality ratio between left and right channels of a stereoscopic video sequence. A complexity measure for each of the channels is first determined. Thereafter, a number of bits representing the video sequence is allocated between the channels according to each channel'"'"'s complexity. The resultant joint rate control allows the bit rates for the two channels to vary according to the complexity measures of their pictures, while the aggregate bit rate is maintained at the channel capacity, which is a constant value.

Claims

1. A method for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising the steps of:

determining a complexity measure for each of said channels;

wherein said channels comprise successive image frames of said stereoscopic video sequence, including a first channel comprising a base layer, and a second channel comprising an enhancement layer;

allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

grouping a number of frames of the first and second channels into successive respective groups of pictures (GOPs); and

forming successive joint groups of pictures (GOPs) from corresponding GOPs of the first and second channels;

wherein;

each joint GOP includes the same number of I, P and B pictures from the first channel, and P and B pictures from the second channel; and

said allocating step allocates the same target number of bits to each joint GOP.

2. A method in accordance with claim 1, wherein said allocating step maintains an aggregate bit rate of the video sequence at a substantially constant rate.

3. A method in accordance with claim 2, wherein said constant rate is determined by a channel capacity of a communications channel over which said video sequence is to be communicated.

4. A method in accordance with claim 1 wherein:

the complexity measure of each channel is based on the complexity (C) of the image frames carried by the channel.

5. A method in accordance with claim 4 wherein:

the successive image frames of the first channel comprise I, P and B pictures; and

the successive image frames of the second channel comprise only P and B pictures.

6. A method in accordance with claim 5 wherein:

a complexity measure C, is maintained for I pictures;

a complexity measure C_1P is maintained for the first channel P pictures;

a complexity measure C_1B is maintained for the first channel B pictures;

a complexity measure C_2P is maintained for the second channel P pictures; and

a complexity measure C_2B is maintained for the second channel B pictures.

7. A method in accordance with claim 6, wherein:

said allocating step is responsive to a target number of bits (T_m,n) determined as;

##EQU36## where;

C_m,n is the complexity measure for the picture type of frame (m,n),

m is a channel index and n is a frame index,

subscripts I, P and B denote, respectively, I, P and B pictures,

subscripts 1 and 2 denote, respectively, the first and second channel,

N_I, N_1P, N_1B, N_2P and N_2B are, respectively, the number of remaining I pictures, first-channel P pictures, first-channel B pictures, second-channel P pictures and second-channel B pictures in the joint GoP,

w denotes a quality balance weighting factor,

K denotes a quantization parameter weighting factor, and

T_r is the remaining number of bits in the joint GoP, defined as T_r =T_r -R_m'"'"',n'"'"', where R_m'"'"',n'"'"' is the number of bits generated for a frame (m'"'"',n'"'"') coded just before frame (m,n).

8. A method in accordance with claim 1 wherein:

each image frame is divided into a plurality of slices; and

the complexity measure of each channel is based on the complexity (C) of the image frame slices carried by the channel.

9. A method in accordance with claim 1 wherein:

the complexity measure of each channel is based on the complexity (C) of the image frames carried by the channel; and

said allocating step is responsive to a target bit rate T_m,n assigned to each frame based on the complexity (C) of the frame, where m is a channel index and n is a frame index;

said method comprising the further step of controlling said target bit rate to prevent the underflow and overflow of buffers in a video encoder processing said video image and a video decoder receiving said video image.

10. A method in accordance with claim 9, wherein said target bit rate is adjusted in accordance with variable lower and upper bounds for each frame.

11. A method in accordance with claim 1, wherein:

the base layer is coded independently, and the enhancement layer is coded, at least in part, based on the first channel.

12. Apparatus for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising:

a processor adapted to determine a complexity measure for each of said channels;

wherein said channels comprise successive image frames of said stereoscopic video sequence, including a first channel comprising a base layer, and a second channel comprising an enhancement layer;

means for allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

means for grousing a number of frames of the first and second channels into successive respective groups of pictures (GOPs); and

means for forming successive joint groups of pictures (GOPs) from corresponding GOPs of the first and second channels;

wherein each joint GOP includes the same number of I, P and B pictures from the first channel, and P and B pictures from the second channel; and

said allocating means allocates the same target number of bits to each joint GOP.

13. Apparatus in accordance with claim 12, wherein said allocating means maintains an aggregate bit rate of the video sequence at a substantially constant rate.

14. Apparatus in accordance with claim 13, wherein said constant rate is determined by a channel capacity of a communications channel over which said video sequence is to be communicated.

15. An apparatus in accordance with claim 12, wherein:

the base layer is coded independently, and the enhancement layer is coded, at least in part, based on the first channel.

16. A method for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising the steps of:

determining a complexity measure for each of said channels; and

allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

wherein;

said channels carry successive image frames of said stereoscopic video sequence;

the complexity measure of a channel is based on the complexity (C) of the image frames carried by the channel; and

the complexity (C) of an image frame comprises the product of an average quantization parameter (Q) used for that frame and the number of bits (R) generated when that frame is quantized using the quantization parameter Q.

17. A method in accordance with claim 16 wherein said allocating step is responsive to a target number of bits (T_m,n) assigned to each frame based on the complexity (C) of the frame, where m is a channel index and n is a frame index.

18. A method in accordance with claim 17, comprising the further steps of:

grouping a number N of frames of each channel into a respective group of pictures (GoP) for the channel;

forming a joint GoP from corresponding GoPs of first and second channels, said joint GoP comprising 2N frames; and

determining a nominal target number of bits (T) for the joint GoP;

wherein said allocating step is responsive to said nominal target number of bits (T).

19. A method in accordance with claim 17 comprising the further step of:

determining whether the target number of bits assigned to a particular frame is within a predetermined range and, if not, adjusting the target rate to be within said range prior to encoding that particular frame.

20. A method in accordance with claim 16 comprising the further step of independently weighting the quantization parameter (Q) for each different type of image frame and the particular channel in which the image frame is carried.

21. A method for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising the steps of:

(a) determining a complexity measure for each of said channels;

(b) allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

wherein;

said channels carry successive image frames of said stereoscopic video sequence;

the complexity measure of a channel is based on the complexity (C) of the image frames carried by the channel;

the successive image frames of a first channel comprise I, P and B pictures;

the successive image frames of a second channel comprise only P and B pictures;

a complexity measure C_I is maintained for I pictures;

a complexity measure C_1P is maintained for the first channel P pictures;

a complexity measure C_1B is maintained for the first channel B pictures;

a complexity measure C_2P is maintained for the second channel P pictures; and

a complexity measure C_2B is maintained for the second channel B pictures;

(c) grouping a number of frames of each channel into a respective group of pictures (GoP) for the channel; and

(d) forming a joint GoP from corresponding GoPs of said first and second channels;

wherein said allocating step is responsive to a target number of bits (T_m,n) determined as;

##EQU37## where;

C_m,n is the complexity measure for the picture type of frame (m,n),

m is a channel index and n is a frame index,

subscripts I, P and B denote, respectively, I, P and B pictures,

subscripts 1 and 2 denote, respectively, the first and second channels,

N_I, N_1P, N_1B, N_2P and N_2B are, respectively, the number of remaining I pictures, first-channel P pictures, first-channel B pictures, second-channel P pictures and second-channel B pictures in the joint GoP,

w denotes a quality balance weighting factor,

K denotes a quantization parameter weighting factor, and

T_r is the remaining number of bits in the joint GoP, defined as T_r =T_r -R_m'"'"',n'"'"', where_-- R_m'"'"',n'"'"' is the number of bits generated for a frame (m'"'"',n'"'"') coded just before frame (m,n).

22. A method for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising the steps of:

determining a complexity measure for each of said channels; and

allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

wherein;

said channels carry successive image frames of said stereoscopic video sequence, each image frame being divided into a plurality of slices;

the complexity measure of a channel is based on the complexity (C) of the image frame slices carried by the channel; and

the complexity (C) of an image frame slice comprises the product of a quantization parameter (Q) used for that slice and the number of bits (R) generated when that slice is quantized using the quantization parameter Q.

23. A method in accordance with claim 22 wherein said allocating step is responsive to a target number of bits (T_m,n,s) assigned to each slice based on the complexity (C) of the slice, where m is a channel index, n is a frame index, and s is a slice index.

24. A method in accordance with claim 23, wherein:

each image frame is divided into S slices resulting in S slice complexity measures for each frame; and

the target number of bits (T_m,n,s) for each slice s of a frame (m,n) is;

##EQU38## where;

T_m,n,r is a remaining number of bits up to slice s-1 for frame (m,n), defined as;

##EQU39## R_s'"'"' is a number of bits generated for slice s'"'"'.

25. A method in accordance with claim 24, wherein the complexity measures C_m,n,s are calculated from a previously coded frame of the same type as the current frame by multiplying the quantization parameter Q for each slice by the number of bits R in that slice.

26. A method for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising the steps of:

(a) determining a complexity measure for each of said channels; and

(b) allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

wherein;

said channels carry successive image frames of said stereoscopic video sequence;

the complexity measure of a channel is based on the complexity (C) of the image frames carried by the channel; and

said allocating step is responsive to a target bit rate T_m,n assigned to each frame based on the complexity (C) of the frame, where m is a channel index and n is a frame index; and

(c) controlling said target bit rate to prevent the underflow and overflow of buffers in a video encoder processing said video image and a video decoder receiving said video image;

wherein;

said target bit rate is adjusted in accordance with variable lower and upper bounds for each frame;

said target bit rate is maintained within said lower and upper bounds according to the parameters ##EQU40## the quantity 2N is the number of frames in a joint group of pictures (GOP), ##EQU41## is the number of bits transmitted for channel m during the nth frame period, ##EQU42## is the fullness of a virtual encoder buffer for channel m at the frame instant of n, B^d_max is the maximum fullness of a decoder buffer of said video decoder, and N'"'"' is a decoding delay of said video decoder; and

all of the terms in the lower bound are determinable at the video encoder.

27. An apparatus for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising:

means for determining a complexity measure for each of said channels; and

means for allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

wherein;

said channels carry successive image frames of said stereoscopic video sequence;

the complexity measure of a channel is based on the complexity (C) of the image frames carried by the channel; and

the complexity (C) of an image frame comprises the product of an average quantization parameter (Q) used for that frame and the number of bits (R) generated when that frame is quantized using the quantization parameter Q.

28. An apparatus for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising:

means for determining a complexity measure for each of said channels;

means for allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

wherein;

said channels carry successive image frames of said stereoscopic video sequence;

the complexity measure of a channel is based on the complexity (C) of the image frames carried by the channel;

the successive image frames of a first channel comprise I, P and B pictures;

the successive image frames of a second channel comprise only P and B pictures;

a complexity measure C_I is maintained for I pictures;

a complexity measure C_1P is maintained for the first channel P pictures;

a complexity measure C_1B is maintained for the first channel B pictures;

a complexity measure C_2P is maintained for the second channel P pictures; and

a complexity measure C_2B is maintained for the second channel B pictures;

means for grouping a number of frames of each channel into a respective group of pictures (GoP) for the channel; and

means for forming a joint GoP from corresponding GoPs of said first and second channels, said joint GoP comprising 2N frames;

wherein said allocating means is responsive to a target number of bits (T_m,n) determined as;

##EQU43## where;

C_m,n is the complexity measure for the picture type of frame (m,n),

m is a channel index and n is a frame index,

subscripts I, P and B denote, respectively, I, P and B pictures,

subscripts 1 and 2 denote, respectively, the first and second channels,

N_I, N_1P, N_1B, N_2P and N_2B are, respectively, the number of remaining I pictures, first-channel P pictures, first-channel B pictures second-channel P pictures and second-channel B pictures in the joint GoP,

w denotes a quality balance weighting factor,

K denotes a quantization parameter weighting factor, and

T_r is the remaining number of bits in the joint GoP, defined as T_r =T_r -R_m'"'"',n'"'"', where_-- R_m'"'"',n'"'"' is the number of bits generated for a frame (m'"'"',n'"'"') coded just before frame (m,n).

29. An apparatus for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising:

means for determining a complexity measure for each of said channels; and

means for allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

wherein;

said channels carry successive image frames of said stereoscopic video sequence, each image frame being divided into a plurality of slices;

the complexity measure of a channel is based on the complexity (C) of the image frame slices carried by the channel; and

the complexity (C) of an image frame slice comprises the product of a quantization parameter (Q) used for that slice and the number of bits (R) generated when that slice is quantized using the quantization parameter Q.

30. An apparatus for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising:

means for determining a complexity measure for each of said channels; and

means for allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

wherein;

said channels carry successive image frames of said stereoscopic video sequence;

the complexity measure of a channel is based on the complexity (C) of the image frames carried by the channel; and

said allocating means is responsive to a target bit rate T_m,n assigned to each frame based on the complexity (C) of the frame, where m is a channel index and n is a frame index; and

means for controlling said target bit rate to prevent the underflow and overflow of buffers in a video encoder processing said video image and a video decoder receiving said video image;

wherein;

said target bit rate is adjusted in accordance with variable lower and upper bounds for each frame;

said target bit rate is maintained within said lower and upper bounds according to the parameters ##EQU44## the quantity 2N is the number of frames in a joint group of pictures (GOP), ##EQU45## is the number of bits transmitted for channel m during the nth frame period, ##EQU46## is the fullness of a virtual encoder buffer for channel m at the frame instant of n, B^d_max is the maximum fullness of a decoder buffer of said video decoder, and N'"'"' is a decoding delay of said video decoder; and

all of the terms in the lower bound are determinable at the video encoder.

31. A method for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising the steps of:

(a) determining a complexity measure for each of said channels; and

(b) allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

wherein;

said channels carry successive image frames of said stereoscopic video sequence;

the complexity measure of a channel is based on the complexity (C) of the image frames carried by the channel; and

said allocating step is responsive to a target bit rate T_m,n assigned to each frame based on the complexity (C) of the frame, where m is a channel index and n is a frame index; and

(c) controlling said target bit rate to prevent the underflow and overflow of buffers in a video encoder processing said video image and a video decoder receiving said video image;

wherein;

said target bit rate is adjusted in accordance with variable lower and upper bounds for each frame; and

said target bit rate is maintained within said lower and upper bounds according to the parameters;

##EQU47## where the quantity R^c_m,n is the number of bits transmitted for channel m during the nth frame period, B^c_m,n is the fullness of a virtual encoder buffer for channel m at the frame instant of n, B^d_max is the maximum fullness of a decoder buffer of said video decoder, and N'"'"' is a decoding delay of said decoder buffer.

32. A method in accordance with claim 31 comprising the further step of predetermining the transmission rates for future frames of channel m to ensure that:

(i) an average input rate to the decoder buffer will be equal to an average output rate thereof, and

(ii) a total transmission rate of left and right stereoscopic channels is equal to a total available channel rate.

33. A method in accordance with claim 32 wherein the transmission rate for a frame n of channel m is set equal to the average number of compressed bits for each of N previous frames.

34. A method in accordance with claim 33 wherein 2N is the number of frames in a joint group of pictures (GOP).

35. An apparatus for balancing a quality ratio between a plurality of channels of a stereoscopic video sequence, comprising:

means for determining a complexity measure for each of said channels; and

means for allocating a number of bits representing said video sequence between said channels according to each channel'"'"'s complexity;

wherein;

said channels carry successive image frames of said stereoscopic video sequence;

the complexity measure of a channel is based on the complexity (C) of the image frames carried by the channel; and

said allocating means is responsive to a target bit rate T_m,n assigned to each frame based on the complexity (C) of the frame, where m is a channel index and n is a frame index; and

means for controlling said target bit rate to prevent the underflow and overflow of buffers in a video encoder processing said video image and a video decoder receiving said video image;

wherein;

said target bit rate is adjusted in accordance with variable lower and upper bounds for each frame; and

said target bit rate is maintained within said lower and upper bounds according to the parameters;

##EQU48## where the quantity ##EQU49## is the number of bits transmitted for channel m during the nth frame period, ##EQU50## is the fullness of a virtual encoder buffer for channel m at the frame instant of n, B^d_max is the maximum fullness of a decoder buffer of said video decoder, and N'"'"' is a decoding delay of said decoder buffer.