FRAME DROPPING ALGORITHM FOR FAST ADAPTATION OF BUFFERED COMPRESSED VIDEO TO NETWORK CONDITION CHANGES

• US 20110235709A1
• Filed: 04/07/2010
• Published: 09/29/2011
• Est. Priority Date: 03/25/2010
• Status: Abandoned Application

• Current Assignee: Apple Inc.
• Original Assignee: Apple Inc.
• Inventors: YANG, Yan, SCHMIDT, Jochen Christian, NORMILE, James Oliver, JEONG, Hyeonkuk, WU, Hsi-Jung, ABUAN, Joe, SHI, Xiaojin, ZHOU, Xiaosong
• Application Number:
  US12/756,100
• Publication Number:
  US 20110235709A1
• Time in Patent Office: Days
• US Class Current: 375/240.12
• CPC Class Codes: H04N 19/105 Selection of the reference unit for prediction within a chosen coding or prediction mode, e.g. adaptive choice of position and number of pixels used for prediction
• CPC Class Codes: H04N 19/12 Selection from among a plurality of transforms or standards, e.g. selection between discrete cosine transform [DCT] and sub-band transform or selection between H.263 and H.264
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/164 Feedback from the receiver or from the transmission channel
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/187 the unit being a scalable video layer
• CPC Class Codes: H04N 19/30 using hierarchical techniques, e.g. scalability H04N19/63 takes precedence
• CPC Class Codes: H04N 19/37 with arrangements for assigning different transmission priorities to video input data or to video coded data
• CPC Class Codes: H04N 19/577 Motion compensation with bidirectional frame interpolation, i.e. using B-pictures
• CPC Class Codes: H04N 19/59 involving spatial sub-sampling or interpolation, e.g. alteration of picture size or resolution
• CPC Class Codes: H04N 21/64723 Monitoring of network processes or resources, e.g. monitoring of network load traffic related reporting in data switching networks H04L43/062
• CPC Class Codes: H04N 21/64761 directed to the server

Abstract

A video coding and transmission system may employ techniques for adapting buffered video to network condition changes. Video data may be coded as reference data and non-reference data. According to the embodiments, non-reference frame may be detected in buffered video while awaiting transmission to a network. When network degradation is detected, one or more of the buffered non-reference frames may be dropped when network degradation is detected. Information about the dropped frames may be passed to an encoder for updating buffer parameters for future encoding. In this manner, a video coding system may provide faster responses to changing network conditions than systems without such buffer management techniques.

Claims

1. A method for adapting buffered video to network condition changes, the method comprising:

detecting non-reference frames in buffered video awaiting output to a network;

dropping one or more of the non-reference frames when network degradation is detected; and

generating information about the dropped frames to pass to an encoder for updating buffer parameters for future encoding.

2. The method of claim 1 further comprising, when network degradation is detected, revising coding parameters of the encoder to match a revised estimate of network conditions.

3. The method of claim 1, further comprising, prior to detection of the network degradation:

coding video data according to predictive coding techniques, the coding generating non-reference frames and reference frames,

detecting network instability, and

when network instability is detected, coding the video data by a decreased number of reference frames.

4. The method of claim 3, wherein coding video data according to predictive coding techniques, the coding generating non-reference frames and reference frames,

detecting network instability, and

when network instability is detected, the encoder engages scalable video coding, coding source video data as coded base layer data and coded enhancement layer data, and

wherein the dropping is performed on frames of the coded enhancement layer data.

5. The method of claim 1, further comprising:

dropping buffered reference frames, and

re-coding source video data corresponding to the dropped reference frames as non-reference frames.

6. The method of claim 5, further comprising:

dropping buffered coded data that relies on the dropped reference frames as a source of prediction, and

re-coding source video data corresponding to the dropped buffered coded data using different reference frame(s) as a source of prediction.

7. A method for adapting video coding to changing network conditions, comprising:

in a stable network state, coding video data according to coding parameters that match an estimate of network bandwidth, the coding involving coding select source data elements as prediction references for coding of other source data elements;

in an unstable network state, coding the video data according to coding parameters of the stable network state but with a reduced rate of prediction references as compared to the network state;

buffering coded data generated during the stable network state and the unstable network state for transmission via a network; and

in a network diminished state, selectively decimating buffered non-reference coded data.

8. The method of claim 7, further comprising, in the network diminished state, revising the estimate network conditions and revising coding parameters to match the revised estimate.

9. The method of claim 7, further comprising, in the network unstable state, identifying decimated data to the encoder for use in subsequent coding operations.

10. The method of claim 7, wherein

in the unstable network state, the coding reduces the number of reference frames in the coded video data, and

in the network diminished state, non-reference frames are decimated.

11. The method of claim 7, wherein

in the unstable network state, the coding reduces the number of reference slices in the coded video data, and

in the network diminished state, non-reference slices are decimated.

12. The method of claim 7, wherein

in the unstable network state, the coding engages scalable video coding, coding source video data as coded base layer data and coded enhancement layer data, and

in the network diminished state, coded enhancement layer data is decimated.

13. The method of claim 7, further comprising, in the network diminished state,

decimating buffered reference frame data, and

recoding source video data corresponding to decimated reference frame data as non-reference frame data.

14. The method of claim 13, further comprising:

decimating buffered coded data that relies on the decimated reference frame data as a source of prediction, and

recoding source video data corresponding to the decimated buffered coded data using different reference frame data as a source of prediction.

15. A video coder system comprising:

a video coder to code source video data according to coding parameters, the coded video data including reference data that is a source of prediction for other coded video data,

a transmit buffer to store coded video data prior to transmission,

a transmitter to read out coded video data from the transmit buffer and transmit it over a network,

a network monitor to estimate network conditions,

a buffer controller to selectively decimate non-reference data from the transmit buffer based on estimated network conditions, and

a coder controller to establish the coding parameters for the video coder based on the estimated network conditions and indicators of decimated data from the buffer controller.

16. The video coder system of claim 15, wherein, when the estimated network conditions indicate network instability, the coder controller reduces a number of reference frames generated by the video coder.

17. The video coder system of claim 15:

wherein, when the estimated network conditions indicate network instability, the coder controller reduces a number of reference slices in the coded video data,

wherein the decimation is performed on non-reference slices of the coded enhancement layer data.

18. The video coder system of claim 15:

wherein, when the estimated network conditions indicate network instability, the coder controller engages scalable video coding by the video coder, causing the video coder to code source video data as coded base layer data and coded enhancement layer data, and

wherein the decimation is performed on frames of the coded enhancement layer data.

The video coder system of claim 15, wherein the video coder system is a component of a videoconferencing terminal.

19. The video coder system of claim 15, wherein the video coder system is a component of a mobile terminal.

20. Computer readable medium storing program instructions that, when executed by a processor, cause the processor to:

detect non-reference frames in a transmit buffer awaiting output to a network;

drop one or more of the non-reference frames from the transmit buffer when network degradation is detected; and

generate information about the dropped frames to an encoder for updating buffer parameters for future encoding.

21. The computer readable medium of claim 20, wherein the instructions further cause the processor to, prior to detection of the network degradation:

code video data according to predictive coding techniques, the coding generating non-reference frames and reference frames,

detect network instability, and

when network instability is detected, code the video data by a decreased number of reference frames.

22. The computer readable medium of claim 20, wherein the instructions further cause the processor to, prior to detection of the network degradation:

code video data according to predictive coding techniques, the coding generating non-reference frames and reference frames,

detect network instability, and

when network instability is detected, engage scalable video coding in the encoder to code source video data as coded base layer data and coded enhancement layer data,

wherein the dropping is performed on frames of the coded enhancement layer data.

23. The computer readable medium of claim 20, wherein the instructions further cause the processor to, prior to detection of the network degradation:

code video data according to predictive coding techniques, the coding generating non-reference slices and reference slices,

detect network instability, and

when network instability is detected, code the video data by a decreased number of reference slices.