Latency-based statistical multiplexing

• US 6,665,872 B1
• Filed: 01/05/2000
• Issued: 12/16/2003
• Est. Priority Date: 01/06/1999
• Status: Expired due to Fees

• Current Assignee: AmTRAN Technology Co., Ltd.
• Original Assignee: Sarnoff Corporation (SRI International, Inc.)
• Inventors: Lee, Xiaobing, Krishnamurthy, Ravi, Sethuraman, Sriram, Chiang, Tihao
• Primary Examiner(s): BRITTON, HOWARD W
• Application Number:
  US09/478,127
• Time in Patent Office: 1,441 Days
• Field of Search: H04/N.758, 375/240.26, 725/95
• US Class Current: 725/95
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/14 Coding unit complexity, e.g. amount of activity or edge presence estimation H04N19/146 takes precedence
• CPC Class Codes: H04N 19/149 by estimating the code amount by means of a model, e.g. mathematical model or statistical model
• CPC Class Codes: H04N 19/15 by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/159 Prediction type, e.g. intra-frame, inter-frame or bidirectional frame prediction
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/42 characterised by implementation details or hardware specially adapted for video compression or decompression, e.g. dedicated software implementation H04N19/635 takes precedence
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 21/2365 Multiplexing of several video streams
• CPC Class Codes: H04N 21/23655 Statistical multiplexing, e.g. by controlling the encoder to alter its bitrate to optimize the bandwidth utilization
• CPC Class Codes: H04N 21/238 Interfacing the downstream path of the transmission network, e.g. adapting the transmission rate of a video stream to network bandwidth; Processing of multiplex streams hybrid fiber coaxial [HFC] networks for downstream channel allocation for video distribution H04L12/2801; flow control in packet networks H04L47/10; real-time communication protocols in data switching networks H04L65/00; scheduling or organising the servicing of application requests H04L67/60
• CPC Class Codes: H04N 21/4347 Demultiplexing of several video streams

Abstract

When two or more different video streams a e compressed for concurrent transmission of multiple compressed video bitstreams over a single shared communication channel, control over both (1) the transmission of data over the shared channel and (2) the compression processing that generates the bitstreams is exercised taking into account the differing levels of latency required for the corresponding video applications. For example, interactive video games typically require lower latency than other video applications such as video streaming, web browsing, and electronic mail. A multiplexer and traffic controller takes these differing latency requirements, along with bandwidth and image fidelity requirements, into account when controlling both traffic flow and compression processing. In addition, an off-line profiling tool analyzes typical video applications off-line in order to generate profiles of different types of video applications that are then accessed in real-time by a call admission manager responsible to controlling the admission of new video application sessions as well as the assignment of admitted applications to specific available video encoders, which themselves may differ in video compression processing power as well as in the degree to which they allow external processors (like the multiplexer and traffic controller) to control their internal compression processing.

Claims

1. A method for controlling transmission over a shared communication channel of multiple compressed video bitstreams generated by a plurality of video encoders and corresponding to multiple video applications, comprising the steps of:

(a) receiving information for each compressed video bitstream wherein at least two of the video applications have different latency requirements;

(b) controlling the transmission of data from the multiple compressed video bitstreams over the shared communication channel taking into account the information for each compressed video bitstream and the latency requirement of each corresponding video application; and

(c) adaptively controlling compression processing of at least one of the video encoders taking into account the information for the corresponding compressed video bitstream and the latency requirement of the corresponding video application.

2. The invention of claim 1, wherein at least two of the video applications have different bandwidth requirements and steps (b) and (c) are both implemented taking into account the bandwidth requirement of one or more of the video applications.

3. The invention of claim 1, further comprising the step of (d) controlling admission of a new video application for transmission of a corresponding compressed video bitstream over the shared communication channel.

4. The invention of claim 3, wherein step (d) comprises the steps of:

(1) receiving a classification for the new video application;

(2) accessing results from off-line profiling of typical video streams corresponding to the classification for the new video application; and

(3) determining whether to admit the new video application based on the off-line profiling results.

5. The invention of claim 4, wherein step (d) further comprises the step of (4) assigning the new video application to an appropriate one of a set of available video encoders, wherein at least two of the available video encoders have different video compression capabilities.

6. The invention of claim 5, wherein the different video compression capabilities include different levels of external control over video compression processing in step (c).

7. The invention of claim 5, wherein the different video compression capabilities include different levels of video compression processing power.

8. The invention of claim 4, wherein:

each video applications is categorized as being either;

a C1 application having relatively high bandwidth and relatively low latency requirements;

a C2 application having relatively intermediate bandwidth and relatively intermediate latency requirements;

or

a C3 application having relatively high latency requirements; and

step (d)(3) comprises the step of admitting the new application if and only if both of the following two rules would be satisfied after admitting the new video application;

(i) a sum of peak bandwidths for all C1 applications+a sum of the average bandwidths for all C2 applications is less than a total bandwidth of the shared communication channel;

(ii) a sum of average bit rates of all applications is less than the total bandwidth of the shared communication channel; and

(iii)encoding resources are available for the new application.

9. The invention of claim 4, wherein:

each video applications is categorized as being either;

a C1 application having relatively high bandwidth and relatively low latency requirements;

a C2 application having relatively intermediate bandwidth and relatively intermediate latency requirements;

or

a C3 application having relatively high latency requirements; and

step (d)(3) comprises the step of admitting the new application if and only if both of the following two rules would be satisfied after admitting the new video application;

(i) maximum of a sum of peak bandwidths of concurrent I frames possible at a time based on GOP structures for C1 applications+a sum of average bandwidths of all C2 and C3 applications is less than a total bandwidth of the shared communication channel;

(ii) a sum of average bit rates of all applications is less than the total bandwidth of the shared communication channel; and

(iii)encoding resources are available for the new application.

10. The invention of claim 1, wherein at least two of the video encoders have different video compression capabilities.

11. The invention of claim 10, wherein the different video compression capabilities include different levels of external control over video compression processing in step (c).

12. The invention of claim 10, wherein the different video compression capabilities include different levels of video compression processing power.

13. The invention of claim 1, wherein the processing of at least two of the video encoders is staggered within a frame time and step (c) comprise the step of controlling the process of encoding at least one compressed video bitstream taking into account the information for at least one other compressed video bitstream earlier in the same frame time.

14. The invention of claim 1, further comprising the step of (d) performing off-line profiling of typical video streams corresponding to different classifications of video applications to generate profiling results for use during at least one of steps (b) and (c).

15. The invention of claim 14, wherein step (d) comprises the step of characterizing a level of interactivity for each typical video stream.

16. The invention of claim 14, wherein step (d) comprises the step of characterizing a desired level of video compression processing power for each typical video stream.

17. The invention of claim 16, wherein step (d) further comprises the steps of (1) identifying a class of video encoders for each typical video stream based on the desired level of video compression processing power and (2) characterizing a level of external control provided by the identified class of video encoders.

18. The invention of claim 14, wherein the profiling results are used during step (b) to determine an acceptable level of buffering for at least one video application.

19. The invention of claim 14, wherein the profiling results are used during step (b) to order packets of data from different video applications.

20. The invention of claim 1, wherein step (b) comprises the step of dropping data from a compressed video bitstream.

21. The invention of claim 20, wherein step (c) comprises the step of instructing a corresponding video encoder to take into account the data dropping for subsequent compression processing.

22. The invention of claim 21, wherein the corresponding video encoder retains a previous reference frame to take into account the data dropping during the subsequent compression processing.

23. The invention of claim 20, wherein step (b) further comprises the step of inserting skip codes into the compressed video bitstream in place of the dropped data.

24. The invention of claim 1, wherein step (b) comprises the steps of:

(1) delaying transmission of one or more frames from a compressed video bitstream during periods of high channel bandwidth usage; and

(2) dropping one or more P frames before the next I frame to re-acquire a desirable latency level.

25. The invention of claim 1, wherein step (c) comprises the step of encoding one or more frames on demand for a video application with a relatively high latency requirement.

26. The invention of claim 1, wherein step (b) comprises the step of scheduling transmission of compressed data corresponding to a video application having a relatively high latency requirement to coincide with relatively low-bit-rate periods of one or more other video applications having a relatively low latency requirement.

27. The invention of claim 26, wherein step (c) comprises the step of encoding frames on demand for a video application having a relatively high latency requirement to achieve the scheduling of step (b).

28. The invention of claim 27, wherein step (c) further comprises the step of requesting frame types for one or more video applications.

29. The invention of claim 1, wherein step (c) comprises the step of staggering I frames between different video applications over different frame times.

30. The invention of claim 1, wherein step (c) comprises the step of instructing compression of a video application to include an adaptive refresh strategy in which intra slices are spread over multiple frames to reduce frequency of bit-rate peaks associated with I frames.

31. The invention of claim 1, wherein the latency requirement for a video application can vary over time and step (c) takes the varying latency requirement into account.

32. The invention of claim 1, wherein step (c) comprises the step of performing advance bit allocation for one or more of the video applications.

33. The invention of claim 1, wherein step (c) comprises the step of changing spatial resolution of a subsequent frame for compression processing of at least one of the video applications.

34. The invention of claim 1, wherein step (c) comprises the step of changing frame rate for compression processing of at least one of the video applications.

35. The invention of claim 1, wherein step (c) comprises the step of controlling compression processing of at least one of the video applications based on advance information acquired from the video application.

36. The invention of claim 1, further comprising the step of instructing at least one of the video applications to delay processing when the latency requirement for the video application is not met.

37. The invention of claim 1, wherein step (b) comprises the step of transmitting data from at least one of the compressed video bitstreams based on region-of-interest information for the corresponding video application in order to prioritize data within a frame of the compressed video bitstream.

38. The invention of claim 37, wherein step (b) comprises the step of transmitting pre-encoded data instead of the compressed video data for at least one region of the frame in the compressed video bitstream.

39. A video processing system for controlling transmission of multiple compressed video bitstreams corresponding to multiple video applications over a shared communication channel, comprising:

(a) a plurality of video encoders, each configured to generate a different compressed video bitstream for a different video application, wherein at least two of the video applications have different latency requirements; and

(b) a controller, configured to;

(1) receive the compressed video bitstreams from the video encoders;

(2) control the transmission of data from the multiple compressed video bitstreams over the shared communication channel taking into account information for each compressed video bitstream and the latency requirement of each corresponding video application; and

(3) adaptively control the compression processing of at least one of the video encoders taking into account the information for the corresponding compressed video bitstream and the latency requirement of the corresponding video application.

40. A controller for controlling transmission of multiple compressed video bitstreams corresponding to multiple video applications over a shared communication channel, in a video processing system further comprising a plurality of video encoders, each configured to generate a different compressed video bitstream for a different video application, wherein at least two of the video applications have different latency requirements, wherein the controller is configured to:

(1) receive the compressed video bitstreams from the video encoders;

(2) control the transmission of data from the multiple compressed video bitstreams over the shared communication channel taking into account information for each compressed video bitstream and the latency requirement of each corresponding video application; and

(3) adaptively control the compression processing of at least one of the video encoders taking into account the information for the corresponding compressed video bitstream and the latency requirement of the corresponding video application.