Distributed control strategy for dynamically encoding multiple streams of video data in parallel for multiplexing onto a constant bit rate channel

• US 7,085,322 B2
• Filed: 01/18/2002
• Issued: 08/01/2006
• Est. Priority Date: 05/29/1998
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Ngai, Agnes Y, Westermann, Edward F
• Primary Examiner(s): RAO, ANAND SHASHIKANT
• Application Number:
  US10/051,004
• Publication Number:
  US 20020094031A1
• Time in Patent Office: 1,656 Days
• Field of Search: 37524001-24029
• US Class Current: 375/240.25
• CPC Class Codes: H04N 19/114 Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames H04N19/107 takes precedence
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/15 by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 21/23655 Statistical multiplexing, e.g. by controlling the encoder to alter its bitrate to optimize the bandwidth utilization

Abstract

A distributed control strategy is provided for dynamically encoding multiple streams of video data in parallel for multiplexing onto a constant bit rate channel. The control strategy is a single pass strategy which allows individual encode bit rates to be dynamically adjusted for each video data stream based in part on relative complexity of the multiple streams of video data, as well as fullness of compressed video data buffers coupled between the encoders and the constant bit rate channel. The control strategy includes encoding the multiple streams of video frames in parallel using multiple encoders, exchanging one or more statistics between the encode processes using an exchange interface, and dynamically adapting encoding of at least one stream of the video frames using the exchanged statistics based on relative complexity of the video frames.

Claims

1. A method of processing multiple streams of video frames comprising:

employing multiple encode processes to encode multiple streams of video frames in parallel;

directly exchanging encode process to encode process at least one input statistic or encode statistic employed by each encode process of the multiple encode processes;

dynamically adapting encoding of at least one stream of video frames of the multiple streams of video frames based on relative complexity of the video frames comprising the multiple streams of video frames employing the at least one input statistic or encode statistic directly exchanged between the encode processes; and

wherein said dynamically adapting comprising employing a joint rate control strategy distributed among said multiple encode processes, and wherein each encode process of said multiple encode processes ascertains said at least one input statistic or encode statistic for its respective stream of video frames being encoded, saves said at least one input statistic or encode statistic, and shares said at least one input statistic or encode statistic directly with said multiple encode processes during said exchanging to allow determination by at least one encode process of the multiple encode processes of a total statistic corresponding to a sum of the at least one input statistics or encode statistics generated by said multiple encode processes, and wherein said directly exchanging further comprises sharing the total statistic by the at least one encode process with the multiple encode processes, and responsive to obtaining of the total statistic, at least one encode process employs the total statistic and its own at least one input statistic or encode statistic during said dynamically adapting encoding to facilitate said dynamically adapting encoding of the at least one stream of video frames of the multiple streams of video frames.

2. The method of claim 1, further comprising providing an exchange interface within the multiple encode processes, wherein said exchange interface facilitates said directly exchanging encode process to encode process of said at least one input statistic or encode statistic.

3. The method of claim 2, wherein said employing comprises employing multiple encoders connected in parallel, each encoder implementing a respective encode process of said multiple encode processes, and wherein said providing the exchange interface includes providing dedicated data and control buses directly connecting the multiple encoders for facilitating said direct exchanging encode process to encode process of the at least one input statistic or encode statistic.

4. The method of claim 1, further comprising multiplexing resultant compressed video streams output by said multiple encode processes onto a constant bit rate channel.

5. The method of claim 4, wherein said distributed rate control strategy facilitates partitioning of said constant bit rate channel capacity among said multiple encode processes.

6. The method of claim 5, further comprising buffering output of each encode process of said multiple encode processes in a respective encode buffer prior to said multiplexing, and allowing expanded upper and lower bounds in at least one encode buffer when a new bit budget is allocated in its respective encode process, wherein an amount of data unloaded by each encode process from its respective encode buffer is a function of a total output of said multiple encode processes at any instance in time and said constant bit rate channel capacity.

7. The method of claim 1, wherein each encode process of said multiple encode processes employs the total statistic and its own at least one input statistic or encode statistic during said dynamically adapting encoding to dynamically adapt encoding of each stream of video frames.

8. The method of claim 7, wherein said dynamically adapting encoding comprises selecting at each encode process a bit rate of a compressed video stream output therefrom, said bit rate being selected using the total statistic and its own at least one input statistic or encode statistic.

9. The method of claim 1, wherein said dynamically adapting encoding comprises modifying a bit rate employed by at least one encode process of said multiple encode processes responsive to said at least one input statistic or encode statistic exchanged between the encode processes.

10. The method of claim 9, wherein said modifying the bit rate comprises modifying the bit rate of the at least one encode process at a group of picture boundary or scene change in the stream of video frames being encoded by said at least one encode process.

11. The method of claim 9, wherein said modifying a bit rate of the at least one encode process comprises modifying the bit rate of the at least one encode process at a picture boundary of the stream of video frames being encoded by said at least one encode process.

12. The method of claim 1, wherein said at least one input statistic or encode statistic comprises at least one of an inter-pixel difference statistic, a frame activity statistic, an encode bit count or a quantization level statistic.

13. A method of processing multiple streams of video frames comprising:

encoding the multiple streams of video frames in parallel employing multiple encode processes, the encoding including employing at least one controllable encode parameter for each stream of video frames;

during the encoding, analyzing each stream of video frames to derive information on at least one characteristic thereof, the at least one characteristic comprising at least one of an intraframe characteristic or an interframe characteristic;

directly exchanging encode process to encode process the information derived on the at least one characteristic of each stream of video frames;

dynamically adapting encoding of at least one stream of video frames using relative information on the at least one characteristic obtained from each stream of video frames of the multiple streams of video frames, the dynamically adapting including for the at least one stream of video frames adjusting the at least one controllable encode parameter employed in the encoding, wherein the encoding of each stream of video frames is dynamically adaptive to relative changes in the at least one characteristic obtained from the multiple streams of video frames by directly exchanging encode process to encode process the information on the at least one characteristic of the multiple streams of video frames; and

wherein said dynamically adapting comprising employing a joint rate control strategy distributed among said multiple encode processes, and wherein each encode process of said multiple encode processes ascertains said at least one input statistic or encode statistic for its respective stream of video frames being encoded, saves said at least one input statistic or encode statistic, and shares said at least one input statistic or encode statistic directly with said multiple encode processes during said exchanging to allow determination by at least one encode process of the multiple encode processes of a total statistic corresponding to a sum of the at least one input statistics or encode statistics generated by said multiple encode processes, and wherein said directly exchanging further comprises sharing the total statistic by the at least one encode process with the multiple encode processes, and responsive to obtaining of the total statistic, at least one encode process employs the total statistic and its own at least one input statistic or encode statistic during said dynamically adapting encoding to facilitate said dynamically adapting encoding of the at least one stream of video frames of the multiple streams of video frames.

14. The method of claim 13, further comprising providing an exchange interface within the multiple encode processes, wherein said exchange interface facilitates said directly exchanging encode process to encode process of said information derived on the at least one characteristic of each stream of video frames.

15. The method of claim 13, further comprising multiplexing resultant compressed video streams output by said multiple encode processes onto a constant bit rate channel, and wherein said distributed rate control strategy facilitates partitioning of said constant bit rate channel capacity among said multiple encode processes.

16. The method of claim 13, wherein said at least one controllable encode parameter comprises a bit rate employed by at least one encode process of said multiple encode processes.

17. A system for processing multiple streams of video frames comprising:

multiple encoders for encoding the multiple streams of video frames in parallel, each encoder receiving one stream of video frames of said multiple streams of video frames;

a direct exchange interface providing encoder to encoder communications paths between said multiple encoders and allowing said multiple encoders to directly exchange the at least one input statistic or encode statistic therebetween;

a distributed joint rate control means integrated within said multiple encoders for dynamically adapting encoding of at least one stream of video frames of the multiple streams of video frames based on relative complexity of the video frames comprising the multiple streams of video frames; and

wherein each encoder ascertains said at least one input statistic or encode statistic for its respective stream of video frames being encoded, saves said at least one input statistic or encode statistic, and shares said at least one input statistic or encode statistic among said multiple encoders using said direct exchange interface to allow determination by at least one encoder of the multiple encoders of a total statistic corresponding to a sum of the at least one input statistics or encode statistics generated by said multiple encoders, and wherein said direct exchange interface facilitates encoder to encoder sharing of the total statistic, wherein at least one encoder employs the total statistic and its own at least one input statistic or encode statistic to dynamically adapt encoding of the at least one stream of video frames of the multiple streams of video frames.

18. The system of claim 17, wherein the direct exchange interface includes dedicated data and control buses directly connecting the multiple encoders for facilitating encoder to encoder exchanging of the at least one input statistic or encode statistic.

19. The system of claim 17, further comprising a multiplexer for multiplexing resultant compressed video streams output by said multiple encoders onto a constant bit rate channel.

20. The system of claim 19, wherein said distributed rate control strategy facilitates partitioning of said constant bit rate channel capacity among said multiple encoders.

21. The system of claim 20, further comprising means for buffering output of each encoder of said multiple encoders in a respective encode buffer prior to said multiplexer, and for allowing expanded upper and lower bounds in at least one encode buffer when a new bit budget is allocated in its respective encoder, wherein an amount of data unloaded by each encoder from its respective encode buffer is a function of a total output of said multiple encoders at any instance in time and said constant bit rate channel capacity.

22. The system of claim 17, wherein each encoder of said multiple encoders employs the total statistic and its own at least one input statistic or encode statistic to dynamically adapt encoding of its stream of video frames.

23. The system of claim 22, wherein said dynamically adapting encoding comprises selecting at each encoder a bit rate of a compressed video stream output therefrom, said bit rate being selected using the total statistic and its own at least one input statistic or encode statistic.

24. The system of claim 17, wherein said dynamically adapting encoding comprises modifying a bit rate employed by at least one encoder of said multiple encoders responsive to said at least one input statistic or encode statistic exchanged between the encode processes.

25. The system of claim 24, wherein said modifying the bit rate comprises modifying the bit rate of the at least one encoder at a group of picture boundary or scene change in the stream of video frames being encoded by said at least one encoder.

26. The system of claim 24, wherein said modifying the bit rate of the at least one encoder comprises modifying the bit rate of the at least one encoder at a picture boundary of the stream of video frames being encoded by said at least one encoder.

27. The system of claim 17, wherein said at least one input statistic or encode statistic comprises at least one of an inter-pixel difference statistic, a frame activity statistic, an encode bit count or a quantization level statistic.

28. A system for processing multiple streams of video frames comprising:

multiple encoders for encoding multiple streams of video frames in parallel, each encoder employing at least one controllable encode parameter for encoding one stream of the video frames;

means for analyzing each stream of video frames to derive information on at least one characteristic thereof, said at least one characteristic comprising at least one of an intraframe characteristic or an interframe characteristic;

a direct exchange interface comprising an encoder to encoder communications bus interconnecting said multiple encoders and allowing said encoders to directly exchange information derived on the at least one characteristic of each stream of video frames;

means for dynamically adapting encoding of at least one stream of video frames using relative information on the at least one characteristic obtained from each stream of video frames of the multiple streams of video frames, the dynamically adapting including for the at least one stream of video frames adjusting the at least one controllable encode parameter employed in the encoding, wherein the encoding of each stream of video frames is dynamically adaptive to relative changes in the at least one characteristic obtained from the multiple streams of video frames by directly exchanging encoder to encoder the information on the at least one characteristic of the multiple streams of video frames, and wherein the means for dynamically adapting encoding comprises a distributed means integrated within each of the multiple encoders; and

wherein said means for dynamically adapting encoding comprises means for employing a joint rate control strategy distributed within said multiple encoders, and wherein each encoder of said multiple encoders ascertains said state information derived on the at least one characteristic of its respective stream of video frames, saves said information, and shares said information among said multiple encoders using said exchange interface to allow determination by at least one encoder of the multiple encoders of a total statistic corresponding to a sum of the information derived on the at least one characteristic by each encoder of the multiple encoders, and wherein said direct exchange interface further comprises means for sharing the total statistic among the multiple encoders, wherein at least one encoder employs the total statistic and its own information during the dynamically adapting encoding to facilitate said dynamically encoding of the at least one stream of video frames of the multiple streams of video frames.

29. The system of claim 28, further comprising means for multiplexing resultant compressed video streams output by said multiple encoders onto a constant bit rate channel, and wherein said distributed rate control strategy facilitates partitioning of said constant bit rate channel capacity among said multiple encoders.

30. The system of claim 28, wherein said at least one controllable encode parameter comprises a bit rate employed by at least one encoder of said multiple encoders.

31. At least one program storage device readable by a machine, tangibly embodying at least one program of instructions executable by the machine to perform a method of processing multiple streams of video frames, comprising:

employing multiple encode processes to encode multiple streams of video frames in parallel;

directly exchanging encode process to encode process at least one input statistic or encode statistic employed by each encode process of the multiple encode processes;

dynamically adapting encoding of at least one stream of video frames of the multiple streams of video frames based on relative complexity of the video frames comprising the multiple streams of video frames employing the at least one input statistic or encode statistic directly exchanged between the encode processes; and

wherein said dynamically adapting comprising employing a joint rate control strategy distributed among said multiple encode processes, and wherein each encode process of said multiple encode processes ascertains said at least one input statistic or encode statistic for its respective stream of video frames being encoded, saves said at least one input statistic or encode statistic, and shares said at least one input statistic or encode statistic directly with said multiple encode processes during said exchanging to allow determination by at least one encode process of the multiple encode processes of a total statistic corresponding to a sum of the at least one input statistics or encode statistics generated by said multiple encode processes, and wherein said directly exchanging further comprises sharing the total statistic by the at least one encode process with the multiple encode processes, and responsive to obtaining of the total statistic, at least one encode process employs the total statistic and its own at least one input statistic or encode statistic during said dynamically adapting encoding to facilitate said dynamically adapting encoding of the at least one stream of video frames of the multiple streams of video frames.

32. The at least one program storage device of claim 31, further comprising providing an exchange interface within the multiple encode processes, wherein said exchange interface facilitates said directly exchanging encode process to encode process of said at least one input statistic or encode statistic.

33. The at least one program storage device of claim 32, further comprising multiplexing resultant compressed video streams output by said multiple encode processes onto a constant bit rate channel.

34. The at least one program storage device of claim 33, wherein said distributed rate control strategy facilitates partitioning of said constant bit rate channel capacity among said multiple encode processes.

35. The at least one program storage device of claim 34, further comprising buffering output of each encode process of said multiple encode processes in a respective encode buffer prior to said multiplexing, and allowing expanded upper and lower bounds in at least one encode buffer when a new bit budget is allocated in its respective encode process, wherein an amount of data unloaded by each encode process from its respective encode buffer is a function of a total output of said multiple encode processes at any instance in time and said constant bit rate channel capacity.

36. The at least one program storage device of claim 31, wherein each encode process of said multiple encode processes employs the total statistic and its own at least one input statistic or encode statistic during said dynamically adapting encoding to dynamically adapt encoding of each stream of video frames.

37. The at least one program storage device of claim 36, wherein said dynamically adapting encoding comprises selecting at each encode process a bit rate of a compressed video stream output therefrom, said bit rate being selected using the total statistic and its own at least one input statistic or encode statistic.

38. The at least one program storage device of claim 31, wherein said dynamically adapting encoding comprises modifying a bit rate employed by at least one encode process of said multiple encode processes responsive to said at least one input statistic or encode statistic exchanged between the encode processes.

39. The at least one program storage device of claim 38, wherein said modifying the bit rate comprises modifying the bit rate of the at least one encode process at a group of picture boundary or scene change in the stream of video frames being encoded by said at least one encode process.

40. The at least one program storage device of claim 38, wherein said modifying a bit rate of the at least one encode process comprises modifying the bit rate of the at least one encode process at a picture boundary of the stream of video frames being encoded by said at least one encode process.

41. The at least one program storage device of claim 31, wherein said at least one input statistic or encode statistic comprises at least one of an inter-pixel difference statistic, a frame activity statistic, an encode bit count or a quantization level statistic.

42. At least one program storage device readable by a machine, tangibly embodying at least one program of instructions executable by the machine to perform a method of processing multiple streams of video frames, comprising:

encoding the multiple streams of video frames in parallel employing multiple encode processes, the encoding including employing at least one controllable encode parameter for each stream of video frames;

during the encoding, analyzing each stream of video frames to derive information on at least one characteristic thereof, the at least one characteristic comprising at least one of an intraframe characteristic or an interframe characteristic;

directly exchanging encode process to encode process the information derived on the at least one characteristic of each stream of video frames;

dynamically adapting encoding of at least one stream of video frames using relative information on the at least one characteristic obtained from each stream of video frames of the multiple streams of video frames, the dynamically adapting including for the at least one stream of video frames adjusting the at least one controllable encode parameter employed in the encoding, wherein the encoding of each stream of video frames is dynamically adaptive to relative changes in the at least one characteristic obtained from the multiple streams of video frames by directly exchanging encode process to encode process the information on the at least one characteristic of the multiple streams of video frames between the encode processes; and

wherein said dynamically adapting encoding comprises employing a joint rate control strategy distributed among said multiple encode processes, and wherein each encode process of said multiple encode processes ascertains said information derived on the at least one characteristic of its respective stream of video frames, saves said information, and shares said information among said multiple encode processes during said exchanging to allow determination by at least one encode process of said multiple encode processes of a total statistic corresponding to a sum of the information derived on the at least one characteristic by each encode process of the multiple encode processes, and wherein said directly exchanging further comprises sharing the total statistic among the multiple encode processes, wherein at least one encode process employs the total statistic and its own information during said dynamically adapting encoding to facilitate said dynamically encoding of the at least one stream of video frame of the multiple streams of video frames.

43. The at least one program storage device of claim 42, further comprising providing an exchange interface within the multiple encode processes, wherein said exchange interface facilitates said directly exchanging encode process to encode process of said information derived on the at least one characteristic of each stream of video frames.

44. The at least one program storage device of claim 42, further comprising multiplexing resultant compressed video streams output by said multiple encode processes onto a constant bit rate channel, and wherein said distributed rate control strategy facilitates partitioning of said constant bit rate channel capacity among said multiple encode processes.

45. The at least one program storage device of claim 42, wherein said at least one controllable encode parameter comprises a bit rate employed by at least one encode process of said multiple encode processes.