Adaptively encoding multiple streams of video data in parallel for multiplexing onto a constant bit rate channel

• US 6,859,496 B1
• Filed: 01/05/1999
• Issued: 02/22/2005
• Est. Priority Date: 05/29/1998
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Boroczky, Lilla, Westermann, Edward F., Ngai, Agnes Y.
• Primary Examiner(s): WONG, ALLEN C
• Application Number:
  US09/225,193
• Time in Patent Office: 2,240 Days
• Field of Search: 375/240, 375/240.26, 375/240.01
• US Class Current: 375/240.26
• CPC Class Codes: H04N 19/114 Adapting the group of pictures [GOP] structure, e.g. number of B-frames between two anchor frames H04N19/107 takes precedence
• CPC Class Codes: H04N 19/124 Quantisation
• CPC Class Codes: H04N 19/142 Detection of scene cut or scene change
• CPC Class Codes: H04N 19/15 by monitoring actual compressed data size at the memory before deciding storage at the transmission buffer
• CPC Class Codes: H04N 19/152 by measuring the fullness of the transmission buffer
• CPC Class Codes: H04N 19/177 the unit being a group of pictures [GOP]
• CPC Class Codes: H04N 19/179 the unit being a scene or a shot
• CPC Class Codes: H04N 19/436 using parallelised computational arrangements
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 19/87 involving scene cut or scene change detection in combination with video compression
• CPC Class Codes: H04N 21/23655 Statistical multiplexing, e.g. by controlling the encoder to alter its bitrate to optimize the bandwidth utilization

Abstract

A control strategy is provided for dynamically encoding multiple streams of video data in parallel for multiplexing onto a constant bit rate channel. The control strategy allows individual encode bit rates to be dynamically adjusted for each video data stream based in part on relative complexity of the multiple streams of video data, as well as fullness of compressed video data buffers and a channel buffer coupled between the encoders and the constant bit rate channel. The control strategy includes analyzing the multiple streams of video to determine relative complexity thereof, encoding the multiple streams of video frames in parallel, and dynamically adapting encoding of at least one stream of the video frames based on the relative complexity of the video frames. The bit rate for each stream of video frames is only changed at GOP boundaries, or if a scene change occurs. The calculated bit rate is preferably further modified based upon buffer fullness.

Claims

1. A method for processing multiple streams of video frames comprising:

analyzing said multiple streams of video frames to determine relative complexity of the video frames comprising said multiple streams of video frames, wherein said analyzing comprises obtaining picture statistics on said relative complexity of said video frames prior to encoding of said multiple streams of video frames;

encoding said multiple streams of video frames in parallel using said relative complexity of the video frames, wherein said encoding comprises adaptively encoding said multiple streams of video frames with change in said relative complexity;

during said encoding, analyzing each stream of video frames to derive coding statistics on at least one characteristic thereof, said coding statistics comprising at least one of an intraframe characteristic or an interframe characteristic, and wherein said adaptively encoding comprises employing both said picture statistics and said coding statistics in adaptively encoding said multiple streams of video frames; and

wherein said adaptively encoding comprises adjusting at least one encode bit rate employed by said encoding to encode said multiple streams of video frames with change in said relative complexity, and wherein said adjusting at least one encode bit rate for at least one stream of said multiple streams of video frames only occurs upon beginning of a new group of frames (GOP) within said at least one stream, wherein for said at least one stream of video frames said encode bit rate is constant within each GOP and is variable between GOPs.

2. The method of claim 1, wherein said adaptively encoding employing both said picture statistics and said coding statistics comprises weighting said picture statistics and said coding statistics by predefined amounts in adjusting the at least one encode bit rate employed by said encoding to encode said multiple streams of video frames.

3. The method of claim 2, wherein said weighting comprises weighting said picture statistics 50% and said coding statistics 50% in adjusting said at least one encode bit rate based on said relative complexity of said multiple streams of video frames.

4. The method of claim 1, further comprising performing said adaptively encoding of said multiple streams of video frames upon detection of a scene change in one stream of said multiple streams of video frames, wherein said encoding comprises beginning a new GOP upon detection of the scene change.

5. The method of claim 1, wherein said analyzing comprises preprocessing said multiple streams of video frames to produce an average frame activity for each video frame of said video frames comprising said multiple streams of video frames, and wherein said encoding comprises employing said average frame activities as representative of said relative complexity of the video frames.

6. The method of claim 1, wherein said adaptively encoding comprises adjusting the at least one encode bit rate employed by said encoding based on said picture statistics and said coding statistics of said multiple streams of video frames, and wherein said encoding produces multiple compressed video streams, and said method further comprises buffering each compressed video stream in a buffer, and wherein said adaptively encoding includes for each stream of video frames modifying said at least one encode bit rate for each stream based on fullness of said buffer receiving said compressed video stream.

7. The method of claim 6, further comprising multiplexing said multiple compressed video streams onto a constant bit rate channel.

8. The method of claim 7, further comprising predefining guard bands at upper and lower limits of said buffer and modifying said at least one encode bit rate whenever buffer fullness is within one of said upper and lower guard bands of the buffer.

9. The method of claim 1, wherein said adaptively encoding comprises employing a rate control algorithm to control an encode bit rate of each stream of said multiple streams of video frames, and wherein said method comprises providing said picture statistics and said coding statistics to said rate control algorithm, said picture statistics and said coding statistics being representative of said relative complexity of the video frames comprising said multiple streams of video frames to be encoded.

10. A method for processing multiple streams of video frames comprising:

analyzing said multiple streams of video frames to derive picture statistics on at least one characteristic thereof, said at least one characteristic comprising at least one of an intraframe characteristic or an interframe characteristic;

encoding said multiple streams of video frames in parallel, said encoding including employing at least one controllable encode parameter for each stream of video frames;

adapting said encoding of said multiple streams of video frames using relative information from said picture statistics obtained from said multiple streams of video frames, said adapting including adjusting said at least one controllable encode parameter employed in said encoding of at least one stream of video frames, wherein said encoding of said multiple streams of video frames is adaptive to relative changes in said at least one characteristic;

during said encoding, analyzing each stream of video frames to derive coding statistics on at least one coding characteristic, said at least one coding characteristic comprising at least one of an intraframe coding characteristic or an interframe coding characteristic, and wherein said adapting comprises employing said coding statistics in combination with said picture statistics in adapting said encoding of said at least one stream of video frames; and

wherein said at least one controllable encode parameter comprises at least one encode bit rate, and wherein said adapting said encoding for at least one stream only occurs upon beginning of a new group of frames (GOP) within said at least one stream, wherein for said at least one stream of video frames said encode bit rate is constant within each GOP and is variable between GOPs.

11. The method of claim 10, wherein said picture statistics are representative of an average frame activity, and wherein said coding statistics comprise at least one of bits used, an average MQUANT, or buffer fullness during encoding of said streams of video frames.

12. The method of claim 10, wherein said adapting comprises employing both said picture statistics and said coding statistics in adjusting said at least one encode bit rate employed in encoding said at least one stream of video frames.

13. The method of claim 12, further comprising employing said picture statistics and said coding statistics in a predefined relation in adjusting said at least one encode bit rate employed in said encoding of said at least one stream of video frames.

14. The method of claim 12, wherein said at least one controllable encode parameter comprises an encode bit rate for each stream of video frames.

15. The method of claim 10, wherein said encoding produces multiple compressed video streams, and wherein said method further comprises multiplexing each compressed video stream onto a constant bit rate channel.

16. The method of claim 15, wherein said method further comprises buffering each compressed video stream prior to said multiplexing, and wherein said analyzing of each stream of video frames to derive said coding statistics comprises deriving at least one of statistics on bits used, average MQUANT, buffer fullness or scene change.

17. The method of claim 10, further comprising employing only said picture statistics by said adapting when starting encoding of said multiple streams of video frames and upon detection of a scene change in one of said multiple streams of video frames, wherein said encoding comprises beginning a new GOP upon detection of the scene change.

18. A system for processing multiple streams of video frames comprising:

means for analyzing said multiple streams of video frames to determine picture statistics representative of relative complexity of the video frames comprising said multiple streams of video frames;

multiple encoders for encoding said multiple streams of video frames in parallel, each encoder receiving one stream of video frames of said multiple streams of video frames, said multiple encoders each comprising means for generating coding statistics on at least one characteristic of said stream of video frames being encoded thereby, said coding statistics comprising at least one of an intraframe characteristic or an interframe characteristic;

a joint controller coupled to each encoder of said multiple encoders and to said means for analyzing for dynamically adapting encoding of at least one stream of video frames of the multiple streams of video frames based on said picture statistics and said coding statistics derived from the video frames comprising the multiple streams of video frames; and

wherein the joint controller comprises means for adjusting at least one encode bit rate employed by at least one encoder of said multiple encoders, wherein said adjusting only occurs upon beginning of a new group of frames (GOP) within said at least one stream, wherein for said at least one stream of video frames said encode bit rate is constant within each GOP and is variable between GOPs.

19. The system of claim 18, wherein said means for analyzing comprises multiple preprocessors coupled to receive said multiple streams of video frames, each preprocessor receiving a different stream of video frames, and wherein said multiple preprocessors output said picture statistics, said picture statistics providing information on said relative complexity of the video frames comprising said multiple streams of video frames, and being provided to said joint controller for dynamically adapting said encoding of said at least one stream of video frames.

20. The system of claim 18, wherein said joint controller comprises means for employing only said picture statistics when starting encoding of said multiple streams of video frames and upon detection of a scene change in said at least one stream of video frames.

21. The system of claim 18, wherein said joint controller comprises means for weighting said picture statistics and said coding statistics by predefined amounts.

22. The system of claim 18, wherein said picture statistics comprise an average frame activity for a video frame of said video frames comprising said multiple streams of video frames to be encoded, and wherein said coding statistics comprise at least one of bits used, an average MQUANT or buffer fullness during encoding of a video frame of said video frames comprising said multiple streams of video frames.

23. A system for processing multiple streams of video frames comprising:

means for analyzing said multiple streams of video frames to derive picture statistics on at least one characteristic thereof, said at least one characteristic comprising at least one of an intraframe characteristic or an interframe characteristic;

multiple encoders for encoding said multiple streams of video frames in parallel, each encoder receiving one stream of video frames of said multiple streams of video frames, and each encoder employing an encode bit rate in encoding its stream of video frames, wherein said multiple encoders further comprises coding means for analyzing each stream of video frames during said encoding to derive coding statistics on at least one coding characteristic, said at least one coding characteristic comprising at least one of an intraframe coding characteristic or an interframe coding characteristic; and

a joint controller coupled to each encoder of said multiple encoders and to said means for analyzing for dynamically adapting encoding of at least one stream of video frames of the multiple streams of video frames based on relative information derived from said picture statistics and said coding statistics, said adapting including adjusting said encode bit rate employed in said encoding of at least one stream of video frames, wherein said adjusting only occurs upon beginning of a new group of frames (GOP) within said at least one stream of video frames, wherein for said at least one stream said encode bit rate is constant within each GOP and is variable between GOPs.

24. The system of claim 23, wherein said picture statistics are representative of an average frame activity, and wherein said coding statistics comprise at least one of bits used, an average MQUANT, or buffer fullness during encoding of said streams of video frames.

25. The system of claim 23, wherein said joint controller comprises means for employing both said picture statistics and said coding statistics in adjusting said at least one encode bit rate employed in encoding said at least one stream of video frames.

26. The system of claim 23, wherein said joint controller comprises means for employing said picture statistics and said coding statistics in a predefined relation in adjusting said at least one encode bit rate employed in said encoding of said at least one stream of video frames.

27. The system of claim 23 wherein said multiple encoders produce multiple compressed video streams, and wherein said system further comprises a multiplexer for multiplexing said compressed video streams onto a constant bit rate channel.

28. They system of claim 27, wherein said system further comprises means for buffering each compressed video stream prior to said multiplexing, and wherein said coding means to derive said coding statistics comprises means for deriving at least one of statistics on bits used, average MQUANT, buffer fullness or scene change.

29. An article of manufacture, comprising:

at least one computer usable medium having computer readable program code means embodied therein for causing the processing of multiple streams of video frames, the computer readable program code means in the article of manufacture comprising;

(i) computer readable program code means for causing a computer to effect analyzing the multiple streams of video frames to determine relative complexity of the video frames comprising the multiple streams of video frames, wherein said analyzing comprises obtaining picture statistics on said relative complexity of said video frames prior to encoding of said multiple streams of video frames;

(ii) computer readable program code means for causing a computer to effect encoding the multiple streams of video frames in parallel using the relative complexity of the video frames, wherein the encoding comprises adaptively encoding the multiple streams of video frames with a change in said relative complexity;

(iii) computer readable program code means for causing a computer to effect analyzing each stream of video frames during said encoding to derive coding statistics on at least one characteristic thereof, said coding statistics comprising at least one of an intraframe characteristic or an interframe characteristic, and wherein said computer readable program code means for causing a computer to effect adaptively encoding comprises computer readable program code means for causing a computer to effect employing both said picture statistics and said coding statistics in adaptively encoding said multiple streams of video frames; and

(iv) computer readable program code means for causing a computer to effect adjusting at least one encode bit rate employed by said encoding to encode said multiple streams of video frames, and wherein said adjusting the at least one encode bit rate for at least one stream of the multiple streams of video frames only occurs upon beginning of a new group of frames (GOP) within said at least one stream, wherein for said at least one stream of video frames said encode bit rate is constant within each GOP and is variable between GOPs.

30. The article of manufacture of claim 29, wherein said computer readable program code means for causing a computer to effect adaptively encoding employing both said picture statistics and said coding statistics comprises computer readable program code means for causing a computer to effect weighting said picture statistics and said coding statistics by predefined amounts in adjusting the at least one encode bit rate employed by said encoding to encode said multiple streams of video frames.

31. An article of manufacture, comprising:

at least one computer usable medium having computer readable program code means embodied therein for causing the processing of multiple streams of video frames, the computer readable program code means in the article of manufacture comprising;

(i) computer readable program code means for causing a computer to effect analyzing said multiple streams of video frames to derive picture statistics on at least one characteristic thereof, said at least one characteristic comprising at least one of an intraframe characteristic or an interframe characteristic;

(ii) computer readable program code means for causing a computer to effect encoding said multiple streams of video frames in parallel, said encoding including employing at least one controllable encode parameter for each stream of video frames;

(iii) computer readable program code means for causing a computer to effect adapting said encoding of said multiple streams of video frames using relative information from said picture statistics obtained from said multiple streams of video frames, said adapting including adjusting said at least one controllable encode parameter employed in said encoding of at least one stream of video frames, wherein said encoding of said multiple streams of video frames is adaptive to changes in said relative information;

(iv) computer readable program code means for causing a computer to effect analyzing each stream of video frames during said encoding to derive coding statistics on at least one coding characteristic, said at least one coding characteristic comprising at least one of an intraframe coding characteristic or an interframe coding characteristic, and wherein said computer readable program code means for causing a computer to effect adapting comprises computer readable program code means for causing a computer to effect employing said coding statistics in combination with said picture statistics in adapting said encoding of said at least one stream of video frames; and

(v) wherein said at least one controllable encode parameter comprises at least one encode bit rate, and wherein said adapting said encoding for at least one stream only occurs upon beginning of a new group of frames (GOP) within said at least one stream, wherein for said at least one stream of video frames said encode bit rate is constant within each GOP and is variable between GOPs.

32. The article of manufacture of claim 31, wherein said picture statistics are representative of an average frame activity, and said coding statistics comprise at least one of bits used, an average MQUANT, or buffer fullness during encoding of said streams of video frames.