Low delay real time digital video mixing for multipoint video conferencing

• US 6,285,661 B1
• Filed: 01/28/1998
• Issued: 09/04/2001
• Est. Priority Date: 01/28/1998
• Status: Expired due to Term

• Current Assignee: Polycom Incorporated (HP Inc.)
• Original Assignee: Picturetel Corporation (HP Inc.)
• Inventors: Kerofsky, Louis J., Zhu, Qin-Fan
• Primary Examiner(s): Nguyen, Chau
• Assistant Examiner(s): BOAKYE, ALEXANDER O
• Application Number:
  US09/014,942
• Time in Patent Office: 1,315 Days
• Field of Search: 370/260-263, 370/391, 370/505, 370/465, 370/468, 370/473, 370/474, 348/14, 348/15, 379/202
• US Class Current: 370/260
• CPC Class Codes: H04M 3/567 Multimedia conference systems
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A multipoint control unit (MCU) includes a plurality of segment processors arranged to extract segment data from a corresponding plurality of input video streams and a plurality of data queues, each one of the data queues being coupled to a corresponding one of the plurality of segment processors for storing segment data provided from the segment processors. The (MCU) also includes a data combiner coupled to the data queues for combining output data from the data queues and a controller coupled to the combiner for selectively providing data to the combiner in accordance with the amount of the data in the data queues. Several different approaches for determining what queues to obtain data from are described to reduce insertion delay Asymmetric picture rates are also used to reduce the video mixing delay. In addition, the multipoint control unit can optionally have the segment processors including a rate reducer and have the combiner controller send bit rate control information to the reducer to control the operation of the reducer.

Claims

1. A method of operating a multipoint control unit for use in videoconferencing of a plurality of audio-video terminals comprises:

feeding a plurality of video streams to the multipoint control unit;

for said video streams,

extracting segments of data;

outputting the segments of data to a data queue;

forming new pictures from the segments of data stored in each of the data queues by;

including the new picture data from those queues that have a complete picture;

and if no queue has a complete picture,

determining if for each of the queues whether the queues have a complete segment of data and for those queues having a complete segment of data including data from those queues in the new picture.

2. The method of claim 1 wherein forming new picture includes determining for each of the data queues which queues have a complete picture.

3. The method of claim 2 wherein for those quadrants not having a complete picture or a complete segment of data including in the current picture header bits for those quadrants.

4. The method as recited in claim 3 wherein the data from the queues corresponding to quadrants in the current picture and header bits for quadrants not in the current picture are combined into a single video stream that is fed to each of the audio-video terminals in the video conference.

5. A method of operating a multipoint control unit for use in videoconferencing of a plurality of audio-video terminals comprises:

feeding a plurality of video streams to the multipoint control unit;

for each one of said streams, extracting segments of data and adaptively reducing the bit rates;

outputting the segments of data to a data queue;

forming new picture data units from the segments of data stored in each of the data queues by;

including the new picture data from those queues that have a complete picture;

and if no queue has a complete picture,

determining if for each of the queues whether the queues have a complete segment of data and for those queues having a segment of data including data from those queues in the new picture.

6. The method of claim 5 wherein for those quadrants not having a complete picture or a complete segment, including in the current picture header bits for such quadrants.

7. The method as recited in claim 6 wherein the data from the queues corresponding to quadrants in the current picture and header bits for quadrants not in the current picture are combined into a single video stream that is fed to each of the audio-video terminals in the video conference.

8. The method of claim 7 wherein the determining further includes:

assigning a threshold in accordance with a number of complete segments of data in the queue.

9. The method of claim 8 wherein the determining further includes:

assigning a threshold in accordance with the number of complete segments of data in the queue.

10. The computer program product of claim 9 wherein instructions that cause the multipoint control unit to determine further comprises instructions to:

determine the number of segments of data awaiting service for each of the queues.

11. The computer program product of claim 10 wherein instructions that causes the multipoint control unit to determine if there is data in the queues further comprises instructions to:

determine for each of the queues whether there is a complete segment of data for the current picture in the queue.

12. The computer program product of claim 11 wherein instructions that cause the multipoint control unit to reduce bitrate use requantization with a requantization stepsize being a sum of a quantization stepsize in the received bitstream and a differential quantization stepsize.

13. The computer program product of claim 11 wherein instructions that causes the multipoint control unit to reduce bitrate use said differential quantization stepsize derived from a bitrate control data and said quantization stepsize in the received bitstream.

14. The computer program product of claim 13 wherein said differential quantization stepsize is derived from a bitrate control data and said quantization stepsize in the received bitstream.

15. The computer program product of claim 14 wherein said differential quantization stepsize is a function that is monotonically increasing with said bitrate control data and monotonically decreasing with said quantization stepsize in the received bitsteam.

16. A multipoint control unit comprises:

a plurality of segment processors arranged to extract segment data from a corresponding plurality of input video streams;

a plurality of data queues, each one of said data queues being coupled to a corresponding one of said plurality of segment processors for storing segment data provided from said segment processors;

a segment combiner coupled to said data queues for combining output data from said data queues; and

a controller coupled to said segment combiner for selectively providing rate control data to said segment processors in accordance with the amount of the data in said data queues.

17. The multipoint control unit of claim 16 wherein said combiner generates a compressed video bitstream with an output picture rate that is no less than a maximum input picture rate of said input video streams.

18. The multipoint control unit of claim 16 wherein said combiner determines the queue fullness of each of the queues.

19. The multipoint control unit of claim 16 wherein said combiner determines the number of segments of data awaiting service for each of the queues.

20. The multipoint control unit of claim 19 wherein said combiner that determines if there is data in the queues further determines if any of the queues has a complete picture of data awaiting transmission.

21. The multipoint control unit of claim 20 wherein said combiner determines for each of the queues whether there is a complete segment of data for the current picture in the queue.

22. The multipoint control unit of claim 16 wherein the segment processors further comprise:

a bit rate reducer; and

the combiner controller sends bit rate control information to the reducer to control the operation of the reducer.

23. The multipoint control unit of claim 22 wherein said controller determines the queue fullness of each of the queues.

24. The multipoint control unit of claim 23 wherein the segment processors use a requantization stepsize being a sum of a quantization stepsize in the received bitstream and a differential quantization stepsize.

25. The multipoint control unit of claim 24 wherein said differential quantization stepsize is derived from a bitrate control data and said quantization stepsize in the received bitstream.

26. The multipoint control unit of claim 24 wherein said differential quantization stepsize is a function that is monotonically increasing with said bitrate control data and monotonically decreasing with said quantization stepsize in the received bitsteam.

27. The multipoint control unit of claim 26 wherein said differential quantization stepsize Δ

qp is related to

28. A method of operating a multipoint control unit for use in a multipoint video conference comprises:

extracting segment data from a plurality of video streams;

storing the segment data in a one of a plurality of data queues;

determining whether there is data in any of the plurality of data queues that can be included in a new picture; and

combining data from said plurality of data queues in accordance with a determination of whether there is data in the data queues;

wherein the determining further comprises;

determining a number of segments of data awaiting service for each of the queues.

29. The method of claim 28 wherein determining if there is data in the queues further comprises:

determining if any of the queues has a complete picture of data awaiting transmission.

30. The method of claim 29 wherein said determining if there is data in the queues further comprises:

determining for each of the queues whether there is a complete segment of data for the current picture in the queue.

31. A method of operating a multipoint control unit for use in a multipoint video conference comprises:

extracting segment data from a plurality of video streams;

storing the segment data in a one of a plurality of data queues;

determining whether there is data in any of the plurality of data queues that can be included in a new picture; and

combining data from said plurality of data queues in accordance with a determination of whether there is data in the data queues;

wherein said extracting further comprises;

selectively reducing output bit rates from each one of said data queues;

wherein said combining further comprises;

combining data from said data queues to provide an overall output video bit stream syntax from the multipoint control unit which is in accordance with a syntax expected from a remote terminal.

32. The method of claim 31 wherein said reducing output bit rate is performed by requantizating with a requantization stepsize being a sum of a quantization stepsize in the received bit stream and a differential quantization stepsize.

33. The method of claim 32 wherein said differential quantization stepsize is derived from a bit rate control data and said quantization stepsize in the received bit stream.

34. The method of claim 33 wherein said differential quantization stepsize is a function that is monotonically increasing with said bit rate control data and monotonically decreasing with said quantization stepsize in the received bit stream.

35. The method of claim 34 wherein said bit rate control is derived from a queue fullness of said data queues.

36. A computer program product residing on a computer readable medium comprising instructions for causing a multipoint control unit to:

extract segment data to provide a plurality of extracted segment streams;

store each one of the segment data streams in a one of a plurality of data queues;

determine which of said data queues has data for inclusion in a new picture; and

combine data from said plurality of data queues in accordance with a determination of which data queues has data for inclusion in the new picture;

wherein an output picture rate of the combined video bit stream is related to the maximum input picture rate of said input video streams;

wherein instructions that causes the multipoint control unit to determine if there is data in the queues further comprises instructions to;

determine if any of the queues has a complete picture of data awaiting transmission.

37. A computer program product residing on a computer readable medium comprising instructions for causing a multipoint control unit to:

extract segment data to provide a plurality of extracted segment streams;

store each one of the segment data streams in a one of a plurality of data queues;

determine which of said data queues has data for inclusion in a new picture; and

combine data from said plurality of data queues in accordance with a determination of which data queues has data for inclusion in the new picture;

wherein an output picture rate of the combined video bit stream is related to the maximum input picture rate of said input video streams;

wherein instructions that causes the multipoint control unit to extract further comprises instructions to;

reduce output bit rates from the extracted segment data streams; and

combine data from said data queues to provide an overall output video bit stream from the multipoint control unit which is in accordance with a syntax expected by a remote terminal.

38. A computer program product residing on a computer readable medium comprising instructions for causing a multipoint control unit to:

extract segment data to provide a plurality of extracted segment streams;

store each one of the segment data streams in a one of a plurality of data queues;

determine which of said data queues has data for inclusion in a new picture; and

combine data from said plurality of data queues in accordance with a determination of which data queues has data for inclusion in the new picture;

wherein an output picture rate of the combined video bit stream is related to the maximum input picture rate of said input video streams;

wherein said bit rate control is derived from the queue fullness of said data queues.