Adaptively controlled multipoint videoconferencing system

• US 5,684,527 A
• Filed: 07/09/1993
• Issued: 11/04/1997
• Est. Priority Date: 07/28/1992
• Status: Expired due to Term

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: Fujitsu Limited
• Inventors: Terui, Yuichi, Usui, Toshiaki
• Primary Examiner(s): WOO, STELLA L
• Application Number:
  US08/088,336
• Time in Patent Office: 1,579 Days
• Field of Search: 348/13, 348/14, 348/15, 348/16, 348/17, 348/18, 348/19, 348/20, 379/96, 379/94, 379/202, 379/203, 379/204, 379/206
• US Class Current: 348/14.09
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/132 Sampling, masking or truncation of coding units, e.g. adaptive resampling, frame skipping, frame interpolation or high-frequency transform coefficient masking
• CPC Class Codes: H04N 19/137 Motion inside a coding unit, e.g. average field, frame or block difference
• CPC Class Codes: H04N 19/146 Data rate or code amount at the encoder output
• CPC Class Codes: H04N 19/172 the region being a picture, frame or field
• CPC Class Codes: H04N 19/174 the region being a slice, e.g. a line of blocks or a group of blocks
• CPC Class Codes: H04N 19/46 Embedding additional information in the video signal during the compression process H04N19/517, H04N19/68, H04N19/70 take precedence
• CPC Class Codes: H04N 19/503 involving temporal prediction adaptive coding with adaptive selection between spatial and temporal predictive coding H04N19/107; adaptive coding with adaptive selection among a plurality of temporal predictive coding modes H04N19/109
• CPC Class Codes: H04N 19/61 in combination with predictive coding
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

In a multipoint videoconferencing system including a plurality of visual telephone terminals and a multipoint control unit (MCU) connected thereto, shares of the respective visual telephone terminals are determined in the MCU, based on determination of a speaker determined from voice levels and/or amount of movement in the respective visual telephone terminals. The respective telephone terminals generate pictures at picture rates defined by the respective shares, and generate video data from the pictures at data rates defined by the respective shares. Video data generated in each visual telephone terminal are multiplexed in time division multiplexer in the MCU, and are distributed to the other terminals.

Claims

1. A method of adaptively controlling a multipoint videoconferencing system including a plurality of visual telephone terminals and a multipoint control unit connected to the plurality of visual telephone terminals, comprising the steps of:

a) determining respective shares of the plurality of visual telephone terminals;

b) generating respective video data of moving pictures in the plurality of visual telephone terminals at data rates corresponding to the respective, determined shares of the visual telephone terminals;

c) distributing the respective video data generated in each visual telephone terminal to the others of the plurality of visual telephone terminals; and

the step (b) further comprising the substeps of;

i) generating pictures at picture rates defined by the shares of the respective visual telephone terminals,

ii) generating the video data from the pictures at video data rates defined by the shares of the respective visual telephone terminals, and

iii) in the substep i), continuously selecting one of an actual picture and a null picture so that the actual picture is selected at the picture rate, thereby to generate the pictures substantially at the picture rate.

2. A method as claimed in claim 1, wherein in the substep ii), the generation of the video data is controlled by inserting null data in an amount which is defined by the by the video data rate, to thereby substantially generate the video data at the video data rate.

3. A method as claimed in claim 1, wherein the step a) includes the substeps of:

i) detecting respective amounts of motion of persons at the plurality of visual telephone terminals; and

ii) calculating the respective shares of the plurality of visual telephone terminals from the respective motion amounts detected at the visual telephone terminals.

4. A method as claimed in claim 3, wherein in the substep ii), the shares are determined so as to be substantially proportional to the motion amount at the respective visual telephone terminals.

5. A method as claimed in claim 4, wherein values of the motion amount are transported on the video data transferred from the visual telephone terminals to the multipoint control unit, and values of the shares are transported on the video data transferred from the multipoint control unit to the visual telephone terminals.

6. A method as claimed in claim 1, wherein the step a) includes the substeps of:

comparing levels of voices made in the visual telephone terminals;

determining a first visual telephone terminal where a speaker exists and second visual telephone terminals where the speaker does not exist, based on the comparison result of the voice levels;

assigning a first share to the first visual telephone terminal; and

assigning second shares smaller than the first share to the second visual telephone terminals.

7. A method as claimed in claim 1, wherein the step a) includes the substeps of:

comparing levels of voices at the visual telephone terminals;

determining a visual telephone terminal where a speaker exists, based on the comparison result of the voice levels;

detecting an amount of motion of persons at the respective visual telephone terminals; and

determining the share for the respective visual telephone terminals, based on the determination of the visual telephone terminal where the speaker exists, and the motion amount detected at the visual telephone terminals.

8. A method as claimed in claim 1, wherein the step c) includes the substeps of:

transmitting the video data from the visual telephone terminals to the multipoint control unit;

multiplexing the video data transmitted from the visual telephone terminals, in the multipoint control unit; and

transmitting the multiplexed video data from the multipoint control unit to the visual telephone terminals.

9. A multipoint videoconferencing system including a plurality of visual telephone terminals and a multipoint control unit connected to the plurality of visual telephone terminals, comprising:

means for determining shares of the respective visual telephone terminals;

means for generating video data of moving pictures in the respective visual telephone terminals at data rates corresponding to the respective shares of the visual telephone terminals, as determined by the determining means;

means for distributing the video data, generated by the generating means in each of the visual telephone terminals, to the others of the plurality of visual telephone terminals;

the video data generating means further comprising;

means for generating pictures at picture rates defined by the shares of the respective visual telephone terminals, and

means for generating the video data from the pictures at video data rates defined by the shares of the respective visual telephone terminals; and

the picture generating means further comprising;

a selector for selecting one of an actual picture and a null picture at a constant rate, and

means for continuously controlling the selector so that the actual picture is selected at the picture rate, to thereby substantially generate the pictures at the picture rate.

10. A system as claimed in claim 9, wherein the video data generating means includes:

a coder for coding the picture into a variable amount of video data;

a buffer for storing the video data output from the coder;

means for adding null data, an amount of which is defined by the video data rate, to video data taken out from the buffer to make video data having a constant rate; and

means for controlling the amount of video data coded in the coder so that occupation of the buffer is maintained.

11. A system as claimed in claim 9, wherein the share determining means includes:

means for detecting an amount of motion of persons at the respective visual telephone terminals; and

means for calculating the shares of the respective visual telephone terminals from the motion amount detected by the detecting means.

12. A system as claimed in claim 11, wherein the share calculating means determines the shares so as to be substantially proportional to the motion amount.

13. A system as claimed in claim 12, further comprising means for inserting values of motion amount into the video data transferred from the visual telephone terminals to the multipoint control unit, and means for inserting values of the shares into the video data transferred from the multipoint control unit to the visual telephone terminals.

14. A system as claimed in claim 9, wherein the share determining means includes:

means for comparing levels of voices at the visual telephone terminals;

means for determining a first visual telephone terminal where a speaker exists and second visual telephone terminals where the speaker does not exist, based on the comparison result of the comparing means; and

means for assigning a first share to the first visual telephone terminal, and for assigning second shares smaller than the first share to the second visual telephone terminals.

15. A system as claimed in claim 9, wherein the share determining means includes:

means for comparing levels of voices at the visual telephone terminals;

first means for determining a visual telephone terminal where a speaker exists, based on the comparison result of the comparing means;

means for detecting amount of motion of persons at the respective visual telephone terminals; and

second means for determining the shares of the respective visual telephone terminals, based on the determination for the first determining means, and the motion amount detected by the detecting means.

16. A system as claimed in claim 9, wherein the distributing means includes:

first means for transmitting the video data transmitted by the transmitting means in the multipoint control unit;

means for multiplexing the video data transmitted by the transmitting means in the multipoint control unit; and

second means for transmitting the multiplexed video data by the multiplexing means to the visual telephone terminals.

17. A visual telephone terminal for use in a multipoint videoconferencing system, including a multipoint control unit connected to the visual telephone terminal and other visual telephone terminals connected to the multipoint control unit, comprising:

means for receiving a share determined in the multipoint control unit;

means for generating video data according to the share received by the receiving means;

means for transmitting the video data generated by the generating means to the multipoint control unit;

means for receiving video data generated in the other visual telephone terminals and multiplexed in the multipoint control unit, from the multipoint control unit;

the means for generating video data further comprising;

means for generating pictures at picture rates defined by the share, and

means for generating the video data from the pictures at a video data rate defined by the share; and

the picture generating means further comprising;

a selector for selecting one of an actual picture and a null picture at a constant rate, and

means for continuously controlling the selector so that the actual picture is selected at the picture rate, to thereby substantially generate the pictures at the picture rate.

18. A visual telephone terminal as claimed in claim 17, wherein the video data generating means includes:

a coder for coding the pictures into a variable amount of video data;

a buffer for storing the video data output from the coder;

means for adding null data, an amount of which is defined by the video data rate, to video data taken out from the buffer to make video data having a constant rate; and

means for controlling the amount of video data coded in the coder so that occupation of the buffer is maintained.

19. A visual telephone terminal as claimed in claim 17, further comprising:

means for detecting an amount of motion of a person; and

means for transmitting the motion amount detected by the detecting means to the multipoint control unit.

20. A method of adaptively controlling a multipoint videoconferencing system including a plurality of visual telephone terminals and a multipoint control unit connected to the plurality of visual telephone terminals, comprising the steps of:

a) determining respective shares of the plurality of visual telephone terminals;

b) generating respective video data of moving pictures in the plurality of visual telephone terminals at data rates corresponding to the respective, determined shares of the visual telephone terminals;

c) distributing the respective video data generated in each visual telephone terminal to the others of the plurality of visual telephone terminals; and

d) performing (the step b) in accordance with the substeps of;

i) generating pictures at picture rates defined by the shares of the respective visual telephone terminals, and

ii) generating the video data from the pictures at video data rates defined by the shares of the respective visual telephone terminals, wherein in the substep ii), the generation of the video data is controlled by inserting null data in an amount which is defined by the video data rate, to thereby substantially generate the video data at the video data rate.

21. A method as claimed in claim 20, wherein the step a) includes the substeps of:

i) detecting respective amounts of motion of persons at the plurality of visual telephone terminals; and

ii) calculating the respective shares of the plurality of visual telephone terminals from the respective motion amounts detected at the visual telephone terminals.

22. A method as claimed in claim 21, wherein in the substep ii), the respective shares are determined so as to be substantially proportional to the respective motion amounts detected at the plurality of visual telephone terminals.

23. A method as claimed in claim 22, wherein values of the respective motion amounts are transported on the corresponding video data transferred from the plurality of visual telephone terminals to the multipoint control unit, and the respective values of the shares are transported on the corresponding video data transferred from the multipoint control unit to the plurality of visual telephone terminals.

24. A method as claimed in claim 20, wherein the step a) includes the substeps of:

comparing levels of voices made in the visual telephone terminals;

determining a first visual telephone terminal where a speaker exists and second visual telephone terminals where the speaker does not exist, based on the comparison result of the voice levels;

assigning a first share to the first visual telephone terminal; and

assigning second shares smaller than the first share to the second visual telephone terminals.

25. A method as claimed in claim 20, wherein the step a) includes the substeps of:

comparing levels of voices at the visual telephone terminals;

determining a visual telephone terminal where a speaker exists, based on the comparison result of the voice levels;

detecting an amount of motion of persons at the respective visual telephone terminals; and

determining the share for the respective visual telephone terminals, based on the determination of the visual telephone terminal where the speaker exists, and the motion amount detected at the visual telephone terminals.

26. A method as claimed in claim 20, wherein the step c) includes the substeps of:

transmitting the video data from the visual telephone terminals to the multipoint control unit;

multiplexing the video data transmitted from the visual telephone terminals, in the multipoint control unit; and

transmitting the multiplexed video data from the multipoint control unit to the visual telephone terminals.

27. A multipoint videoconferencing system including a plurality of visual telephone terminals and a multipoint control unit connected to the plurality of visual telephone terminals, comprising:

means for determining shares of the respective visual telephone terminals;

means for generating video data of moving pictures in the respective visual telephone terminals at data rates corresponding to the respective shares of the visual telephone terminals, as determined by the determining means;

means for distributing the video data, generated by the generating means in each of the visual telephone terminals, to the others of the plurality of visual telephone terminals, wherein the video data generating means includes;

means for generating pictures at picture rates defined by the shares of the respective visual telephone terminals; and

means for generating the video data from the pictures at video data rates defined by the shares of the respective visual telephone terminals, wherein the video data generating means includes;

a coder for coding the picture into a variable amount of video data;

a buffer for storing the video data output from the coder;

means for adding null data, an amount of which is defined by the video data rate, to video data taken out from the buffer to make video data having a constant rate; and

means for controlling the amount of video data coded in the coder so that occupation of the buffer is maintained.

28. A system as claimed in claim 27, wherein the share determining means includes:

means for detecting an amount of motion of persons at the respective visual telephone terminals; and

means for calculating the shares of the respective visual telephone terminals from the motion amount detected by the detecting means.

29. A system as claimed in claim 28, wherein the share calculating means determines the shares so as to be substantially proportional to the motion amount.

30. A system as claimed in claim 29, further comprising means for inserting values of motion amount into the video data transferred from the visual telephone terminals to the multipoint control unit, and means for inserting values of the shares into the video data transferred from the multipoint control unit to the visual telephone terminals.

31. A system as claimed in claim 27, wherein the share determining means includes:

means for comparing levels of voices at the visual telephone terminals;

means for determining a first visual telephone terminal where a speaker exists and second visual telephone terminals where the speaker does not exist, based on the comparison result of the comparing means; and

means for assigning a first share to the first visual telephone terminal, and for assigning second shares smaller than the first share to the second visual telephone terminals.

32. A system as claimed in claim 27, wherein the share determining means includes:

means for comparing levels of voices at the visual telephone terminals;

first means for determining a visual telephone terminal where a speaker exists, based on the comparison result of the comparing means;

means for detecting amount of motion of persons at the respective visual telephone terminals; and

second means for determining the shares of the respective visual telephone terminals, based on the determination for the first determining means, and the motion amount detected by the detecting means.

33. A system as claimed in claim 27, wherein the distributing means includes:

first means for transmitting the video data transmitted by the transmitting means in the multipoint control unit;

means for multiplexing the video data transmitted by the transmitting means in the multipoint control unit; and

second means for transmitting the multiplexed video data by the multiplexing means to the visual telephone terminals.

34. A visual telephone terminal for use in a multipoint videoconferencing system, including a multipoint control unit connected to the visual telephone terminal and other visual telephone terminals connected to the multipoint control unit, comprising:

means for receiving a share determined in the multipoint control unit;

means for generating video data of a moving picture at a data rate corresponding to the share received by the receiving means;

means for transmitting the video data generated by the generating means to the multipoint control unit; and

means for receiving video data generated in the other visual telephone terminals and multiplexed in the multipoint control unit, from the multipoint control unit, wherein the video data generating means includes;

means for generating pictures at picture rates defined by the share; and

means for generating the video data from the pictures at a video data rate defined by the share, wherein the video data generating means includes;

a coder for coding the pictures into a variable amount of video data;

a buffer for storing the video data output from the coder;

means for adding null data, an amount of which is defined by the video data rate, to video data taken out from the buffer to make video data having a constant rate; and

means for controlling the amount of video data coded in the coder so that occupation of the buffer is maintained.

35. A visual telephone terminal as claimed in claim 34, further comprising:

means for detecting an amount of motion of a person; and

means for transmitting the motion amount detected by the detecting means to the multipoint control unit.