Reduced bandwidth usage in video conferencing

• US 8,749,612 B1
• Filed: 12/01/2011
• Issued: 06/10/2014
• Est. Priority Date: 12/01/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: MacDonald, Andrew John, Ellner, Lars Henrik
• Primary Examiner(s): Ramakrishnaiah, Melur
• Application Number:
  US13/309,141
• Time in Patent Office: 922 Days
• Field of Search: 348 1401- 1416, 370260-261, 709/204, 709/233, 709/228
• US Class Current: 348/14.09
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

Video conferencing with reduced bandwidth usage is disclosed. One method comprises receiving an audio stream at a server station connected to a plurality of participants in the video conference, at least one participant being a client station connected to the server station by a network and supplying the audio stream, the server station including a first memory and processor, and the client station including a second memory and processor. A mixing algorithm is performed that determines which video streams of the participants to mix to form a conference video stream for display by at least one of the participants. A video stream from the client station is selectively transmitted to the server station responsive to the mixing algorithm, and the server station transmits the conference video stream and/or the streams to be mixed to the participant(s).

Claims

1. A method of conducting a video conference with reduced bandwidth usage, comprising:

receiving an audio stream at a server station connected to a plurality of participants in the video conference, at least one participant being a client station connected to the server station by a network and supplying the audio stream, the server station including a first memory and a first processor configured to execute instructions stored in the first memory, and the client station including a second memory and a second processor configured to execute instructions stored in the second memory;

performing a mixing algorithm using each of the first processor and the second processor, the mixing algorithm determining which video streams of the participants to mix to form a conference video stream for display by at least one of the participants;

selectively transmitting the client video stream from the client station to the server station over the network when a result of the mixing algorithm performed by the second processor indicates that the client video stream would be one of the video streams of the participants to be mixed to form the conference video stream responsive to performing the mixing algorithm using the first processor;

receiving a client video stream from the client station that is selectively transmitted to the server station over the network responsive to the mixing algorithm; and

transmitting at least one of the conference video stream or the video streams of the participants to be mixed to form the conference video stream from the server station to the client station.

2. The method of claim 1, further comprising:

transmitting the client video stream from the client station to the server station upon the client station establishing a connection to the server station; and

ending transmission of the client video stream from the client station to the server station when the result of the mixing algorithm performed by the second processor indicates that the client video stream would not be one of the video streams of the participants to be mixed to form the conference video stream responsive to performing the mixing algorithm using the first processor.

3. The method of claim 1 further comprising:

substituting a video stream of another participant for the client video stream to form the conference video stream when the mixing algorithm determines that the client video stream should be used to form the conference video stream and the client station does not transmit the client video stream to the server station.

4. The method of claim 1 wherein the mixing algorithm determines which video streams of the participants to mix to form the conference video stream using a plurality of audio streams received by the server station, each of the plurality of audio streams associated with one of the participants.

5. The method of claim 1, further comprising:

transmitting a conference audio stream from the server station while transmitting the conference video stream, the conference audio stream formed of at least one of a plurality of audio streams received by the server station and each of the plurality of audio streams associated with one of the participants.

6. The method of claim 1 wherein performing the mixing algorithm includes:

performing the mixing algorithm using only the first processor for a plurality of mix cycles, the method further comprising;

transmitting a signal to each of the participants whose video stream is to be mixed to form the conference video stream in a new cycle based on the mixing algorithm when those participants were not mixed to form the conference video stream in a previous cycle.

7. The method of claim 6, further comprising:

transmitting the client video stream when the client station receives the signal from the server station.

8. The method of claim 7, further comprising:

transmitting a list of the participants whose video streams form the conference video stream in the new cycle with the conference video stream; and

ending transmission of the client video stream from the client station to the server station when the client station is not included in the list.

9. The method of claim 6, further comprising:

transmitting a list of the participants whose video streams form the conference video stream in the new cycle with the at least one of the conference video stream or the video streams of the participants to be mixed to form the conference video stream.

10. An apparatus for conducting a video conference with reduced bandwidth usage, the apparatus comprising:

a client station including a first memory and a first processor configured to execute instructions stored in the first memory to;

perform a mixing algorithm that determines if a client video stream generated at the client station would be included in a conference video stream for display by at least one participant of the video conference; and

selectively transmit the client video stream to a server station to be mixed to form the conference video stream when a result of the mixing algorithm performed by the first processor indicates that the client video stream would be included in the conference video stream.

11. The apparatus of claim 10, further comprising:

the server station including a second memory and a second processor configured to execute instructions stored in the second memory to;

receive an audio stream from the client station, the client station connected to the server station by a network and the client station being one of a plurality of participants in the video conference;

perform the mixing algorithm;

receive the client video stream from the client station that is selectively transmitted to the server station over the network responsive to the mixing algorithm; and

transmit, responsive to performing the mixing algorithm, at least one of the conference video stream or the video streams of the participants to be mixed to form the conference video stream from the server station to the at least one participant.

12. The apparatus of claim 11 wherein the second processor is configured to perform the mixing algorithm by performing the mixing algorithm for a plurality of mix cycles;

and wherein the second processor is further configured to;

transmit a signal to each of the participants whose video stream is to be mixed to form the conference video stream in a new cycle based on the mixing algorithm when those participants were not mixed to form the conference video stream in a previous cycle.

13. The apparatus of claim 12 wherein the signal is a NACK signal.

14. The apparatus of claim 12, wherein:

the first processor configured to execute instructions stored in the first memory to;

transmit the client video stream from the client station to the server station when the client station receives the signal from the server station.

15. The apparatus of claim 14 wherein the second processor is further configured to:

transmit a list of the participants whose video streams form the conference video stream in the new cycle with the conference video stream; and

wherein the first processor is configured to;

end transmission of the client video stream from the client station to the server station when the client station is not included in the list.

16. The apparatus of claim 15 wherein the list is a CSRC list;

and wherein the client station is in the CSRC list when a SSRC of the client station is included in the CSRC.

17. An apparatus for conducting a video conference with reduced bandwidth usage, the apparatus comprising:

a client station including a first memory and a first processor configured to execute instructions stored in the first memory to;

connect to a server station through a network, the server station including a second memory and a second processor configured to execute instructions stored in the second memory to;

receive an audio stream from the client station, the client station being one of a plurality of participants in the video conference;

perform a mixing algorithm that determines which video streams of the participants to mix to form a conference video stream for display by at least one of the participants; and

transmit at least one of the conference video stream or the video streams of the participants to be mixed to form the conference video stream from the server station to the at least one of the participants;

perform the mixing algorithm;

selectively transmit a client video stream from the client station to the server station over the network when a result of the mixing algorithm performed by the second processor indicates that the client video stream would be one of the video streams of the participants to be mixed to form the conference video stream responsive to performing the mixing algorithm using the first processor; and

display the conference video stream.

18. The apparatus of claim 17 wherein the first processor is configured to:

receive the video streams of the participants to be mixed to form the conference video stream; and

mix the video streams to form the conference video stream.

19. The apparatus of claim 17 wherein the mixing algorithm is deterministic and wherein the first processor is configured to estimate an input into the mixing algorithm associated with at least one participant other than the client station.