Video conferencing over IP networks

• US 9,635,315 B2
• Filed: 11/09/2015
• Issued: 04/25/2017
• Est. Priority Date: 08/07/2006
• Status: Expired due to Fees

• Current Assignee: Krush Technologies, LLC
• Original Assignee: ooVoo LLC (Oovoo Holdings LLC)
• Inventors: Feng, Chang, Pesherov, Sergey
• Primary Examiner(s): Eskandarnia, Arvin
• Application Number:
  US14/935,987
• Publication Number:
  US 20160112673A1
• Time in Patent Office: 533 Days
• CPC Class Codes: G06Q 10/10 Office automation; Time management
• CPC Class Codes: H04L 41/509 wherein the managed service relates to media content delivery, e.g. audio, video or TV
• CPC Class Codes: H04L 47/2416 Real-time traffic
• CPC Class Codes: H04L 47/2475 for supporting traffic characterised by the type of applications
• CPC Class Codes: H04L 47/38 by adapting coding or compression rate
• CPC Class Codes: H04L 47/56 implementing delay-aware scheduling
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 7/0008 Synchronisation information channels, e.g. clock distribution lines
• CPC Class Codes: H04M 1/2535 adapted for voice communication over an Internet Protocol [IP] network Voice over Internet Protocol (VoIP) network equipment and services H04M7/006; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04M 3/5307 for recording messages comprising any combination of audio and non-audio components
• CPC Class Codes: H04M 3/5315 where the non-audio components are still images or video still image mailbox systems H04N1/324, intermediate storage of video signals for videophones H04N7/147
• CPC Class Codes: H04M 3/568 audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04M 7/0072 Speech codec negotiation H04L65/1069 takes precedence
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 Conference systems

Abstract

A method for communication includes establishing multiple communication links over a packet network between a server and plurality of client computers that are to participate in a video teleconference. The client computers may also create secondary communication links that function similarly to links between the server and client computers. The server receives from the client computers uplink audio packets and uplink video packets, which respectively contain audio and video data captured by each of the client computers. The server determines the optimal bandwidth for each client computer. The server also mixes the audio data from the uplink audio packets so as to create respective streams of mixed audio data for transmission to the client computers and transmits to the client computers downlink audio packets containing the respective streams of mixed audio data. The server relays the video data to the client computers in downlink video packets. The client computers receive and synchronize the video data with the mixed audio data.

Claims

1. A method for communication, comprising:

establishing a primary communication link over a packet network;

establishing at least one secondary communication link over a packet network;

receiving uplink audio packets and uplink video packets over at least one communication link, which respectively contain audio and video data, wherein receiving the uplink video packets comprises controlling a quality of the video data by transmitting instructions, wherein controlling a quality of the video data comprises determining a current uplink bandwidth, a current downlink bandwidth, a maximum uplink bandwidth, a minimum uplink bandwidth, a maximum downlink bandwidth, and a minimum downlink bandwidth;

mixing the audio data from the uplink audio packets to create respective streams of mixed audio data for transmission;

transmitting downlink audio packets containing the respective streams of mixed audio data over at least one communication link;

relaying the video data in downlink video packets;

analyzing relative time differences between the mixed audio data and each relayed video data stream;

generating at least one corresponding synchronization packet based on the analyzed relative time differences between mixed audio data and each relayed video data stream; and

,

transmitting over at least one communication link the at least one corresponding synchronization packet containing synchronization information for synchronizing the relayed video data in the downlink video packets with the downlink audio packets containing the respective streams of mixed audio data in addition to transmitting the downlink video packets and the downlink audio packets.

2. The method according to claim 1, wherein determining the maximum uplink bandwidth includes using a video codec.

3. The method according to claim 1, wherein determining the minimum uplink bandwidth includes using a video codec.

4. The method according to claim 1, wherein determining the maximum downlink bandwidth includes using a forward error correction method.

5. The method according to claim 1, wherein determining the current downlink bandwidth is between the minimum downlink bandwidth and the maximum downlink bandwidth.

6. The method according to claim 1, wherein determining the current uplink bandwidth is between the minimum uplink bandwidth and the maximum uplink bandwidth.

7. The method according to claim 1, wherein determining the current downlink bandwidth is equal to the current uplink video bandwidth.

8. The method according to claim 1, wherein determining the current uplink bandwidth and the current downlink bandwidth includes simultaneously maximizing the current uplink bandwidth and the current downlink bandwidth.

9. The method according to claim 8, wherein simultaneously maximizing the current uplink bandwidth and the current downlink bandwidth includes using a linear programming model.

10. The method according to claim 9, wherein the linear programming model comprises using a Simplex linear programming method.

11. The method according to claim 1, wherein transmitting the instructions comprises instructions to reduce the quality of the video data transmitted in the uplink video packets responsively to detecting the delay in receiving one or more of the downlink audio and video packets.

12. The method according to claim 1, wherein controlling the quality comprises instructing an increase or decrease of at least one quality parameter selected from a group of quality parameters consisting of an image resolution, a degree of image compression, a frame rate and a bandwidth.

13. The method according to claim 1, wherein receiving the uplink packets comprises detecting a delay in the audio data and eliminating an interval of silent audio data in order to compensate for the delay.

14. The method according to claim 13, wherein eliminating the interval comprises marking at least one block of the audio data as a silent block and eliminating the silent block from the mixed audio data.

15. The method according to claim 1, wherein relaying the video data comprises passing the video data from the uplink video packets to the downlink video packets without transcoding of the video data at the server.

16. The method according to claim 1, wherein receiving the uplink audio and video packets comprises receiving synchronization data, generating synchronization information on the synchronization data, and transmitting the synchronization information for use in synchronizing the video data with the mixed audio data.

17. The method according to claim 1, wherein the at least one secondary communication link includes a p2p communication link.

18. The method according to claim 1, wherein transmitting over at least one communication link the at least one corresponding synchronization packet includes transmitting over all communication links simultaneously.

19. The method according to claim 1, wherein transmitting downlink audio packets includes transmitting over all communication links simultaneously.

20. The method according to claim 1, wherein receiving uplink audio packets and uplink video packets includes receiving over all communication links simultaneously.

21. A communication apparatus, comprising:

a server, which is coupled to establish a primary communication link over a packet network with remote computers that are to participate in a video teleconference and to receive uplink audio packets and uplink video packets, which respectively comprise audio and video data, over the primary communication link, wherein receiving the uplink video packets comprises controlling a quality of the video data by transmitting instructions, wherein controlling a quality of the video data comprises determining a current uplink bandwidth, a current downlink bandwidth, a maximum uplink bandwidth, a minimum uplink bandwidth, a maximum downlink bandwidth, and a minimum downlink bandwidth;

the server, which is also coupled to establish at least one secondary communication link over a packet network with remote computers that are to participate in a video teleconference and to receive uplink audio packets and uplink video packets, which respectively comprise audio and video data, over the communication links;

wherein the server is configured to mix the audio data from the uplink audio packets so as to create respective streams of mixed audio data for transmission to the remote computers, and to generate at least one synchronization packet containing synchronization information, and to transmit to the remote computers over at least one communication link downlink audio packets containing the respective streams of mixed audio data in addition to the at least one synchronization packet while relaying the video data from the uplink video packets to the client computers in downlink video packets.

22. The communication apparatus according to claim 21, wherein determining the maximum uplink bandwidth includes using a video codec.

23. The communication apparatus according to claim 21, wherein determining the minimum uplink bandwidth includes using a video codec.

24. The communication apparatus according to claim 21, wherein determining the maximum downlink bandwidth includes using a forward error correction method.

25. The communication apparatus according to claim 21, wherein determining the current downlink bandwidth is between the minimum downlink bandwidth and the maximum downlink bandwidth.

26. The communication apparatus according to claim 21, wherein determining the current uplink bandwidth is between the minimum uplink bandwidth and the maximum uplink bandwidth.

27. The communication apparatus according to claim 21, wherein determining the current downlink bandwidth is equal to the current uplink video bandwidth.

28. The communication apparatus according to claim 21, wherein determining the current uplink bandwidth and the current downlink bandwidth includes simultaneously maximizing the current uplink bandwidth and the current downlink bandwidth.

29. The communication apparatus according to claim 28, wherein simultaneously maximizing the current uplink bandwidth and the current downlink bandwidth includes using a linear programming model.

30. The communication apparatus according to claim 29, wherein the linear programming model comprises using a Simplex linear programming method.

31. The communication apparatus according to claim 21, wherein to transmit to the remote computers over at least one communication link comprises transmitting instructions to reduce the quality of the video data transmitted in the uplink video packets responsively to detecting the delay in receiving one or more of the downlink audio and video packets.

32. The communication apparatus according to claim 21, wherein controlling the quality comprises instructing an increase or decrease of at least one quality parameter selected from a group of quality parameters consisting of an image resolution, a degree of image compression, a frame rate and a bandwidth.

33. The communication apparatus according to claim 21, wherein receiving the uplink packets comprises detecting a delay in the audio data and eliminating an interval of silent audio data in order to compensate for the delay.

34. The communication apparatus according to claim 33, wherein eliminating the interval comprises marking at least one block of the audio data as a silent block and eliminating the silent block from the mixed audio data.

35. The communication apparatus according to claim 21, wherein the at least one secondary communication link includes a p2p communication link.

36. The communication apparatus according to claim 21, wherein to transmit to the remote computers over at least one communication link includes transmitting over all communication links simultaneously.

37. The communication apparatus according to claim 21, wherein to receive uplink audio packets and uplink video packets comprises receiving over all communication links simultaneously.

38. A server, comprising:

a network interface, which is coupled to establish a primary communication link over a packet network with a plurality of remote computers that are to participate in a video teleconference, and to receive from the remote computers uplink audio packets and uplink video packets, which respectively contain audio and video data captured by each of the remote computers wherein receiving the uplink video packets comprises controlling a quality of the video data by transmitting instructions, wherein controlling a quality of the video data comprises determining a current uplink bandwidth, a current downlink bandwidth, a maximum uplink bandwidth, a minimum uplink bandwidth, a maximum downlink bandwidth, and a minimum downlink bandwidth; and

a network interface, which is coupled to establish at least one secondary communication link over a packet network with a plurality of remote computers that are to participate in a video teleconference, and to receive from the remote computers uplink audio packets and uplink video packets;

a processor, which is configured to mix the audio data from the uplink audio packets so as to create respective streams of mixed audio data for transmission to the remote computers and to generate at least one synchronization packet containing synchronization information, and to transmit to the remote computers via the network interface downlink audio packets containing the respective streams of mixed audio data in addition to the at least one synchronization packet while relaying the video data from the uplink video packets to the remote computers in downlink video packets.

39. A computer program product, comprising a non-transitory computer usable medium having a computer readable program code embodied therein, said computer readable program code adapted to be executed to implement a method for implementing a method for communication, said method comprising:

establishing a primary communication link over a packet network;

establishing at least one secondary communication link over a packet network;

receiving uplink audio packets and uplink video packets over at least one communication link, which respectively contain audio and video data, wherein receiving the uplink video packets comprises controlling a quality of the video data by transmitting instructions, wherein controlling a quality of the video data comprises determining a current uplink bandwidth, a current downlink bandwidth, a maximum uplink bandwidth, a minimum uplink bandwidth, a maximum downlink bandwidth, and a minimum downlink bandwidth;

mixing the audio data from the uplink audio packets to create respective streams of mixed audio data for transmission;

transmitting downlink audio packets containing the respective streams of mixed audio data over at least one communication link;

relaying the video data in downlink video packets;

analyzing relative time differences between the mixed audio data and each relayed video data stream;

generating at least one corresponding synchronization packet based on the analyzed relative time differences between mixed audio data and each relayed video data stream; and

,

transmitting over at least one communication link the at least one corresponding synchronization packet containing synchronization information for synchronizing the relayed video data in the downlink video packets with the downlink audio packets containing the respective streams of mixed audio data in addition to transmitting the downlink video packets and the downlink audio packets.

40. The computer program product according to claim 39, wherein determining the maximum uplink bandwidth includes using a video codec.

41. The computer program product according to claim 39, wherein determining the minimum uplink bandwidth includes using a video codec.

42. The computer program product according to claim 39, wherein determining the maximum downlink bandwidth includes using a forward error correction method.

43. The computer program product according to claim 39, wherein determining the current downlink bandwidth is between the minimum downlink bandwidth and the maximum downlink bandwidth.

44. The computer program product according to claim 39, wherein determining the current uplink bandwidth is between the minimum uplink bandwidth and the maximum uplink bandwidth.

45. The computer program product according to claim 39, wherein determining the current downlink bandwidth is equal to the current uplink video bandwidth.

46. The computer program product according to claim 39, wherein determining the current uplink bandwidth and the current downlink bandwidth includes simultaneously maximizing the current uplink bandwidth and the current downlink bandwidth.

47. The computer program product according to claim 46, wherein simultaneously maximizing the current uplink bandwidth and the current downlink bandwidth includes using a linear programming model.

48. The computer program product according to claim 47, wherein the linear programming model comprises using a Simplex linear programming method.

49. The computer program product according to claim 39, wherein transmitting the instructions comprises instructions to reduce the quality of the video data transmitted in the uplink video packets responsively to detecting the delay in receiving one or more of the downlink audio and video packets.

50. The computer program product according to claim 39, wherein controlling the quality comprises instructing an increase or decrease of at least one quality parameter selected from a group of quality parameters consisting of an image resolution, a degree of image compression, a frame rate and a bandwidth.

51. The computer program product according to claim 39, wherein receiving the uplink packets comprises detecting a delay in the audio data and eliminating an interval of silent audio data in order to compensate for the delay.

52. The computer program product according to claim 51, wherein eliminating the interval comprises marking at least one block of the audio data as a silent block and eliminating the silent block from the mixed audio data.

53. The computer program product according to claim 39, wherein relaying the video data comprises passing the video data from the uplink video packets to the downlink video packets without transcoding of the video data at the server.

54. The computer program product according to claim 39, wherein receiving the uplink audio and video packets comprises receiving synchronization data, generating synchronization information on the synchronization data, and transmitting the synchronization information for use in synchronizing the video data with the mixed audio data.

55. The computer program product according to claim 39, wherein the at least one secondary communication link includes a p2p communication link.

56. The computer program product according to claim 39, wherein transmitting over at least one communication link the at least one corresponding synchronization packet includes transmitting over all communication links simultaneously.

57. The computer program product according to claim 39, wherein transmitting downlink audio packets includes transmitting over all communication links simultaneously.

58. The computer program product according to claim 39, wherein receiving uplink audio packets and uplink video packets includes receiving over all communication links simultaneously.

59. A client computer, comprising:

a user interface; and

a processor, which is configured to establish a primary communication link and at least one secondary communication link over a packet network so as to participate in a video teleconference, and to transmit uplink audio packets and uplink video packets over at least one communication link, which respectively contain audio and video data,

wherein the processor is configured to receive over at least one communication link downlink audio packets containing the a stream of mixed audio data, to receive over at least one communication link synchronization packets containing synchronization information generated, and to receive over at least one communication link downlink video packets containing the video data transmitted by other remote computers in the video teleconference, and to synchronize the video data with the mixed audio data based on the synchronization information for output via the user interface.

60. The client computer according to claim 59, wherein the at least one secondary communication link includes a p2p communication link.

61. The client computer according to claim 59, wherein to transmit uplink audio packets and uplink video packets over at least one communication link includes transmitting over all communication links simultaneously.

62. The client computer according to claim 59, wherein to receive over at least one communication link includes receiving over all communication links simultaneously.