Method and apparatus for multiple media digital communication system

• US 5,623,490 A
• Filed: 04/02/1996
• Issued: 04/22/1997
• Est. Priority Date: 06/09/1993
• Status: Expired due to Term

• Current Assignee: RPX Corporation
• Original Assignee: Intelligence-At-Large
• Inventors: Richter, Andreas, Reed, Jr., Ogden C.
• Primary Examiner(s): MARCELO, MELVIN C
• Application Number:
  US08/626,580
• Time in Patent Office: 385 Days
• Field of Search: 370/58.1, 370/60, 370/60.1, 370/62, 370/85.6, 370/94.1, 370/94.2, 370/94.3, 370/110.1, 379/53, 379/54, 379/201, 379/202, 379/205, 358/85, 358/86, 340/825.5, 340/825.51, 340/825.52, 364/514 A
• US Class Current: 370/263
• CPC Class Codes: H04L 12/1818 Conference organisation arrangements, e.g. handling schedules, setting up parameters needed by nodes to attend a conference, booking network resources, notifying involved parties
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 12/1881 with schedule organisation, e.g. priority, sequence management
• CPC Class Codes: H04L 12/6418 Hybrid transport
• CPC Class Codes: H04L 2012/6464 Priority
• CPC Class Codes: H04L 2012/6481 Speech, voice
• CPC Class Codes: H04L 2012/6483 Video, e.g. MPEG
• CPC Class Codes: H04L 2012/6489 Buffer Management, Threshold setting, Scheduling, Shaping
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/22 Traffic shaping
• CPC Class Codes: H04L 47/2416 Real-time traffic
• CPC Class Codes: H04L 47/30 in combination with information about buffer occupancy at either end or at transit nodes
• CPC Class Codes: H04L 47/50 Queue scheduling
• CPC Class Codes: H04L 47/52 by attributing bandwidth to queues
• CPC Class Codes: H04L 47/56 implementing delay-aware scheduling
• CPC Class Codes: H04L 47/6215 Individual queue per QOS, rate or priority
• CPC Class Codes: H04L 47/6225 Fixed service order, e.g. Round Robin
• CPC Class Codes: H04L 47/624 Altering the ordering of packets in an individual queue
• CPC Class Codes: H04L 49/90 Buffering arrangements
• CPC Class Codes: H04L 49/9094 Arrangements for simultaneous transmit and receive, e.g. simultaneous reading/writing from/to the storage element
• CPC Class Codes: H04L 65/1101 : Session protocols
• CPC Class Codes: H04L 65/4038 : with floor control
• CPC Class Codes: H04L 65/80 : Responding to QoS
• CPC Class Codes: H04L 9/40 : Network security protocols
• CPC Class Codes: H04M 2201/42 : Graphical user interfaces
• CPC Class Codes: H04M 3/563 : User guidance or feature selection
• CPC Class Codes: H04M 3/567 : Multimedia conference systems
• CPC Class Codes: H04M 3/568 : audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04M 7/006 : Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP) , including next generation networks with a packet-switched transport layer H04L65/00 takes precedence; aspects not specific to the type of network H04M3/00; special services in those networks H04M3/42
• CPC Class Codes: H04N 21/23614 : Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/2368 : Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/242 : Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/4302 : Content synchronisation processes, e.g. decoder synchronisation
• CPC Class Codes: H04N 21/43072 : of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 : Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4348 : Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 7/147 : Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 : Conference systems
• CPC Class Codes: H04N 7/52 : Systems for transmission of a pulse code modulated video signal with one or more other pulse code modulated signals, e.g. an audio signal or a synchronizing signal assembling of a multiplex stream by combining a video stream with other content or additional data, remultiplexing of multiplex streams, insertion of stuffing bits into the multiplex stream, assembling of a packetised elementary stream at server side H04N21/236; disassembling of a multiplex stream, remultiplexing of multiplex streams, extraction or processing of Service Information, disassembling of packetised elementary stream at client side H04N21/434

Abstract

The present invention is embodied in a digital communication system where multiple media data sources are time multiplexed into a packetized data stream, each packet having an assigned priority and the packetized data stream transmitted in substantially the order of assigned priority. At both the transmit side, and the receive side, audio packets are given priority processing over video packets, which in turn have priority over text/graphics data packets. Continuous real time audio playback is maintained at the receiver by delaying the playback of received audio in a first in/first out (FIFO) buffer providing a delay at least equal to the predicted average packet delay for the communication system. Optionally, the average system delay is continuously monitored, and the audio playback delay time is adjusted accordingly. Audio playback is slowed or accelerated in order to shrink or grow the difference in time between the sender and receiver. In another aspect of the invention, a conference of three or more callers is created by broadcasting a common packetized data stream to all conference callers.

Claims

1. A method of communicating media data packets from a first processing machine to at least one remote processing machine connected to said first processing machine over a communication system, said method comprising the steps of:

enabling at said first processing machine a respective connection processing stream for each remote processing machine with which said first processing machine communicates over said system during a connection;

enabling at said first processing machine at least one media processing stream associated with each said connection processing stream;

establishing an address table in said first processing machine which associates a media processing stream located in said at least one remote processing machine with a respective connection processing stream in said first processing machine;

processing with said media processing stream in said first machine at least one type of media data to produce media data packets;

consulting said address table to determine a communication path for sending said media data packets to a media processing stream located at said remote processing machine; and

transmitting said media data packets to said remote processing machine through a said connection stream associated with said media processing stream in said first processing machine and with an address of said media processing stream at said remote processing machine.

2. A method in accordance with claim 1, wherein said first processing machine receives media data packets from said remote processing machine, said method further comprising:

receiving at a receiving connection processing stream in said first processing machine media data packets directed thereto from a said remote processing machine; and

directing received media data packets from said receiving connection processing stream to a receiving media data processing stream for processing the same.

3. A method in accordance with claim 1, wherein plural types of media data packets are transmitted by said first processing machine, including audio data packets and video data packets, said method further comprising:

providing an association between a source of audio data and a source of video data and respective audio and video media processing streams which respectively provide audio and video data packets; and

transmitting each of said audio and video data packets through a said connection processing stream associated with said audio and video media processing streams in said first processing machine to audio and video data packet processing streams in said remote processing machine in accordance with address information contained in said table.

4. A method in accordance with claim 3, wherein said plural types of media data packets further includes text/graphics data packets, said method further comprising:

providing an association between a source of text/graphics data and a respective text/graphics media processing stream which provides text/grahics data packets; and

transmitting said text/graphics data packets through said associated connection processing stream in said first processing machine to a text/graphics media processing stream in said remote processing machine.

5. A method in accordance with claim 3, wherein said audio data packets and video data packets have respective associated priorities, said associated connection processing stream in said first processing machine transmitting said audio data packets and video data packets originating at said first processing machine to said remote processing machine in accordance with said associated priorities, wherein said audio data packets have a higher priority than said video data packets.

6. A method in accordance with claim 4, wherein said audio data packets, video data packets and graphics and/or text data packets have respective associated priorities, said associated connection processing stream in said first processing machine transmitting said audio data packets and video data packets and graphics/text data packets originating at said first processing machine to said communication system in accordance with said associated priorities, said audio data packets having a higher priority than said video data packets and said video data packets having a higher priority than said text/graphics data packets.

7. A method in accordance with claim 1, wherein each of said media data packets contains one or more addresses therein identifying a recipient for said packets.

8. A method in accordance with claim 7, wherein each of said media data packets contains a plurality of addresses therein identifying respective recipients for said packets.

9. A method in accordance with claim 2, further comprising checking in said first processing machine for errors sufficient to corrupt received media data packets, and requesting a retransmission of said received media data packets provided there is sufficient time for receipt of a retransmission based on the average system delay for a retransmission.

10. A method in accordance with claim 1, wherein each of said media data packets includes a destination count area containing a count of the number of destinations each packet should reach, and a data area representing how many of those destinations have not yet been reached.

11. A method in accordance with claim 1, further comprising the step of placing information in said address table at least during the establishment of a connection between said first processing machine and said at least one remote processing machine.

12. A method in accordance with claim 2, further comprising checking in said first processing machine for errors sufficient to corrupt received media data packets and for discarding corrupt received media data packets.

13. A method in accordance with claim 2, wherein said received media data packets contain audio data, said method further comprising the step of delaying an audio digital signal recovered from said received media data packets by a given delay to form a delayed recovered audio signal.

14. A method in accordance with claim 13, wherein said step of delaying said recovered audio signal to form a delayed recovered audio signal, further comprises:

setting said given delay substantially equal to the delay of said communication system.

15. A method in accordance with claim 14, wherein said step of setting said given delay substantially equal to the delay of said communication system, further comprises:

measuring the delay of said communication system;

increasing said given delay to be substantially equal to said measured delay; and

decreasing said given delay to be substantially equal to said measured delay,

wherein the delay of said delayed recovered audio signal increases and decreases substantially in accordance with said measured delay of said communication system.

16. A method in accordance with claim 2, wherein said received media data packets contain video data, said method further comprising the step of delaying a video digital signal recovered from said media data packets by a given delay to form a delayed recovered video signal.

17. A method in accordance with claim 16, wherein said step of delaying said recovered video signal to form a delayed recovered video signal, further comprises:

setting said given delay substantially equal to the delay of said communication system.

18. A method in accordance with claim 17, wherein said step of setting said given delay substantially equal to the delay of said communication system, further comprises:

measuring the delay of said communication system;

increasing said given delay to be substantially equal to said measured delay; and

decreasing said given delay to be substantially equal to said measured delay,

wherein the delay of said delayed recovered video signal increases and decreases substantially in accordance with said measured delay of said communication system.

19. A method in accordance with claim 1, wherein said at least one media processing stream processes video information to produce video data packets which are provided to said connection processing stream.

20. A method in accordance with claim 19, wherein said at least one media processing stream processes video frame information as sub-frames to produce video data packets which are provided to said connection processing stream.

21. A method in accordance with claim 2, wherein connection processing streams of said first processing machine process transmitted and received media data packets in accordance with priority information contained therein.

22. A method in accordance with claim 21, wherein said received media data packets include received audio data packets which have a higher assigned priority than other received media data packets, and wherein said first processing machine processes received audio data packets before processing other types of received media data packets.

23. A method in accordance with claim 2, wherein said first processing machine includes at least one media processing stream for processing received audio data packets, and at least one media processing stream for processing received video data packets associated with the audio data packets, each of said video processing stream and audio processing stream providing respective delays in the reproduction of video and audio data which are substantially equal to ensure synchronization of the audio and video data.

24. A method in accordance with claim 23, wherein the delay provided by said video processing stream and audio processing stream is set as the maximum delay through the communication system encountered by the received video data packets and received audio data packets.

25. A method in accordance with claim 11 wherein said at least one media processing stream processes audio information to create audio data packets which are provided to a said connection stream.

26. A method in accordance with claim 25 wherein said at least one media processing stream also process other media data packets as wherein said audio data packets and other media data packets have respective associated priorities, said connection processing stream in said first processing machine transmitting said audio data packets and other media data packets originating at said first processing machine to said communications system in accordance with said associated priorities, said audio data packets having a higher priority than said other media data packets.

27. A method in accordance with claim 3 wherein said audio data packets and video data packets have respective associated priorities, said connection processing stream in said first processing machine transmitting said audio data packets and video data packets originating at said first processing machine to said communications system in accordance with said associated priorities, said audio data packets having a higher priority than said video data packets.

46. An apparatus in accordance with claim 27, wherein said at least one media processing stream processes video information to produce video data packets which are provided to said associated connection processing stream.

47. An apparatus in accordance with claim 46, wherein said at least one media processing stream processes video frame information as sub-frames to produce video data packets which are provided to said associated connection processing stream.

28. An apparatus for communicating media data packets from a first processing machine to at least one remote processing machine connected to said first processing machine over a communication system, said apparatus comprising:

means for enabling at said first processing machine a respective connection processing stream for each remote processing machine with which said first processing machine communicates over said system during a connection;

means for enabling at said first processing machine at least one media processing stream associated with each said connection processing stream;

an address table provided in said first processing machine which associates a media processing stream located at in said at least one remote processing machine with a respective connection stream in said first processing machine;

means for processing with said media processing stream in said first processing machine at least one type of media data to produce media data packets;

means for consulting said address table to determine a communication path for sending said media data packets to a media processing stream located at said remote processing machine; and

means for transmitting said media data packets to said remote processing machine through a said connection stream associated with said media processing stream in said first processing machine and with an address of said media processing stream at said remote processing machine.

29. An apparatus in accordance with claim 28, wherein said first processing machine receives media data packets from said remote processing machine, said apparatus further comprising:

means for receiving at a receiving connection stream in said first processing machine media data packets directed thereto from a said remote processing machine; and

means for directing received media data packets from said receiving connection processing stream to a receiving media data processing stream for processing the same.

30. An apparatus in accordance with claim 28, wherein plural types of media data packets are transmitted by said first processing machine, including audio data packets and video data packets, said apparatus further comprising:

means for providing an association between a source of audio data and a source of video data and respective audio and video media processing streams which respectively provide audio and video data packets; and

means for transmitting each of said audio and video data packets through a said connection processing stream associated with said audio and video media processing streams said first processing machine to an audio and video data packet processing streams in said remote processing machine in accordance with address information contained in said table.

31. An apparatus in accordance with claim 30, wherein said plural types of media data packets further includes text/graphics data packets, said apparatus further comprising:

means for providing an association between a source of text/graphics data and a respective text/graphics media processing stream which provides text/grahics data packets; and

means for transmitting said text/graphics data packets through said associated connection processing stream in said first processing machine to a text/graphics media processing stream in said remote processing machine.

32. An apparatus in accordance with claim 30, wherein said audio data packets and video data packets have respective associated priorities, said associated connection processing stream in said first processing machine transmitting said audio data packets and video data packets originating at said first processing machine to said remote processing machine in accordance with said associated priorities, wherein said audio data packets have a higher priority than said video data packets.

33. An apparatus in accordance with claim 31, wherein said audio data packets, video data packets and text/graphics data packets have respective associated priorities, said associated connection processing stream in said first processing machine transmitting said audio data packets and video data packets and graphics/text data packets originating at said first processing machine to said communication system in accordance with said associated priorities, said audio data packets having a higher priority than said video data packets and said video data packets having a higher priority than said text/graphics data packets.

34. An apparatus in accordance with claim 28, wherein each of said media data packets contains one or more addresses therein identifying a recipient for said packets.

35. An apparatus in accordance with claim 34, wherein each of said media data packets contains a plurality of addresses therein identifying respective recipients for said packets.

36. An apparatus in accordance with claim 29, further comprising means for checking in said first processing machine for errors sufficient to corrupt received media data packets, and for requesting a retransmission of said received media data packets provided there is sufficient time for receipt of a retransmission based on the average system delay for a retransmission.

37. An apparatus in accordance with claim 29, wherein each of said media data packets includes a destination count area containing a count of the number of destinations each packet should reach, and a data area representing how many of those destinations have not yet been reached.

38. An apparatus in accordance with claim 28, further comprising means for placing information in said address table at least during the establishment of a connection between said first processing machine and said at least one remote processing machine.

39. An apparatus in accordance with claim 29, further comprising means for checking in said first processing machine for errors sufficient to corrupt received media data packets and for discarding corrupt received media data packets.

40. An apparatus in accordance with claim 29, wherein said received media data packets contain audio data, said apparatus further comprising means for delaying an audio digital signal recovered from said received media data packets by a given delay to form a delayed recovered audio signal.

41. An apparatus in accordance with claim 40, wherein said means for delaying delays said recovered audio signal to form a delayed recovered audio signal and further comprises:

means for setting said given delay substantially equal to the delay of said communication system.

42. An apparatus in accordance with claim 41, wherein said means for setting said given delay substantially equal to the delay of said communication system, further comprises:

means for measuring the delay of said communication system;

means for increasing said given delay to be substantially equal to said measured delay; and

means for decreasing said given delay to be substantially equal to said measured delay,

wherein the delay of said delayed recovered audio signal increases and decreases substantially in accordance with said measured delay of said communication system.

43. An apparatus in accordance with claim 29, wherein said received media data packets contain video data, said apparatus further comprising means for delaying a video digital signal recovered from said media data packets by a given delay to form a delayed recovered video signal.

44. A method in accordance with claim 43, wherein said means for delaying said recovered video signal to form a delayed recovered video signal, further comprises:

means for setting said given delay substantially equal to the delay of said communication system.

45. An apparatus in accordance with claim 44, wherein said means for setting said given delay substantially equal to the delay of said communication system, further comprises:

means for measuring the delay of said communication system;

means for increasing said given delay to be substantially equal to said measured delay; and

means for decreasing said given delay to be substantially equal to said measured delay,

wherein the delay of said delayed recovered video signal increases and decreases substantially in accordance with said measured delay of said communication system.

48. An apparatus in accordance with claim 29, wherein connection processing streams of said first processing machine process transmitted and received media data packets in accordance with priority information contained therein.

49. An apparatus in accordance with claim 29, wherein said first processing machine includes at least one media processing stream for processing received audio data packets, and at least one media processing stream for processing received video data packets associated with the audio data packets, each of said video processing stream and audio processing stream providing respective delays in the reproduction of video and audio data which are substantially equal to ensure synchronization of the audio and video data.

50. An apparatus in accordance with claim 49, wherein the delay provided by said video processing stream and audio processing stream is set as the maximum delay through the communication system encountered by the received video data packets and received audio data packets.

51. An apparatus in accordance with claim 48 wherein said received media data packets include received audio data packets which have a higher assigned priority than other received media data packets and wherein said first processing machine processes received audio data packets with said higher priority before processing other types of received media data packets.

52. An apparatus in accordance with claim 28 wherein said at least one processing stream processes audio information to create audio data packets which are provided to said associated connection stream.