Method and apparatus for multiple media digital communication system

• US 6,104,706 A
• Filed: 11/10/1999
• Issued: 08/15/2000
• Est. Priority Date: 06/09/1993
• Status: Expired due to Term

• Current Assignee: RPX Corporation
• Original Assignee: Intelligence At Large Incorporated, Micron Technology, Inc.
• Inventors: Reed, Jr., Ogden Cartwright, Richter, Andreas
• Primary Examiner(s): MARCELO, MELVIN C
• Application Number:
  US09/437,269
• Time in Patent Office: 279 Days
• Field of Search: 370/263, 370/412, 370/260-271, 370/389, 370/462, 345/330
• US Class Current: 370/263
• CPC Class Codes: H04L 12/1818 Conference organisation arrangements, e.g. handling schedules, setting up parameters needed by nodes to attend a conference, booking network resources, notifying involved parties
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 12/1881 with schedule organisation, e.g. priority, sequence management
• CPC Class Codes: H04L 12/6418 Hybrid transport
• CPC Class Codes: H04L 2012/6464 Priority
• CPC Class Codes: H04L 2012/6481 Speech, voice
• CPC Class Codes: H04L 2012/6483 Video, e.g. MPEG
• CPC Class Codes: H04L 2012/6489 Buffer Management, Threshold setting, Scheduling, Shaping
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/22 Traffic shaping
• CPC Class Codes: H04L 47/2416 Real-time traffic
• CPC Class Codes: H04L 47/30 in combination with information about buffer occupancy at either end or at transit nodes
• CPC Class Codes: H04L 47/50 Queue scheduling
• CPC Class Codes: H04L 47/52 by attributing bandwidth to queues
• CPC Class Codes: H04L 47/56 implementing delay-aware scheduling
• CPC Class Codes: H04L 47/6215 Individual queue per QOS, rate or priority
• CPC Class Codes: H04L 47/6225 Fixed service order, e.g. Round Robin
• CPC Class Codes: H04L 47/624 Altering the ordering of packets in an individual queue
• CPC Class Codes: H04L 49/90 Buffering arrangements
• CPC Class Codes: H04L 49/9094 Arrangements for simultaneous transmit and receive, e.g. simultaneous reading/writing from/to the storage element
• CPC Class Codes: H04L 65/1101 : Session protocols
• CPC Class Codes: H04L 65/4038 : with floor control
• CPC Class Codes: H04L 65/80 : Responding to QoS
• CPC Class Codes: H04L 9/40 : Network security protocols
• CPC Class Codes: H04M 2201/42 : Graphical user interfaces
• CPC Class Codes: H04M 3/563 : User guidance or feature selection
• CPC Class Codes: H04M 3/567 : Multimedia conference systems
• CPC Class Codes: H04M 3/568 : audio processing specific to telephonic conferencing, e.g. spatial distribution, mixing of participants echo suppression in two-way loud-speaking telephone systems H04M9/02; sound field processing per se H04S7/30
• CPC Class Codes: H04M 7/006 : Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP) , including next generation networks with a packet-switched transport layer H04L65/00 takes precedence; aspects not specific to the type of network H04M3/00; special services in those networks H04M3/42
• CPC Class Codes: H04N 21/23614 : Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/2368 : Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/242 : Synchronization processes, e.g. processing of PCR [Program Clock References] arrangements for synchronising broadcast or distribution via plural systems in broadcast distribution systems H04H20/18
• CPC Class Codes: H04N 21/4302 : Content synchronisation processes, e.g. decoder synchronisation
• CPC Class Codes: H04N 21/43072 : of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 : Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4348 : Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 7/147 : Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 : Conference systems
• CPC Class Codes: H04N 7/52 : Systems for transmission of a pulse code modulated video signal with one or more other pulse code modulated signals, e.g. an audio signal or a synchronizing signal assembling of a multiplex stream by combining a video stream with other content or additional data, remultiplexing of multiplex streams, insertion of stuffing bits into the multiplex stream, assembling of a packetised elementary stream at server side H04N21/236; disassembling of a multiplex stream, remultiplexing of multiplex streams, extraction or processing of Service Information, disassembling of packetised elementary stream at client side H04N21/434

Abstract

The present invention is embodied in a digital communication system where multiple media data sources are time multiplexed into a packetized data stream, each packet having an assigned priority and the packetized data stream transmitted in substantially the order of assigned priority. At both the transmit side, and the receive side, audio packets are given priority processing over video packets, which in turn have priority over text/graphics data packets. Continuous real time audio playback is maintained at the receiver by delaying the playback of received audio in a first in/first out (FIFO) buffer providing a delay at least equal to the predicted average packet delay for the communication system. Optionally, the average system delay is continuously monitored, and the audio playback delay time is adjusted accordingly. Audio playback is slowed or accelerated in order to shrink or grow the difference in time between the sender and receiver. In another aspect of the invention, a conference of three or more callers is created by broadcasting a common packetized data stream to all conference callers.

Claims

1. A method of communicating media data packets from a first processing machine to at least one remote processing machine connected to said first processing machine over a communication system, said method comprising the steps of:

enabling at said first processing machine a connection processing stream for said remote processing machine with which said first processing machine communicates over said system during a connection;

enabling at said first processing machine at least one media processing stream associated with said connection processing stream;

establishing a relationship in said first processing machine which associates a media processing stream located in said remote processing machine with said connection processing stream in said first processing machine;

processing with said media processing stream in said first machine at least one type of media data to produce media data packets;

determining a communication path for sending said media data packets to said media processing stream located at said remote processing machine and through said connection stream at said first process machine based on said established relationship;

transmitting said media data packets to said media processing stream in said remote processing machine through said communication path;

receiving an acknowledgment message from said remote processing machine containing information representing when at least one media data packet from said first processing machine was processed at said remote processing machine;

using the received acknowledgment message to determine when a remote processing machine received and processed a media data packet sent to it from said first processing machine.

2. A method as in claim 1 wherein the media data packets transmitted and received by said first processing machine are multimedia data packets.

3. A method as in claim 2 wherein said multimedia data packets include audio and video data packets.

4. A method as in claim 3 wherein said multimedia data packets further include text/graphics data packets.

5. A method of communicating media data packets from a first processing machine to at least one remote processing machine over a communication system, said method comprising:

enabling at said first processing machine a connection processing stream for said remote processing machine with which said first processing machine communicates over said system during a connection;

enabling at said first processing machine at least one media processing stream associated with said connection processing stream;

establishing a relationship in said first processing machine which associates a media processing stream located in said remote processing machine with said connection processing stream in said first processing machine;

processing with said media processing stream in said first processing machine at least one type of media data to produce media data packets;

determining a communication path for sending said media data packets to said media processing stream located at said one remote processing machine and through said connection stream at said first processing machine based on said established relationship;

transmitting said media data packets to said media processing stream in said remote processing machine through said communication path;

storing in said first processing machine a first time relating to a first event associated with the transmission of at least one of said transmitted media data packets,

recording in said remote processing machine a second time relating to a second event associated with the transmission of said at least one of said transmitted media data packets,

comparing said first time to said second time to determine end-to-end system delay; and

receiving an acknowledgment message from said remote processing machine containing information related to said end-to-end system delay.

6. A method in accordance with claim 5, wherein said media data is audio data and said media data packets are audio data packets.

7. A method in accordance with claim 5, wherein plural types of media data packets are transmitted by said first processing machine, including audio data packets and video data packets, said method further comprising:

providing an association between a source of audio data and a source of video data and respective audio and video media processing streams which respectively provide audio and video data packets; and

transmitting each of said audio and video data packets through said connection processing stream associated with said audio and video media processing streams to audio and video data packet processing streams in said remote processing machine in accordance with a relationship established in said first processing machine which associates the audio or video data packet processing stream in said remote machine with said connection processing stream.

8. A method in accordance with claim 7, wherein said plural types of media data packets further includes text/graphics data packets, said method further comprising:

providing an association between a source of text/graphics data and a respective text/graphics media processing stream which provides text/graphics data packets; and

transmitting said text/graphics data packets through said associated connection processing stream in said first processing machine to a text/graphics media processing stream in said remote processing machine in accordance with a relationship established in said first processing machine which associates the audio and video data packet processing streams in said remote machine with said connection processing stream.

9. A method in accordance with claim 5, wherein said first processing machine receives media data packets from said remote processing machine, said method further comprising:

receiving at said connection processing stream in said first processing machine media data packets directed thereto from said remote processing machine; and

directing received media data packets from said connection processing stream to a receiving media data processing stream for processing the same.

10. A method as in claim 9, further comprising the step of:

delaying received media packets, said delay being set in accordance with an end to end system delay.

11. A method as in claim 10, wherein setting said delay further comprises:

increasing and decreasing said delay to be substantially equal to said determined end to end system delay,

wherein the delay of said received media packets increases and decreases substantially in accordance with said end to end system delay.

12. A method as in claim 11, wherein said received media data packets are audio data packets.

13. A method as in claim 11, wherein said received media data packets are video data packets.

14. A method as in claim 11, wherein said received media data packets are text/graphics data packets.

15. A method as in claim 5, wherein said first event comprises storing said at least one media data packet in a buffer associated with said first processing machine, and said second event comprises processing at least a portion of said at least one media data packet in said remote processing machine.

16. A method as in claim 5, wherein said first event comprises storing said at least one media data packet in a buffer associated with said first processing machine, and said second event comprises storing said at least one media data packet in a buffer associated with said remote processing machine.

17. A method as in claim 5, wherein said act of comparing comprises calculating the difference between said first time and said second time.

18. A method of communicating packets associated with media data from a first processing machine to at least one remote processing machine over a communication system, said method comprising:

enabling at said first processing machine a connection processing stream for said remote processing machine with which said first processing machine communicates over said system during a connection;

enabling at said first processing machine at least one media processing stream associated with said connection processing stream;

establishing a relationship in said first processing machine which associates a media processing stream located in said remote processing machine with said connection processing stream in said first processing machine;

processing with said media processing stream in said first processing machine at least one type of media data to produce packets associated with media data;

determining a communication path for sending said packets associated with media data to said media processing stream located at said one remote processing machine and through said connection stream at said first processing machine based on said established relationship;

transmitting said packets associated with media data to said media processing stream in said remote processing machine through said communication path;

recording in said first processing machine a time of transmission for at least one of said packets associated with media data;

receiving an acknowledgment message from said remote processing machine containing information representing a receiver event time related to when said at least one packet associated with media data sent from said first processing machine was processed at said remote processing machine; and

determining at said first processing machine a system path delay based on said recorded time of transmission of said at least one packet associated with data media and said information in said acknowledgment message.

19. A method as in claim 18, wherein said media data is audio data and said packets associated with media data are audio data packets.

20. A method in accordance with claim 19, wherein plural types of packets associated with media data are transmitted by said first processing machine, including audio data packets and video data packets, said method further comprising:

providing an association between a source of audio data and a source of video data and respective audio and video media processing streams which respectively provide audio and video data packets; and

transmitting each of said audio and video data packets through said connection processing stream associated with said audio and video media processing streams to audio and video data packet processing streams in said remote processing machine in accordance with a relationship established in said first processing machine which associates the audio or video data packet processing stream in said remote machine with said connection processing stream.

21. A method in accordance with claim 20, wherein said plural types of packets associated with media data further includes text/graphics data packets, said method further comprising:

providing an association between a source of text/graphics data and a respective text/graphics media processing stream which provides text/graphics data packets; and

transmitting said text/graphics data packets through said associated connection processing stream in said first processing machine to a text/graphics media processing stream in said remote processing machine in accordance with a relationship established in said first processing machine which associates the audio and video data packet processing streams in said remote machine with said connection processing stream.

22. A method in accordance with claim 21, wherein said first processing machine receives packets associated with media data from said remote processing machine, said method further comprising:

receiving at said connection processing stream in said first processing machine packets associated with media data directed thereto from said remote processing machine; and

directing received packets associated with media data from said connection processing stream to a receiving media data processing stream for processing the same.

23. A method as in claim 22, further comprising the step of:

delaying received media packets, said delay being set in accordance with an end to end system delay.

24. A method as in claim 23, wherein setting said delay further comprises:

increasing and decreasing said delay to be substantially equal to said determined end to end system delay,

wherein the delay of said received media packets increases and decreases substantially in accordance with said end to end system delay.

25. A method as in claim 24, wherein said received packets associated with media data are audio data packets.

26. A method as in claim 24, wherein said received packets associated with media data are video data packets.

27. A method as in claim 24, wherein said received packets associated with media data are text/graphics data packets.