Method and apparatus for multiple media digital communication system

• US 6,738,357 B1
• Filed: 06/19/2000
• Issued: 05/18/2004
• Est. Priority Date: 06/09/1993
• Status: Expired due to Fees

• Current Assignee: RPX Corporation
• Original Assignee: BTG International Inc. (Boston Scientific Corporation), Micron Technology, Inc.
• Inventors: Reed, Jr., Ogden Cartwright, Richter, Andreas
• Primary Examiner(s): MARCELO, MELVIN C
• Application Number:
  US09/596,835
• Time in Patent Office: 1,429 Days
• Field of Search: 370/263, 370/412, 345/753
• US Class Current: 370/263
• CPC Class Codes: H04L 12/18 for broadcast or conference , e.g. multicast
• CPC Class Codes: H04L 12/1813 for computer conferences, e.g. chat rooms instant messaging H04L51/04; protocols for multimedia communication H04L65/1101; arrangements for multi-party communication H04L65/403; telephonic conference arrangements H04M3/56; television conference systems H04N7/15
• CPC Class Codes: H04L 12/1827 Network arrangements for conference optimisation or adaptation
• CPC Class Codes: H04L 12/1881 with schedule organisation, e.g. priority, sequence management
• CPC Class Codes: H04L 12/6418 Hybrid transport
• CPC Class Codes: H04L 12/66 Arrangements for connecting between networks having differing types of switching systems, e.g. gateways
• CPC Class Codes: H04L 2012/6464 Priority
• CPC Class Codes: H04L 2012/6481 Speech, voice
• CPC Class Codes: H04L 2012/6483 Video, e.g. MPEG
• CPC Class Codes: H04L 2012/6489 Buffer Management, Threshold setting, Scheduling, Shaping
• CPC Class Codes: H04L 47/15 in relation to multipoint traffic arrangements for broadcast or multicast in data networks H04L12/18
• CPC Class Codes: H04L 47/22 Traffic shaping
• CPC Class Codes: H04L 47/2416 Real-time traffic
• CPC Class Codes: H04L 47/2433 Allocation of priorities to traffic types
• CPC Class Codes: H04L 47/283 in response to processing delays, e.g. caused by jitter or round trip time [RTT]
• CPC Class Codes: H04L 47/50 Queue scheduling
• CPC Class Codes: H04L 47/56 implementing delay-aware scheduling
• CPC Class Codes: H04L 47/6215 Individual queue per QOS, rate or priority
• CPC Class Codes: H04L 47/6225 Fixed service order, e.g. Round Robin
• CPC Class Codes: H04L 47/624 Altering the ordering of packets in an individual queue
• CPC Class Codes: H04L 47/6285 : Provisions for avoiding starvation of low priority queues
• CPC Class Codes: H04L 49/90 : Buffering arrangements
• CPC Class Codes: H04L 65/1101 : Session protocols
• CPC Class Codes: H04L 65/4038 : with floor control
• CPC Class Codes: H04L 65/611 : for multicast or broadcast systems for broadcast or conference H04L12/18; arrangements for broadcast or distribution combined with broadcast H04H20/00; arrangements for broadcast applications with a direct linkage to broadcast information or to broadcast space-time H04H60/00; selective distribution of broadcast services, e.g. multimedia broadcast multicast service [MBMS], H04W4/06
• CPC Class Codes: H04L 65/752 : adapting media to network capabilities
• CPC Class Codes: H04L 65/80 : Responding to QoS
• CPC Class Codes: H04L 9/40 : Network security protocols
• CPC Class Codes: H04M 2203/2011 : Service processing based on information specified by a party before or during a call, e.g. information, tone or routing selection
• CPC Class Codes: H04M 3/436 : Arrangements for screening incoming calls , i.e. evaluating the characteristics of a call before deciding whether to answer it based on the calling party profile H04M3/42059; based on location H04M3/42348; based on presence H04M3/42365; diversion H04M3/54
• CPC Class Codes: H04M 3/563 : User guidance or feature selection
• CPC Class Codes: H04M 3/567 : Multimedia conference systems
• CPC Class Codes: H04M 7/006 : Networks other than PSTN/ISDN providing telephone service, e.g. Voice over Internet Protocol (VoIP) , including next generation networks with a packet-switched transport layer H04L65/00 takes precedence; aspects not specific to the type of network H04M3/00; special services in those networks H04M3/42
• CPC Class Codes: H04N 21/23614 : Multiplexing of additional data and video streams arrangements for simultaneous broadcast of plural pieces of information H04H20/28
• CPC Class Codes: H04N 21/2368 : Multiplexing of audio and video streams
• CPC Class Codes: H04N 21/43072 : of multiple content streams on the same device
• CPC Class Codes: H04N 21/4341 : Demultiplexing of audio and video streams
• CPC Class Codes: H04N 21/4348 : Demultiplexing of additional data and video streams
• CPC Class Codes: H04N 21/439 : Processing of audio elementary streams
• CPC Class Codes: H04N 21/44004 : involving video buffer management, e.g. video decoder buffer or video display buffer
• CPC Class Codes: H04N 21/4788 : communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 21/6375 : for requesting retransmission , e.g. of data packets lost or corrupted during transmission from serverARQ protocols H04L1/18; implementation or adaptation of Internet protocol [IP], of transmission control protocol [TCP] or of user datagram protocol [UDP] H04L69/16
• CPC Class Codes: H04N 21/6583 : Acknowledgement
• CPC Class Codes: H04N 7/147 : Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 : Conference systems
• CPC Class Codes: H04N 7/52 : Systems for transmission of a pulse code modulated video signal with one or more other pulse code modulated signals, e.g. an audio signal or a synchronizing signal assembling of a multiplex stream by combining a video stream with other content or additional data, remultiplexing of multiplex streams, insertion of stuffing bits into the multiplex stream, assembling of a packetised elementary stream at server side H04N21/236; disassembling of a multiplex stream, remultiplexing of multiplex streams, extraction or processing of Service Information, disassembling of packetised elementary stream at client side H04N21/434

Abstract

The present invention is embodied in a digital communication system where multiple media data sources are time multiplexed into a packetized data stream, each packet having an assigned priority and the packetized data stream transmitted in substantially the order of assigned priority. At both the transmit side, and the receive side, audio packets are given priority processing over video packets, which in turn have priority over text/graphics data packets. Continuous real time audio playback is maintained at the receiver by delaying the playback of received audio in a first in/first out (FIFO) buffer providing a delay at least equal to the predicted average packet delay for the communication system. Optionally, the average system delay is continuously monitored, and the audio playback delay time is adjusted accordingly. Audio playback is slowed or accelerated in order to shrink or grow the difference in time between the sender and receiver. In another aspect of the invention, a conference of three or more callers is created by broadcasting a common packetized data stream to all conference callers.

Claims

1. A method for communicating between first, second and third callers over a packet switched network, said method comprising:

invoking at least one connection routine at said first caller to cause control messages to be sent to respective connection routines at said second and third callers to enable a media type selection for each of said second and third callers, at least one of said media type selections including a selection of at least two different media types; and

generating at said first caller digital media packets corresponding to each of said at least two different media types;

wherein, for each of said at least two media types, said plurality of corresponding digital packets are formed into a common data stream, which is transmitted from said first caller to be delivered to and processed by said second and third callers in accordance with said media type selections, said common data stream being transmitted over said network via one or more packet switched communication channels, said one or more communication channels providing indeterminate system delays and bandwidth limitations that give rise to indeterminate packet loss.

2. A method in accordance with claim 1, wherein said method further comprises:

assigning a priority to each of said digital media packets at said first caller;

wherein said digital media packets are transmitted from said first caller in substantially said order of said assigned priorities.

3. A method in accordance with claim 1, wherein said selections are made by said first and second callers.

4. A method in accordance with claim 1, wherein said digital media data packets include video packets and audio packets.

5. A method in accordance with claim 2, wherein said step of transmitting said plurality of digital packets over said communication system in substantially said order of said assigned priority further comprises:

providing a queue for holding said plurality of digital packets respectively corresponding to each of said plurality of digital media prior to transmission over said communication system;

placing successive ones of said plurality of digital packets in said queue in order said assigned priority; and

transmitting the first digital packet of said queue over said communication system.

6. A method in accordance with claim 1, wherein said digital media data packets include video packets, audio packets, and collaborative data packets.

7. A method in accordance with claim 1, wherein said media type selections are bidirectional media type selections.

8. A method in accordance with claim 1, wherein said media type selection for said second caller is the same as said media type selection for said third caller.

9. A method in accordance with claim 1, wherein said media type selection for said second caller is different than said media type selection for said third caller.

10. A method in accordance with claim 4, wherein said video packets are assigned a lower priority than said audio packets.

11. A method in accordance with claim 6, wherein said collaborative data packets are assigned a lower priority than said video packets.

12. A method in accordance with claim 11, wherein said collaborative data packets comprise collaborative application data packets, and said video packets are assigned a lower priority than said audio packets.

13. An apparatus for communicating with a remote processing machine over a packet switched network, said apparatus comprising:

at least one processor; and

at least one connection routine running on said at least one processor, said at least one connection routine being operative to negotiate with said remote processing machine a selection of at least one media type from a plurality of media types including audio, video and data, and to configure, according to said selection, at least one media routine running on said at least one processor to process media data packets received from and to be transmitted to said remote processing machine over said packet switched network via one or more packet switched communication channels, said one or more communication channels providing indeterminate system delays and bandwidth limitations that give rise to indeterminate packet loss;

wherein a number of media types that said apparatus is capable of processing and a number of media types that said remote processing machine selects may be the same or different, and wherein the apparatus sends a first message to the remote processing machine indicating an available set of media types and the remote processing machine sends back an indication of the selection in a second message sent in response to the first message.

14. An apparatus for communicating with a remote processing machine over a packet switched network, said apparatus comprising:

at least one processor; and

at least one connection routine running on said at least one processor, said at least one connection routine being operative to negotiate with said remote processing machine a selection of at least one media type from a plurality of media types including audio, video and data, and to configure, according to said selection, at least one media routine running on said at least one processor to process media data packets received from and to be transmitted to said remote processing machine over said packet switched network via one or more packet switched communication channels, said one or more communication channels providing indeterminate system delays and bandwidth limitations that give rise to indeterminate packet loss;

wherein said selection is of only audio, and said negotiation comprises sending a first message and receiving a response message, said negotiation involving a message format that supports the description and selection of the audio, video and data media types.

15. The apparatus of claim 14, wherein said selection is made by said remote processing machine.

16. A method of transmitting data packets from a first processing machine to one or more of a plurality of remote processing machines in data communication therewith, comprising:

providing at least one source of media data;

processing said media data to produce a plurality of data packets, each of said packets having a plurality of data fields associated therewith, at least one of said data fields comprising at least one destination address associated with a respective one of said plurality of remote processing machines; and

transmitting each of said plurality of packets to at least one of said plurality of remote processing machines over switched packet network via one or more packet switched communication channels, said one or more communication channels providing indeterminate system delays and bandwidth limitations that give rise to indeterminate packet loss;

wherein a first number of said plurality of packets may be transmitted for delivery to a first subset of said plurality of remote processing machines while a second number of said plurality of packets may be transmitted for delivery to a second subset of said plurality of remote processing machines, said first and second subsets not being identical.

17. The method of claim 16, wherein said first subset comprises all of said plurality of remote processing machines, and said second subset comprises a lesser number thereof.

18. The method of claim 16, further comprising generating, for each of said plurality of packets, a count of the number of said plurality of remote processing machines to which said packet was delivered, but not received.

19. A method of controlling the transmission of media data packets across a switched packet network from a first processing machine to a remote processing machine, said method comprising:

generating a first plurality of media data packets based on media data received from a media data source;

transmitting said first plurality of packets from said first processing machine to said remote processing machine over said network via one or more packet switched communication channels, said one or more communication channels providing indeterminate system delays and bandwidth limitations that give rise to indeterminate packet loss;

for at least one of said first plurality of media data packets, determining a delay from transmission of said packet from the first processing machine to receipt of said packet by the remote processing machine; and

adjusting the sampling rate of said media data packets at said remote processing machine based on said delay.

20. The method of claim 19, wherein said adjusting comprises performing audio re-sampling.

21. The method of claim 19, further comprising buffering said media data packets received at said remote processing machine for a period related to said delay.

22. The method of claim 21, wherein said buffering further comprises selecting a buffer adapted to store a number of said media data packets greater than or equal to a rate of receipt of said media data packets at said remote processing machine multiplied by said delay.

23. A method of controlling the transmission of media data packets across a digital data network from a first processing machine to a remote processing machine, comprising:

generating a first plurality of media data packets based on media data received from a media data source;

transmitting said first plurality of packets from said first processing machine to said remote processing machine;

determining a parameter associated with transmission quality associated with at least one of said first plurality of packets; and

adjusting a playback rate of said media data packets at said remote processing machine based on said parameter.

24. The method of claim 23, wherein said parameter associated with transmission quality comprises a delay from transmission of said packets from said first processing machine to receipt of said packets by said remote processing machine.

25. A method for communicating over a packet switched network, said method comprising:

invoking a connection routine at a first caller to cause control messages to be sent to respective second and third callers to enable a media type selection for each of said second and third callers, and in response to said selections, receiving from said second and third callers media type selection control messages used to negotiate respective sets of media type selections, at least one of said sets of media type selections including at least a selection of two different media types;

generating at said first caller digital media packets corresponding to each of said at least two different media types; and

transmitting from said first caller said digital media packets for delivery to and processing by said second and third callers in accordance with said media type selections,

wherein said media packets are transmitted over said network via one or more packet switched communication channels, said one or more communication channels providing indeterminate system delays and bandwidth limitations that give rise to indeterminate packet loss.

26. The method in accordance with claim 25, wherein said selections are made by said first and second callers.

27. The method in accordance with claim 25, wherein said media type selections are bidirectional media type selections.