Real-time priority-based media communication

• US 8,065,426 B2
• Filed: 09/04/2009
• Issued: 11/22/2011
• Est. Priority Date: 02/13/2003
• Status: Expired due to Fees

• Current Assignee: Adobe Inc.
• Original Assignee: Adobe Systems Incorporated (Adobe Inc.)
• Inventors: Edelman, Bradley A., Lozben, Slavik, Shetty, Pritham, Gay, Jonathan
• Primary Examiner(s): Chea; Philip
• Application Number:
  US12/554,775
• Publication Number:
  US 20090327510A1
• Time in Patent Office: 809 Days
• Field of Search: 709/219, 709/231, 341/60, 725/103
• US Class Current: 709/231
• CPC Class Codes: H04L 47/10 Flow control; Congestion control
• CPC Class Codes: H04L 47/2416 Real-time traffic
• CPC Class Codes: H04L 47/2433 Allocation of priorities to traffic types
• CPC Class Codes: H04L 65/60 Network streaming of media packets
• CPC Class Codes: H04L 65/65 Network streaming protocols, e.g. real-time transport protocol [RTP] or real-time control protocol [RTCP]
• CPC Class Codes: H04L 65/762 at the source reformatting of additional data in video distribution servers H04N21/2355
• CPC Class Codes: H04N 21/2353 specifically adapted to content descriptors, e.g. coding, compressing or processing of metadata
• CPC Class Codes: H04N 21/2389 Multiplex stream processing, e.g. multiplex stream encrypting
• CPC Class Codes: H04N 21/262 Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists scheduling strategies for dispatcher in multiprogramming arrangements G06F9/4881; arrangements for scheduling broadcast services or broadcast-related services H04H60/06; flow control in packet networks H04L47/10; establishing a time schedule or organising the servicing of application requests H04L67/62
• CPC Class Codes: H04N 7/173 with two-way working, e.g. subscriber sending a programme selection signal

Abstract

Representative embodiments are disclosed of a real-time priority-based communication system and method for communicating media streams made up of multiple media message sub-streams, the communication system comprising a chunk configurator for dividing media message sub-streams into chunks, a state machine configured to translate between information regarding the media message sub-streams and the chunks and state associations to the information, the state associations written into a header for each of the chunks, a queue for holding the chunks waiting to be transmitted, and a processor for executing a scheduling algorithm, wherein the scheduling algorithm determines ones of the chunks in the queue to transmit next.

Claims

1. A non-transitory computer readable medium storing a program to cause at least one processor to perform operations comprising:

obtaining one or more streams;

processing the obtained one or more streams, wherein the processing includes;

dividing each of the one or more streams into a plurality of chunks, wherein each chunk of the plurality of chunks has an associated data payload and an associated chunk control byte including a chunk type identifier,

if a chunk of the plurality of chunks is associated with a new stream from among the one or more streams, writing to the chunk control byte of the chunk

a new stream chunk type identifier as the chunk type identifier, and

rendering information relating to a media rendering state corresponding to the new stream chunk comprising a plurality of new parameters, wherein the media rendering state corresponding to the new stream chunk uses no parameters from a previous chunk of the one or more streams,

if the chunk is associated with a new media of the new stream, writing to the chunk control byte of the chunk

a new media chunk type identifier as the chunk type identifier, and

rendering information relating to a media rendering state corresponding to the new media chunk comprising a plurality of new parameters, wherein the media rendering state corresponding to the new media chunk uses at least one other parameter from a previous chunk of the new stream,

if the chunk is associated with a new time of the new stream, writing to the chunk control byte of the chunk

a new time chunk type identifier as the chunk type identifier, and

rendering information relating to a media rendering state corresponding to the new time chunk comprising at least one new parameter, wherein the media rendering state corresponding to the new time chunk uses a plurality of other parameters from a previous chunk of the new stream, and

else, writing to the chunk control byte a continuation chunk type identifier as the chunk type identifier, wherein a media rendering state corresponding to the continuation chunk uses rendering information from previous chunks; and

outputting the processed one or more streams.

2. The computer readable medium of claim 1, wherein the parameters include one or more of a stream ID parameter, a length parameter, a media type parameter, and a time delta parameter.

3. The computer readable medium of claim 1, the operations further comprising:

writing, to the associated chunk control byte of each of the plurality of chunks, an associated chunk identifier to indicate whether the associated data payload includes low-level protocol instructions or media content.

4. The computer readable medium of claim 1, wherein the outputting comprises:

storing the plurality of chunks of the processed one or more streams to a storage device.

5. The computer readable medium of claim 1, wherein the outputting comprises:

playing the plurality of chunks of the processed one or more streams by a media player of a computing device.

6. The computer readable medium of claim 5, wherein the playing comprises:

reading, from a chunk control byte of a chunk from among the plurality of chunks, rendering information relating to the media rendering state associated with the chunk and one or more subsequent chunks; and

controlling a media rendering state for the media player based on the information read from the chunk control byte for playing the chunk and one or more subsequent chunks.

7. The computer readable medium of claim 1, wherein the outputting comprises:

streaming the plurality of chunks of the processed one or more streams from a media streaming server computer.

8. The computer readable medium of claim 7, wherein the streaming comprises:

queuing the plurality of chunks for transmission based on a media type of each of the plurality of chunks; and

transmitting ones of the queued plurality of chunks based on a scheduling algorithm.

9. The computer readable medium of claim 8, wherein the queuing comprises:

staging the plurality of chunks according to a media type of the chunk, wherein the media type comprises one of an audio format chunk, a video format chunk, and a data format chunk.

10. The computer readable medium of claim 8, wherein the queuing comprises:

assigning a priority to each of the one or more streams;

totaling a priority for each stream;

calculating a desired transmission rate for each of the one or more streams using the assigned priority and the totaled priority;

counting a sub-total of the plurality of associated chunks transmitted per the one or more streams during a preset period;

counting a total number of the plurality of associated chunks transmitted during the preset period;

calculating an actual transmission rate for each stream of the one or more streams using the sub-total and the total; and

calculating a ratio between the desired transmission rate and the actual transmission rate.

11. A system comprising:

a storage device; and

one or more computer systems communicatively coupled with the storage device and operable to interact with a user device, the one or more computer systems being configured to perform operations including;

obtaining one or more streams;

processing the obtained one or more streams, wherein the processing includes;

dividing each of the one or more streams into a plurality of chunks, wherein each chunk has an associated data payload and an associated chunk control byte including a chunk type identifier,

if a chunk of the plurality of chunks is associated with a new stream from among the one or more streams, writing to the chunk control byte of the chunk

a new stream chunk type identifier as the chunk type identifier, and

rendering information relating to a media rendering state corresponding to the new stream chunk comprising a plurality of new parameters, wherein the media rendering state corresponding to the new stream chunk uses no parameters from a previous chunk of the one or more streams,

if the chunk is associated with a new media of the new stream, writing to the chunk control byte of the chunk

a new media chunk type identifier as the chunk type identifier, and

rendering information relating to a media rendering state corresponding to the new media chunk comprising a plurality of new parameters, wherein the media rendering state corresponding to the new media chunk uses at least one other parameter from a previous chunk of the new stream,

if the chunk is associated with a new time of the new stream, writing to the chunk control byte of the chunk

a new time chunk type identifier as the chunk type identifier, and

rendering information relating to a media rendering state corresponding to the new time chunk comprising at least one new parameter, wherein the media rendering state corresponding to the new time chunk uses a plurality of other parameters from a previous chunk of the new stream, and

else, writing to the chunk control byte a continuation chunk type identifier as the chunk type identifier, wherein a media rendering state corresponding to the continuation chunk uses rendering information from previous chunks; and

outputting the processed one or more streams.

12. The system of claim 11, wherein the parameters include one or more of a stream ID parameter, a length parameter, a media type parameter, and a time delta parameter.

13. The system of claim 11, wherein the one or more computer systems consist of one computer system, the user device is a user interface device, and the one computer system comprises the user interface device,

wherein the one computer system is further configured to perform operations including storing the plurality of chunks of the processed one or more streams to the storage device.

14. The system of claim 11, wherein the one or more computer systems consist of one computer system, the user device is a user interface device, and the one computer system comprises the user interface device, wherein the one computer system is further configured to perform operations including:

playing the plurality of chunks of the processed one or more streams by using a media player of the user interface device, wherein the playing operation comprises operations including;

reading, from a chunk control byte of a chunk from among the plurality of chunks, rendering information relating to the media rendering state associated with the chunk and one or more subsequent chunks; and

controlling a media rendering state for the media player based on the information read from the chunk control byte for playing the chunk and one or more subsequent chunks.

15. The system of claim 11, wherein:

the one or more computer systems comprise a media streaming server operable to interact with the user device through a data communication network, the user device is operable to interact with the media streaming server as a client, and

the media streaming server is configured to perform said outputting the processed one or more streams at least in part by streaming the plurality of chunks of the processed one or more streams from the media streaming server to the client.

16. The system of claim 15, wherein the media streaming server is further configured to perform the streaming at least in part by:

queuing the plurality of chunks for transmission based on a media type of each of the plurality of chunks; and

transmitting ones of the queued plurality of chunks based on a scheduling algorithm.

17. The system of claim 16, wherein the media streaming server is further configured to perform the queuing at least in part by staging the plurality of chunks according to a media type of the chunk, wherein the media type comprises one of an audio format chunk, a video format chunk, and a data format chunk.

18. The system of claim 16, wherein the media streaming server is further configured to perform the queuing at least in part by:

assigning a priority to each of the one or more streams;

totaling a priority for each stream;

calculating a desired transmission rate for each of the one or more streams using the assigned priority and the totaled priority;

counting a sub-total of the plurality of associated chunks transmitted per the one or more streams during a preset period;

counting a total number of the plurality of associated chunks transmitted during the preset period;

calculating an actual transmission rate for each stream of the one or more streams using the sub-total and the total; and

calculating a ratio between the desired transmission rate and the actual transmission rate.

19. A method comprising:

obtaining one or more streams;

processing the obtained one or more streams, wherein the processing includes;

dividing each of the one or more streams into a plurality of chunks, wherein each chunk of the plurality of chunks has an associated data payload and an associated chunk control byte including a chunk type identifier,

if a chunk of the plurality of chunks is associated with a new stream from among the one or more streams, writing to the chunk control byte of the chunk

a new stream chunk type identifier as the chunk type identifier, and

rendering information relating to a media rendering state corresponding to the new stream chunk comprising a plurality of new parameters, wherein the media rendering state corresponding to the new stream chunk uses no parameters from a previous chunk of the one or more streams,

if the chunk is associated with a new media of the new stream, writing to the chunk control byte of the chunk

a new media chunk type identifier as the chunk type identifier, and

rendering information relating to a media rendering state corresponding to the new media chunk comprising a plurality of new parameters, wherein the media rendering state corresponding to the new media chunk uses at least one other parameter from a previous chunk of the new stream,

if the chunk is associated with a new time of the new stream, writing to the chunk control byte of the chunk

a new time chunk type identifier as the chunk type identifier, and

rendering information relating to a media rendering state corresponding to the new time chunk comprising at least one new parameter, wherein the media rendering state corresponding to the new time chunk uses a plurality of other parameters from a previous chunk of the new stream, and

else, writing to the chunk control byte a continuation chunk type identifier as the chunk type identifier, wherein a media rendering state corresponding to the continuation chunk uses rendering information from previous chunks; and

streaming the processed one or more streams from a media streaming server system.

20. The method of claim 19, wherein the

parameters include one or more of a stream ID parameter, a length parameter, a media type parameter, and a time delta parameter.

21. The method of claim 19, wherein the streaming of the processed one or more streams comprises:

queuing the plurality of chunks for transmission based on a media type of each of the plurality of chunks, said queuing including staging the plurality of chunks according to a media type of the chunk, wherein the media type comprises one of an audio format chunk, a video format chunk, and a data format chunk; and

transmitting ones of the queued plurality of chunks based on a scheduling algorithm.

22. The method of claim 21, wherein the queuing comprises:

assigning a priority to each of the one or more streams;

totaling a priority for each stream;

calculating a desired transmission rate for each of the one or more streams using the assigned priority and the totaled priority;

counting a sub-total of the plurality of associated chunks transmitted per the one or more streams during a preset period;

counting a total number of the plurality of associated chunks transmitted during the preset period;

calculating an actual transmission rate for each stream of the one or more streams using the sub-total and the total; and

calculating a ratio between the desired transmission rate and the actual transmission rate.

23. The method of claim 19, further comprising storing the processed one or more streams to a storage system.

24. The method of claim 19, further comprising:

playing the processed one or more streams by using a media player of a computer system, said playing comprising, for each of the processed one or more streams,

reading, from a chunk control byte of a chunk from among the plurality of chunks of the processed stream, rendering information relating to the media rendering state associated with the chunk and one or more subsequent chunks; and

controlling a media rendering state for the media player based on the information read from the chunk control byte for playing the chunk and one or more subsequent chunks.

25. The method of claim 19, further comprising writing, to the associated chunk control byte of each of the plurality of chunks, an associated chunk identifier to indicate whether the associated data payload includes low-level protocol instructions or media content.

26. A non-transitory computer readable medium storing a program to cause at least one processor to perform operations comprising:

receiving a plurality of chunks associated with one or more streams, wherein each chunk of the plurality of chunks has an associated data payload and an associated chunk control byte including rendering information relating to a media rendering state associated with one chunk and one or more subsequent chunks, such that a portion of the rendering information that remains unchanged from the one chunk to a subsequent chunk is included in the chunk control byte of the one chunk but is missing from the chunk control byte of the subsequent chunk, wherein the chunk control byte includes a chunk type identifier;

for each chunk of the plurality of chunks, determining playback information associated with the chunk at least in part by

reading the rendering information from the associated chunk control byte,

designating the rendering information read from the associated chunk control byte as the playback information associated with the chunk, and

in response to determining that a portion of the rendering information from the associated chunk control byte is missing, adding to the playback information associated with the chunk a corresponding portion of rendering information read from a chunk control byte of a preceding chunk;

reassembling the one or more streams based on respective playback information associated with each of the plurality of chunks; and

issuing the reassembled one or more streams.

27. The computer readable medium of claim 26, wherein said determining the playback information associated with the chunk comprises:

if the associated chunk type identifier is a new stream chunk type identifier,

associating the chunk with a new stream from among the one or more streams, and

reading from the associated chunk control byte rendering information relating to a media rendering state corresponding to a new stream chunk comprising a new plurality of parameters, wherein the media rendering state corresponding to the new stream chunk uses no parameters from a previous chunk of the one or more streams;

if the associated chunk type identifier is a new media chunk type identifier,

associating the chunk with a new media of the new stream, and

reading from the associated chunk control byte rendering information relating to a media rendering state corresponding to a new media chunk comprising a new plurality of parameters, wherein the media rendering state corresponding to the new media chunk uses at least one other parameter from a previous chunk of the new stream;

if the associated chunk type identifier is a new time chunk type identifier,

associating the chunk with a new time of the new stream, and

reading from the associated chunk control byte rendering information relating to a media rendering state corresponding to a new time chunk comprising at least one parameter that is new, wherein the media rendering state corresponding to the new time chunk uses a plurality of parameters from a previous chunk of the new stream; and

else if the associated chunk type identifier is a continuation chunk type identifier, associating the chunk with the new stream, wherein a media rendering state corresponding to a continuation chunk uses a plurality of parameters from previous chunks of the new stream.

28. The computer readable medium of claim 27, wherein the parameters include one or more of a stream ID parameter, a length parameter, a media type parameter, and a time delta parameter.

29. The computer readable medium of claim 27, wherein said reassembling the one or more streams comprises selecting, at least in part by using a chunk type identifier associated with a chunk of the new stream, parameters from the previous one or more chunks of the new stream for use with respect to one or more subsequent chunks of the new stream.

30. The computer readable medium of claim 29, wherein said issuing the reassembled one or more streams comprises controlling a state machine to set the selected parameters for use by the one or more subsequent chunks of the new stream.

31. The computer readable medium of claim 27, wherein said determining the playback information associated with the chunk comprises

reading a stream identifier from the associated chunk control byte, and

determining, at least in part by using the associated chunk type identifier, whether the chunk is a low-level protocol message or a media content message.

32. The computer readable medium of claim 27, wherein said determining the playback information associated with the chunk comprises

reading a stream identifier from the associated chunk control byte, and

determining, at least in part by using the associated chunk type identifier, a quantity of bytes of the associated chunk control byte used to identify a stream with which the chunk is associated.

33. The computer readable medium of claim 26, wherein said issuing the reassembled one or more streams comprises storing the reassembled one or more streams on a storage system.

34. The computer readable medium of claim 26, wherein said issuing the reassembled one or more streams comprises playing back the reassembled one or more streams by a media player of a computing device.

35. A system comprising:

one or more processors and storage configured to store instructions, that when executed by the one or more processors cause the system to perform operations comprising;

receiving a plurality of chunks associated with one or more streams, wherein each chunk of the plurality of chunks has an associated data payload and an associated chunk control byte including rendering information relating to a media rendering state associated with one chunk and one or more subsequent chunks, such that a portion of the rendering information that remains unchanged from the one chunk to a subsequent chunk is included in the chunk control byte of the one chunk but is missing from the chunk control byte of the subsequent chunk, wherein the chunk control byte includes a chunk type identifier;

for each chunk of the plurality of chunks, determining playback information associated with the chunk at least in part by

reading the rendering information from the associated chunk control byte,

designating the rendering information read from the associated chunk control byte as the playback information associated with the chunk, and

in response to determining that a portion of the rendering information from the associated chunk control byte is missing, adding to the playback information associated with the chunk a corresponding portion of rendering information read from a chunk control byte of a preceding chunk;

reassembling the one or more streams based on respective playback information associated with each of the plurality of chunks; and

issuing the reassembled one or more streams.

36. The system of claim 35, wherein said determining the playback information associated with the chunk comprises:

if the associated chunk type identifier is a new stream chunk type identifier,

associating the chunk with a new stream from among the one or more streams, and

reading from the associated chunk control byte rendering information relating to a media rendering state corresponding to a new stream chunk comprising a new plurality of parameters, wherein the media rendering state corresponding to the new stream chunk uses no parameters from a previous chunk of the one or more streams;

if the associated chunk type identifier is a new media chunk type identifier,

associating the chunk with a new media of the new stream, and

reading from the associated chunk control byte rendering information relating to a media rendering state corresponding to a new media chunk comprising a new plurality of parameters, wherein the media rendering state corresponding to the new media chunk uses at least one other parameter from a previous chunk of the new stream;

if the associated chunk type identifier is a new time chunk type identifier,

associating the chunk with a new time of the new stream, and

reading from the associated chunk control byte rendering information relating to a media rendering state corresponding to a new time chunk comprising at least one parameter that is new, wherein the media rendering state corresponding to the new time chunk uses a plurality of other parameters from a previous chunk of the new stream; and

else if the associated chunk type identifier is a continuation chunk type identifier, associating the chunk with the new stream, wherein a media rendering state corresponding to a continuation chunk uses a plurality of parameters from previous chunks of the new stream.

37. The system of claim 36, wherein the parameters include one or more of a stream ID parameter, a length parameter, a media type parameter, and a time delta parameter.

38. The system of claim 36, wherein said reassembling the one or more streams comprises selecting, at least in part by using a chunk type identifier associated with a chunk of the new stream, parameters from the previous one or more chunks of the new stream for use with respect to one or more subsequent chunks of the new stream.

39. The system of claim 38, wherein said issuing the reassembled one or more streams comprises controlling a state machine to set the selected parameters for use by the one or more subsequent chunks of the new stream.

40. The system of claim 36, wherein said determining the playback information associated with the chunk comprises

reading a stream identifier from the associated chunk control byte, and

determining, at least in part by using the associated chunk type identifier, whether the chunk is a low-level protocol message or a media content message.

41. The system of claim 36, wherein said determining the playback information associated with the chunk comprises

reading a stream identifier from the associated chunk control byte, and

determining, at least in part by using the associated chunk type identifier, a quantity of bytes of the associated chunk control byte used to identify a stream with which the chunk is associated.

42. The system of claim 35, wherein said issuing the reassembled one or more streams comprises storing the reassembled one or more streams on the storage.

43. The system of claim 35, further comprising:

a display,

wherein said issuing the reassembled one or more streams comprises playing back the reassembled one or more streams on the display by a media player executing on the one or more processors.

44. A method comprising:

receiving a plurality of chunks associated with one or more streams, wherein each chunk of the plurality of chunks has an associated data payload and an associated chunk control byte including rendering information relating to a media rendering state associated with one chunk and one or more subsequent chunks, such that a portion of the rendering information that remains unchanged from the one chunk to a subsequent chunk is included in the chunk control byte of the one chunk but is missing from the chunk control byte of the subsequent chunk, wherein the chunk control byte includes a chunk type identifier;

for each chunk of the plurality of chunks, determining playback information associated with the chunk at least in part by

reading the rendering information from the associated chunk control byte,

designating the rendering information read from the associated chunk control byte as the playback information associated with the chunk, and

in response to determining that a portion of the rendering information from the associated chunk control byte is missing, adding to the playback information associated with the chunk a corresponding portion of rendering information read from a chunk control byte of a preceding chunk;

reassembling the one or more streams based on respective playback information associated with each of the plurality of chunks; and

issuing the reassembled one or more streams.

45. The method of claim 44, wherein said determining the playback information associated with the chunk comprises:

if the associated chunk type identifier is a new stream chunk type identifier,

associating the chunk with a new stream from among the one or more streams, and

reading from the associated chunk control byte rendering information relating to a media rendering state corresponding to a new stream chunk comprising a new plurality of parameters, wherein the media rendering state corresponding to the new stream chunk uses no parameters from a previous chunk of the one or more streams;

if the associated chunk type identifier is a new media chunk type identifier,

associating the chunk with a new media of the new stream, and

reading from the associated chunk control byte rendering information relating to a media rendering state corresponding to a new media chunk comprising a new plurality of parameters, wherein the media rendering state corresponding to the new media chunk uses at least one other parameter from a previous chunk of the new stream;

if the associated chunk type identifier is a new time chunk type identifier,

associating the chunk with a new time of the new stream, and

reading from the associated chunk control byte rendering information relating to a media rendering state corresponding to a new time chunk comprising at least one parameter that is new, wherein the media rendering state corresponding to the new time chunk uses a plurality of other parameters from a previous chunk of the new stream; and

else if the associated chunk type identifier is a continuation chunk type identifier, associating the chunk with the new stream, wherein a media rendering state corresponding to a continuation chunk uses a plurality of parameters from previous chunks of the new stream.

46. The method of claim 45, wherein the parameters include one or more of a stream ID parameter, a length parameter, a media type parameter, and a time delta parameter.

47. The method of claim 45, wherein said reassembling the one or more streams comprises selecting, at least in part by using a chunk type identifier associated with a chunk of the new stream, parameters from the previous one or more chunks of the new stream for use with respect to one or more subsequent chunks of the new stream.

48. The method of claim 47, wherein said issuing the reassembled one or more streams comprises controlling a state machine to set the selected parameters for use by the one or more subsequent chunks of the new stream.

49. The method of claim 45, wherein said determining the playback information associated with the chunk comprises

reading a stream identifier from the associated chunk control byte, and

determining, at least in part by using the associated chunk type identifier, whether the chunk is a low-level protocol message or a media content message.

50. The method of claim 45, wherein said determining the playback information associated with the chunk comprises

reading a stream identifier from the associated chunk control byte, and

determining, at least in part by using the associated chunk type identifier, a quantity of bytes of the associated chunk control byte used to identify a stream with which the chunk is associated.

51. The method of claim 44, wherein said issuing the reassembled one or more streams comprises storing the reassembled one or more streams on a storage system.

52. The method of claim 44, wherein said issuing the reassembled one or more streams comprises playing back the reassembled one or more streams by a media player of a computing device.