Interactive streaming media production tool using communication optimization

• US 7,376,153 B2
• Filed: 04/26/2004
• Issued: 05/20/2008
• Est. Priority Date: 04/23/2001
• Status: Active Grant

• Current Assignee: Telefonaktiebolaget LM Ericsson
• Original Assignee: Envivio Incorporated (Telefonaktiebolaget LM Ericsson)
• Inventors: MacAulay, Alexander, Coleman, Mike
• Primary Examiner(s): ABELSON, RONALD B
• Application Number:
  US10/831,960
• Publication Number:
  US 20050002401A1
• Time in Patent Office: 1,485 Days
• Field of Search: None
• US Class Current: 370/487
• CPC Class Codes: H04N 21/23805 Controlling the feeding rate to the network, e.g. by controlling the video pump
• CPC Class Codes: H04N 21/2402 Monitoring of the downstream path of the transmission network, e.g. bandwidth available traffic monitoring in data switching networks H04L43/00; monitoring data switching networks utilization H04L43/0876
• CPC Class Codes: H04N 21/2405 Monitoring of the internal components or processes of the server, e.g. server load allocation of resources in multiprogramming arrangements G06F9/50; performance measurement of computer activity G06F11/34
• CPC Class Codes: H04N 21/25833 involving client hardware characteristics, e.g. manufacturer, processing or storage capabilities allocation of resources considering hardware capabilities in multiprogramming arrangements G06F9/5044; allocation of resources considering software capabilities in multiprogramming arrangements G06F9/5055
• CPC Class Codes: H04N 21/262 Content or additional data distribution scheduling, e.g. sending additional data at off-peak times, updating software modules, calculating the carousel transmission frequency, delaying a video stream transmission, generating play-lists scheduling strategies for dispatcher in multiprogramming arrangements G06F9/4881; arrangements for scheduling broadcast services or broadcast-related services H04H60/06; flow control in packet networks H04L47/10; establishing a time schedule or organising the servicing of application requests H04L67/62
• CPC Class Codes: H04N 21/854 Content authoring

Abstract

The invention provides interactive streaming mixed media production. A mixed media content creation tool includes a resource-computation element that computes the likely resource load to be imposed on a server, communication link, and client device, when sent by the server to the client and presented to a user at the client. The resource-computation element is responsive to amounts of communication bandwidth available using the communication link, presentation buffer space and computing resources available at the client, and other measures of resource load. The resource-computation element attempts to optimize resource usage by the mixed media content. The system also includes a presentation element that responds to the resource-computation element, and to user requests, and interactively presents measures of projected resource usage by the mixed media content, comparisons of that projected resource usage with available resources, and comparisons of that projected resource usage before and after optimization.

Claims

1. A method for calculating a total necessary communication bandwidth and transmission time corresponding to a stream of mixed media including multiple media streams, including steps of

receiving inputs at a fitter;

determining a feasibility of sending said mixed media in light of said inputs;

dividing data comprising said mixed media stream into data packets;

determining a minimal send time for each of said data packets, wherein said minimal send time is the earliest time each of said packets may be sent with respect to a maximum permitted send ahead time imposed by the network; and

assigning a send time to each said data packet, the assigned time being no earlier than the determined minimal send time;

the method including steps of converting raw data present in the stream of mixed media from a first format to a second format.

2. A method as in claim 1, wherein said inputs include a maximum packet size and an available channel bandwidth as a function of time, a time offset of for each said media stream, a maximum decoding buffer size, a maximum permitted send ahead time, and an overhead of transport protocol per packet.

3. A method as in claim 1, including a step of notifying a content creator of the feasibility of sending said mixed media.

4. A method as in claim 1, wherein said step of dividing the data is performed by a fitter.

5. A method as in claim 1, wherein said step of determining the minimal send time includes simulating a decoding buffer.

6. A method as in claim 5, wherein said minimal send time is set to a maximum of (1) a date stamp of a current said packet minus the maximum permitted send ahead time or (2) the date stamp of the packet most recently removed from said decoding buffer.

7. A method as in claim 1, wherein said step of assigning said send time is performed starting with said data packets associated with an end of a performance of said mixed media and moving toward said data packets associated with a beginning of the performance of said mixed media.

8. A method as in claim 1, wherein said send time is set to a nominal send time minus a size of said packet divided by an available bandwidth at said nominal send time.

18. A system that calculates a total necessary communication bandwidth and transmission time corresponding to a stream of mixed media including multiple media streams, comprising:

input and output elements disposed to receive inputs and to send mixed media;

a processor and memory that stores instructions that, when executed by said processor, cause the system to effect the steps of any one of claims 1-8 and 9-17.

19. A computer program product comprising program code means that, when executed on a computer system, causes the computer system to effect the steps of any one of claims 1-8 and 9-17.

20. A memory that stores instructions that, when executed by a processor of a system capable of calculating communication bandwidth and transmission time corresponding to a stream of mixed media with multiple media streams, cause the system to effect the steps of any one of claims 1-8 and 9-17.

9. A method comprising steps of

importing a media stream;

encoding said media stream;

computing a bit rate associated with one or more elements included in said media stream;

generating a graphical image responsive to said step of computing, wherein said graphical image includes a time display and a set of information on bit rate usage;

updating said time display and said set of information; and

exporting said media stream;

the method including steps of converting raw data present in the stream of mixed media from a first format to a second format.

10. A method as in claim 9, wherein said step of encoding includes steps of converting a format associated with said media stream to an MPEG 4 format.

11. A method as in claim 9, wherein said step of computing includes computing a bit rate that is responsive to a buffer space at an end user, a communication link or a server.

12. A method as in claim 9, wherein said time display includes a timeline, comprising a life cycle associated with said one or more elements.

13. A method as in claim 9, wherein said step of updating is responsive to one or more changes made in said media stream.

14. A method including steps of

composing a mixed media including one or more objects;

calculating a total necessary communication bandwidth and transmission time corresponding to said mixed media;

modifying said mixed media in response to said step of calculating, wherein said step of modifying changes an order of transmission or a specific transmission time for said objects; and

displaying a graphical representation of optimization before and after the step of calculating;

the method including steps of converting raw data present in the stream of mixed media from a first format to a second format.

15. A method as in claim 14, wherein said step of modifying includes adding, deleting, or changing attributes associated with said object.

16. A method as in claim 14, wherein said step of displaying includes displaying a set of different media taking part in a global resource usage.

17. A method as in claim 14, wherein step of displaying includes providing a visual feedback of a download and decoding time required to transfer and decode said mixed media from a server to a player.