Media processing methods, systems and application program interfaces

• US 20050204289A1
• Filed: 12/08/2003
• Published: 09/15/2005
• Est. Priority Date: 12/08/2003
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Dunbar, Geoffrey T., Weiss, Rebecca C., Nelson, Patrick N., Mohammed, Sohail Baig, Sherwani, Adil, Debique, Kirt A., Grigorovitch, Alexandre V., Barde, Sumedh N., Speed, Robin C.B.
• Granted Patent: US 7,900,140 B2
• Time in Patent Office: Days
• US Class Current: 715/731
• CPC Class Codes: H04L 65/612 for unicast
• CPC Class Codes: H04L 65/765 intermediate
• CPC Class Codes: H04L 65/80 Responding to QoS

Abstract

Media processing methods, systems and application program interfaces (APIs) are described. In but one embodiment, a media engine component, also referred to as a media engine, provides a simple and unified way of rendering media from an origin to a destination of choice without requiring intimate knowledge about the underlying components, their connectivity and management. Clients of the media engine need not worry about how to render the particular media, but rather can simply focus on what media to render and where to render the media. In at least one embodiment, a media session is provided and is used by the media engine and provides a mechanism by which additional components are made transparent to the application and, in at least some embodiment, the media engine. In some embodiments, the media engine and media session provide a simple API for building, configuring, and manipulating a pipeline of components (e.g. media sources, transforms, and sinks) for media flow control between an origin and one or more destinations.

Claims

1. A system comprising:

one or more computer-readable media; and

a media engine embodied on the one or more computer-readable media and configured to communicatively interact with an application to present a presentation;

the media engine being configured to use;

one or more media sources individual ones of which serving as a source of media content;

one or more transforms communicatively linked with the one or more media sources and configured to operate on data received from the one or more media sources; and

one or more media sinks configured to sink a media stream.

2. The system of claim 1, wherein the media engine exposes an application program interface that is used by an application to interact directly with the media engine, and indirectly with components used by the media engine.

3. The system of claim 1 further comprising a destination associated with the media engine and configured to provide one or more media sinks.

4. The system of claim 1, wherein the media engine is configured to provide support for both linear and non-linear media sources.

5. The system of claim 1, wherein the media engine is configured to provide transport control for the media content.

6. The system of claim 1, wherein the media engine is configured to provide for asynchronous building and management of a media pipeline given a source of media content.

7. The system of claim 1, wherein the media engine is configured to provide source resolution for the media content.

8. The system of claim 1, wherein the media engine is configured to provide access to at least some of its used components.

9. The system of claim 1, wherein the media engine is configured to enable adjustment of a media processing pipeline configuration.

10. The system of claim 1, wherein the media engine is configured to support multiple different modes of stream selection.

11. The system of claim 10, wherein one mode comprises a mode in which the media engine selects which media streams are used.

12. The system of claim 10, wherein one mode comprises a mode in which the application selects which media streams are used.

13. The system of claim 10, wherein one mode comprises a mode in which the media engine selects which media streams are used, and another mode comprises a mode in which the application selects which media streams are used.

14. The system of claim 1, wherein the media engine is configured to present a presentation on a computing device that is remote from a computing device on which the media engine resides.

15. A system comprising:

one or more computer-readable media;

a media engine embodied on the one or more computer-readable media and configured to communicatively interact with an application to present a presentation;

the media engine being configured to a provide plurality of open methods that can be called by an application to specify data sources in different manners, the media engine being configured to use;

one or more media sources individual ones of which serving as a source of media content;

one or more transforms communicatively linked with one or more media sources and configured to operate on data received from the one or more media sources; and

one or more media sinks configured to sink a media stream.

16. The system of claim 15, wherein the media engine is configured to send events associated with a media presentation to an application.

17. The system of claim 15, wherein one of the open methods specifies a URL as a data source.

18. The system of claim 15, wherein one of the open methods specifies a media source created by the application.

19. The system of claim 15, wherein one of the open methods specifies an object that has an interface from which a media source object can be obtained.

20. The system of claim 15, wherein one of the open methods specifies an object from which a byte stream can be obtained.

21. The system of claim 15, wherein one of the open methods specifies a topology to be used.

22. The system of claim 15, wherein the open methods are selected from a group of open methods that:

specify a URL as a data source,

specify a media source created by the application,

specify an object that has an interface from which a media source object can be obtained,

specify an object from which a byte stream can be obtained, and

specify a topology to be used.

23. The system of claim 15, wherein the media engine is configured to provide methods to start a presentation, stop a presentation, and pause a presentation.

24. The system of claim 23, wherein the media engine is configured to generate and send an event to an application associated with each of said start, stop and pause methods.

25. The system of claim 15, wherein the media engine further comprises a plurality of information methods that can be used by the application to obtain information that pertains to the presentation.

26. The system of claim 25, wherein one of the information methods enables the application to be exposed to multiple capabilities of the media engine.

27. The system of claim 25, wherein one of the information methods enables the application to ascertain when the system'"'"'s capabilities change.

28. The system of claim 25, wherein one of the information methods enables the application to obtain metadata associated with the presentation.

29. The system of claim 25, wherein one of the information methods enables the application to obtain metadata associated with the presentation, the metadata being obtained in the form of a property store that can be queried for the metadata.

30. The system of claim 25, wherein one of the information methods enables the application to ascertain a current destination.

31. The system of claim 25, wherein one of the information methods enables the application to ascertain statistics associated with the media engine.

32. The system of claim 25, wherein one of the information methods enables the application to ascertain a current state of the media engine.

33. The system of claim 25, wherein one of the information methods enables the application to retrieve a clock according to which the media engine is presenting.

34. The system of claim 25, wherein the information methods are selected from a group of information methods comprising methods that enable the application to:

(1) be exposed to multiple capabilities of the media engine;

(2) obtain metadata associated with the presentation;

(3) ascertain a current destination;

(4) ascertain statistics associated with the media engine;

(5) ascertain a current state of the media engine; and

(6) retrieve a clock according to which the media engine is presenting.

35. The system of claim 15, wherein the media engine is configured to generate a plurality of events associated with the presentation, the media engine being configured to send the events to the application.

36. The system of claim 35, wherein one event is associated with a new presentation that is to be presented.

37. The system of claim 35, wherein one event is associated with a completion of an open method.

38. The system of claim 35, wherein one event is associated with completion of an operation begun by calling a start method on the media engine.

39. The system of claim 35, wherein one event is associated with completion of an operation begun by calling a stop method on the media engine.

40. The system of claim 35, wherein one event is associated with completion of an operation begun by calling a pause method on the media engine.

41. The system of claim 35, wherein one event is associated with rendering of a last data sample from an active media source.

42. The system of claim 35, wherein one event is associated with completion of an operation begun by calling a close method on the media engine.

43. The system of claim 35, wherein one event is associated with a switch between presentations.

44. The system of claim 35, wherein one event is associated with a presentation destination change.

45. The system of claim 35, wherein one event is associated with a state change on the media engine.

46. The system of claim 35, wherein one event is associated with a change in a set of allowed operations on the media engine.

47. The system of claim 35, wherein one event is associated with a media rate change.

48. The system of claim 15, wherein the media engine is configured to present a presentation on a computing device that is remote from a computing device on which the media engine resides.

49. A system comprising:

one or more computer-readable media;

a media engine embodied on the one or more computer-readable media and configured to communicatively interact with an application to present a presentation;

the media engine being configured to use a media session;

the media session being configured to use;

one or more media sources individual ones of which serving as a source of media content;

one or more transforms communicatively linked with one or more media sources and configured to operate on data received from the one or more media sources; and

one or more media sinks configured to sink a media stream.

50. The system of claim 49, wherein the media engine exposes application program interfaces that are used by an application to interact directly with the media engine, and indirectly with components used by the media engine.

51. The system of claim 49 further comprising a destination associated with the media engine and configured to provide one or more media sinks.

52. The system of claim 49, wherein at least some components used by the media session are not visible to the application or media engine.

53. The system of claim 49, wherein the media session is configured to:

receive information from the media engine, said information being associated with (a) media content that is to be the subject of a presentation, and (b) a destination that is configured to provide one or more media sinks, and

cause the media content to be presented.

54. The system of claim 49, wherein the media session is configured to manage data flow from said one or more media sources to said one or more media sinks.

55. The system of claim 49, wherein the media session exposes one or more methods that enable the media engine to configure the media session for a presentation.

56. The system of claim 49, wherein the media session exposes one or more methods that enable the media engine to configure the media session for a presentation, wherein one method comprises a method through which a topology on the media session is initialized.

57. The system of claim 49, wherein the media session exposes one or more methods that enable the media engine to configure the media session for a presentation, wherein one method comprises a method through which one or more components can subscribe to receive notifications from a clock that is used to control the presentation.

58. The system of claim 49, wherein the media session provides methods for starting, stopping and pausing a presentation.

59. The system of claim 49, wherein the media session provides a preroll method that is used by the media engine to notify the media session to prepare for the start of a presentation.

60. The system of claim 49, wherein the media session further comprises a plurality of information methods that can be used by the media engine to obtain information from the media session.

61. The system of claim 49, wherein the media session further comprises a plurality of information methods that can be used by the media engine to obtain information from the media session, wherein one information method enables the media engine to ascertain a globally unique identifier that is associated with a particular implementation of a media session.

62. The system of claim 49, wherein the media session further comprises a plurality of information methods that can be used by the media engine to obtain information from the media session, wherein one information method enables the media engine to ascertain capabilities associated with the media session.

63. The system of claim 49, wherein the media session is further configured to generate a plurality of events associated with the presentation, the media session being configured to send the events to the media engine.

64. The system of claim 49, wherein the media session is further configured to generate a plurality of events associated with the presentation, the media session being configured to send the events to the media engine, wherein the media engine is configured to forward at least some of the events generated by the media session to the application.

65. The system of claim 49, wherein the media session is further configured to generate a plurality of events associated with the presentation, the media session being configured to send the events to the media engine, wherein one event comprises a session started event that is generated when a session is started.

66. The system of claim 49, wherein the media session is further configured to generate a plurality of events associated with the presentation, the media session being configured to send the events to the media engine, wherein one event comprises a session stopped event that is generated when a session is stopped.

67. The system of claim 49, wherein the media session is further configured to generate a plurality of events associated with the presentation, the media session being configured to send the events to the media engine, wherein one event comprises a session ended event that is generated with a session is ended.

68. The system of claim 49, wherein the media session is further configured to generate a plurality of events associated with the presentation, the media session being configured to send the events to the media engine, wherein one event comprises a session paused event that is generated when a session is paused.

69. The system of claim 49, wherein the media session is further configured to generate a plurality of events associated with the presentation, the media session being configured to send the events to the media engine, wherein one event comprises a rate change event that is generated when a media rate is changed.

70. The system of claim 49, wherein the media session is further configured to generate a plurality of events associated with the presentation, wherein the events are selected from a group of events comprising:

(1) a session started event that is generated when a session is started;

(2) a session stopped event that is generated when a session is stopped;

(3) a session ended event that is generated with a session is ended;

(4) a session paused event that is generated when a session is paused;

(5) a rate change event that is generated when a media rate is changed.

71. The system of claim 49 further comprising a media processor used by the media session and using at least one of said media sources and at least one transform.

72. The system of claim 49, wherein the media engine and media session are configured to present a presentation on a computing device that is remote from a computing device on which the media engine and media session reside.

73. A system comprising:

one or more computer-readable media;

a media engine embodied on the one or more computer-readable media and configured to communicatively interact with an application to present a presentation;

the media engine being configured to use a media session;

the media session being configured to use at least one media processor, one or more bit pumps communicatively linked with the media processor, and one or more media sinks communicatively linked with respective bit pumps;

the media processor being configured to use one or more media sources and one or more transforms communicatively linked with one or more media sources and configured to operate on data received from the one or more media sources.

74. The system of claim 73, wherein the one or more bit pumps are configured to pull data from the media processor.

75. The system of claim 73, wherein the one or more bit pumps are configured to pull data from the media processor and to push pulled data to one or more media sinks.

76. The system of claim 73, wherein the media engine and media session are configured to present a presentation on a computing device that is remote from a computing device on which the media engine and media session reside.

77. A system comprising:

one or more computer-readable media;

11 a media engine embodied on the one or more computer-readable media and configured to communicatively interact with an application to present a presentation;

the media engine being configured to use;

one or more media sources individual ones of which serving as a source of media content;

one or more transforms communicatively linked with one or more media sources and configured to operate on data received from the one or more media sources; and

one or more media sinks configured to sink a media stream;

the media engine further being configured to first partially resolve a topology that is to be utilized to present the presentation, and then cause a full topology to be resolved and activated.

78. The system of claim 77, wherein the media engine is configured to set up a media session which uses said one or more media sources, said one or more transforms, and said one or more media sinks, said media session being configured to fully resolve a partial topology that has been resolved by said media engine.

79. The system of claim 78, wherein the media session is configured to fully resolve said partial topology by at least ascertaining transforms that are to be placed between the media sources and the media sinks.

80. The system of claim 78, wherein the media engine is configured to receive calls from the application and forward the calls to the media session, said calls comprising calls to start, stop and pause the presentation.

81. The system of claim 78, wherein the media session is configured to create a media processor that uses one or more media sources and one or more transforms.

82. The system of claim 78, wherein the media session is configured to create a media processor that uses one or more media sources and one or more transforms, wherein the media session is configured to set a topology on the media processor.

83. The system of claim 78, wherein the media session is configured to make determinations as to which time sources are to be used to drive the presentation.

84. The system of claim 78, wherein the media session is configured to prevent drift between a rate of media sources and a rate of a time source being used in live scenarios.

85. The system of claim 78, wherein the media session is configured to receive calls from the media processor to at least start, stop and pause the presentation.

86. The system of claim 78, wherein the media session is configured to receive calls from the media processor to at least start, stop and pause the presentation, wherein the media session is configured to send events to the media engine associated with calls that the media session receives from the media engine.

87. The system of claim 78, wherein the media session is configured to reduce glitches associated with a presentation by prerolling media data samples to one or more media sinks.

88. The system of claim 78, wherein the media session is configured to validate one or more component that handle data of the presentation.

89. The system of claim 77, wherein the media engine partially resolves said topology by at least determining one or more media sources and one or more media sinks for the presentation.

90. The system of claim 77, wherein the media engine is configured to present a presentation on a computing device that is remote from a computing device on which the media engine resides. 91. A method comprising:

receiving a call from an application that specifies parameters associated with a presentation, at least one of the parameters specifying a destination configured to provide one or more media sinks;

ascertaining whether source resolution is needed in order to create a media source for the presentation;

l if source resolution is needed, resolving a source to provide a media source for the presentation, otherwise ascertaining a media source for the presentation;

determining whether topology resolution is needed;

if topology resolution is needed, first performing a partial topology resolution to partially resolve a topology and second performing a full topology resolution to fully resolve the topology, otherwise not performing topology resolution;

activating the topology; and

notifying the application that it can now control progress of the presentation. 92. The method of claim 91, wherein another of said parameters specifies an URL associated with media content that is to be the subject of the presentation. 93. The method of claim 91, wherein another of said parameters specifies a media source object that is to serve as a media source. 94. The method of claim 91, wherein another of said parameters specifies an object that implements an interface from which a media source object can be obtained. 95. The method of claim 91, wherein another of said parameters specifies an object from which sequential data can be obtained. 11 96. The method of claim 91, wherein another of said parameters specifies a topology object that specifies a topology that is to be used in presenting the presentation. 97. The method of claim 91 further comprising prior to said act of determining whether topology resolution is needed, creating a media session that uses at least the media source. 98. The method of claim 97, wherein the act of performing a partial topology resolution comprises at least creating one or more media sinks for sinking data associated with the presentation. 99. The method of claim 97, wherein the act of performing a full topology resolution is performed by the media session. 25 100. The method of claim 97 further comprising, using the media session to create a media processor, the media processor encapsulating one or more media sources and one or more transforms. 5 101. The method of claim 97, wherein components used by the media session are not visible to the application. 102. The method of claim 97, wherein components used by the media session are not visible to the application and a media engine that uses the media session and serves as a point of contact for the application to control progress of the l presentation. 103. The method of claim 91, wherein the acts of (a) receiving, (b) ascertaining whether source resolution is needed, and (c) resolving a source or otherwise ascertaining a media source are performed by a media engine that uses multiple components that process data to provide the presentation. 104. The method of claim 91, wherein the acts of (a) receiving, (b) ascertaining whether source resolution is needed, and (c) resolving a source or otherwise ascertaining a media source are performed by a media engine that uses multiple components that process data to provide the presentation, and wherein the media engine is the only point of contact for the application to call to control the progress of the presentation. 24 105. The method of claim 91, wherein the acts of (a) receiving, (b) ascertaining whether source resolution is needed, and (c) resolving a source or otherwise ascertaining a media source are performed by a media engine that uses multiple components that process data to provide the presentation, and wherein the media engine is the only point of contact for the application to call to control the progress of the presentation, and wherein the media engine is configured to call used components responsive to receiving a call from the application. 9 106. The method of claim 91, wherein the act of notifying the application comprises notifying the application that it can now control progress of a presentation that is to be presented on a computing device that is remote from a computing device on which the application resides. 1 107. A method comprising;

receiving a call from an application that specifies parameters associated with a presentation, at least one of the parameters specifying a destination configured to provide one or more media sinks, said call being received by a media engine that is configured to use a media session which itself is configured to use one or more media sources, one or more transforms and one or more media sinks;

ascertaining, with the media engine, whether source resolution is needed in order to create a media source for the presentation;

if source resolution is needed, resolving a source to provide a media source for the presentation, otherwise ascertaining a media source for the presentation;

creating a media session that uses one or more media sources;

determining, with the media engine, whether topology resolution is needed;

if topology resolution is needed, first performing, with the media engine, a partial topology resolution to partially resolve a topology and second performing, with the media session, a full topology resolution to fully resolve the topology, otherwise not performing topology resolution, said first performing at least creating one or more media sinks for the presentation, said second performing at least resolving a pipeline between the one or more media sources and the one or more sinks;

activating, with the media engine, the topology; and

notifying the application that it can now control progress of the presentation.