Reusable and extensible framework for multimedia application development

• US 8,885,708 B2
• Filed: 12/11/2009
• Issued: 11/11/2014
• Est. Priority Date: 12/11/2009
• Status: Active Grant

• Current Assignee: Dialogic, Inc. (Enghouse Systems Limited)
• Original Assignee: Dialogic, Inc. (Enghouse Systems Limited)
• Inventors: Hayden, John R., Kirnum, Robert D., Fisher, Joseph A., Nixon, Brian M., Eledath, Arun V., Singh, Ranjan
• Primary Examiner(s): Senfi, Behrooz
• Assistant Examiner(s): Jebari, Mohammed
• Application Number:
  US12/635,928
• Publication Number:
  US 20110142032A1
• Time in Patent Office: 1,796 Days
• Field of Search: None
• US Class Current: 375/240.12
• CPC Class Codes: H04L 12/66 Arrangements for connecting between networks having differing types of switching systems, e.g. gateways
• CPC Class Codes: H04N 19/115 Selection of the code volume for a coding unit prior to coding
• CPC Class Codes: H04N 19/40 using video transcoding, i.e. partial or full decoding of a coded input stream followed by re-encoding of the decoded output stream

Abstract

Systems and methods of developing and/or implementing multimedia applications. The system provides an extensible framework including an application layer, a framework utility layer, and a core engine layer. The framework utility layer includes an application programming interface, a video codec sub-framework (XCF), a video packetization sub-framework (XPF), and a video/text overlay sub-framework (XOF). The core engine layer includes one or more core codec engines and one or more core rendering engines. The XCF, XPF, and XOF sub-frameworks are effectively decoupled from the multimedia applications executing on the application layer, and the core codec and rendering engines of the core engine layer, allowing the XCF, XPF, and XOF sub-frameworks and core codec/rendering engines to be independently extensible. The system also fosters enhanced reuse of existing multimedia applications across a plurality of multimedia systems.

Claims

1. A multimedia system, comprising:

a multimedia framework; and

at least one processor operative to execute the multimedia framework and at least one multimedia application out of at least one memory,

wherein the multimedia framework includes an application layer, a framework utility layer, and a core engine layer, the core engine layer including at least one core engine, the multimedia framework defining a hierarchy of classes including a base class, at least one first subclass, and at least one lower level subclass, each of the first subclass and the lower level subclass having an associated class definition, the lower level subclass inheriting the class definition of the first subclass,

wherein the framework utility layer includes an application programming interface and at least one extensible sub-framework, the extensible sub-framework including one or more first objects, the first subclass corresponding to the one or more first objects,

wherein each of at least some of the one or more first objects is configured to export a first object interface to the application programming interface, such that the extensible sub-framework is decoupled from the multimedia application executing on the application layer via the first object interface,

wherein each of at least some of the one or more first objects is further configured to export a second object interface to the core engine via the lower level subclass, such that the extensible sub-framework is decoupled from the core engine via the second object interface, the class definition of the lower level subclass specifying one or more methods invokable by the multimedia application for accessing one or more functions of the core engine,

wherein the at least one extensible sub-framework includes an extensible codec sub-framework, wherein the at least one core engine includes at least one core codec engine, and wherein the extensible codec sub-framework is operative to utilize the at least one core codec engine while performing operations dictated by the respective multimedia application,

wherein the at least one core codec engine defines one or more encoders,

wherein a first one of the one or more first objects is an encoder object, and

wherein at least a first one of the one or more methods specified by the class definition of the lower level subclass is invokable by the multimedia application for configuring one or more of the encoders defined by the at least one core codec engine.

2. The multimedia system of claim 1 wherein the at least one core codec engine further defines one or more decoders,

wherein a second one of the one or more first objects is a decoder object, and

wherein at least a second one of the one or more methods specified by the class definition of the lower level subclass is invokable by the multimedia application for configuring one or more of the decoders defined by the at least one core codec engine.

3. A multimedia system, comprising:

a multimedia framework; and

at least one processor operative to execute the multimedia framework and at least one multimedia application out of at least one memory,

wherein the multimedia framework includes an application layer, a framework utility layer, and a core engine layer, the core engine layer including at least one core engine, the multimedia framework defining a hierarchy of classes including a base class, at least one first subclass, and at least one lower level subclass, each of the first subclass and the lower level subclass having an associated class definition, the lower level subclass inheriting the class definition of the first subclass,

wherein the framework utility layer includes an application programming interface and at least one extensible sub-framework, the extensible sub-framework including at least one first object, the first subclass corresponding to the first object,

wherein the first object is configured to export a first object interface to the application programming interface, such that the extensible sub-framework is decoupled from the multimedia application executing on the application layer via the first object interface,

wherein the first object is further configured to export a second object interface to the core engine via the lower level subclass, such that the extensible sub-framework is decoupled from the core engine via the second object interface, the class definition of the lower level subclass specifying one or more methods invokable by the multimedia application for accessing one or more functions of the core engine,

wherein the at least one extensible sub-framework includes an extensible codec sub-framework, wherein the at least one core engine includes at least one core codec engine, and wherein the extensible codec sub-framework is operative to utilize the at least one core codec engine while performing operations dictated by the respective multimedia application,

wherein the at least one core codec engine defines at least one specified encoder, and

wherein at least a first one of the one or more methods specified by the class definition of the lower level subclass is invokable by the multimedia application for accessing at least one function of the specified encoder defined by the at least one core codec engine.

4. The multimedia system of claim 3 wherein the at least one core codec engine further defines at least one specified decoder, and

wherein at least a second one of the one or more methods specified by the class definition of the lower level subclass is invokable by the multimedia application for accessing at least one function of the specified decoder defined by the at least one core codec engine.

5. A multimedia system, comprising:

a multimedia framework; and

at least one processor operative to execute the multimedia framework and at least one multimedia application out of at least one memory,

wherein the multimedia framework includes an application layer, a framework utility layer, and a core engine layer, the core engine layer including at least one core engine, the multimedia framework defining a hierarchy of classes including a base class, at least one first subclass, and at least one lower level subclass, each of the first subclass and the lower level subclass having an associated class definition, the lower level subclass inheriting the class definition of the first subclass,

wherein the framework utility layer includes an application programming interface and at least one extensible sub-framework, the extensible sub-framework including at least one first object, the first subclass corresponding to the first object,

wherein the first object is configured to export a first object interface to the application programming interface, such that the extensible sub-framework is decoupled from the multimedia application executing on the application layer via the first object interface,

wherein the first object is further configured to export a second object interface to the core engine via the lower level subclass, such that the extensible sub-framework is decoupled from the core engine via the second object interface, the class definition of the lower level subclass specifying one or more methods invokable by the multimedia application for accessing one or more functions of the core engine,

wherein the at least one extensible sub-framework includes an extensible codec sub-framework, wherein the at least one core engine includes at least one core code engine, and wherein the extensible codec sub-framework is operative to utilize the at least one core codec engine while performing operations dictated by the respective multimedia application, and

wherein the application programming interface includes an encoder module containing at least one first format definition corresponding to at least a first one of the one or more methods specified by the class definition of the lower level subclass.

6. The multimedia system of claim 5 wherein the application programming interface further includes a decoder module containing at least one second format definition corresponding to at least a second one of the one or more methods specified by the class definition of the lower level subclass.

7. A multimedia system comprising:

a multimedia framework; and

at least one processor operative to execute the multimedia framework and at least one multimedia application out of at least one memory,

wherein the multimedia framework includes an application layer, a framework utility layer, and a core engine layer, the core engine layer including at least one core engine, the multimedia framework defining a hierarchy of classes including a base class, at least one first subclass, and at least one lower level subclass, each of the first subclass and the lower level subclass having an associated class definition, the lower level subclass inheriting the class definition of the first subclass,

wherein the framework utility layer includes an application programming interface and at least one extensible sub-framework, the extensible sub-framework including one or more first objects, the first subclass corresponding to the one or more first objects,

wherein each of at least some of the one or more first objects is configured to export a first object interface to the application programming interface, such that the extensible sub-framework is decoupled from the multimedia application executing on the application layer via the first object interface,

wherein each of at least some of the one or more first objects is further configured to export a second object interface to the core engine via the lower lever subclass, such that the extensible sub-framework is decoupled from the core engine via the second object interface, the class definition of the lower level subclass specifying one or more methods invokable by the multimedia application for accessing one or more functions of the core engine,

wherein the at least one extensible sub-framework includes an extensible codec sub-framework, wherein the at least one core engine includes at least one core codec engine, and wherein the extensible codec sub-framework is operative to utilize the at least one core codec engine while performing operations dictated by the respective multimedia application,

wherein the at least one extensible sub-framework includes an extensible packetization sub-framework in communication with the extensible codec sub-framework,

wherein at least a first one of the one or more first objects is a packetizer object, and

wherein at least a first one of the one or more methods specified by the class definition of the lower level subclass is invokable by the multimedia application for packetizing multimedia data.

8. The multimedia system of claim 7 wherein at least a second one of the one or more first objects is a depacketizer object, and

wherein at least a second one of the one or more methods specified by the class definition of the lower level subclass is invokable by the multimedia application for depacketizing at least one packet containing multimedia data.

9. A multimedia system, comprising:

a multimedia framework; and

at least one processor operative to execute the multimedia framework and at least one multimedia application out of at least one memory,

wherein the multimedia framework includes an application layer, a framework utility layer, and a core engine layer, the core engine layer including at least one core engine, the multimedia framework defining a hierarchy of classes including a base class, at least one first subclass, and at least one lower level subclass, each of the first subclass and the lower level subclass having an associated class definition, the lower level subclass inheriting the class definition of the first subclass,

wherein the framework utility layer includes an application programming interface and at least one extension sub-framework, the extensible sub-framework including at least one first object, the first subclass corresponding to the first object,

wherein the first object is configured to export a first object interface to the application programming interface, such that the extensible sub-framework is decoupled from the multimedia application executing on the application layer via the first object interface,

wherein the first object is further configured to export a second object interface to the core engine via the lower level subclass, such that the extensible sub-framework is decoupled from the core engine via the second object interface, the class definition of the lower level subclass specifying one or more methods invokable by the multimedia application for accessing one or more functions of the core engine,

wherein the at least one extensible sub-framework includes an extensible codec sub-framework, wherein the at least one core engine includes at least one core codec engine, and wherein the extensible codec sub-framework is operative to utilize the at least one core codec engine while performing operations dictated by the respective multimedia application,

wherein the at least one extensible sub-framework includes an extensible packetization sub-framework in communication with the extensible codec sub-framework,

wherein the at least one core codec engine defines at least one specified packetizer, and

wherein at least a first one of the one or more methods specified by the class definition of the lower level subclass is invokable by the multimedia application for accessing at least one function of the specified packetizer defined by the at least one core codec engine.

10. The multimedia system of claim 9, wherein the at least one core codec engine further defines at least one specified depacketizer, and

wherein at least a second one of the one or more methods specified by the class definition of the lower level subclass is invokable by the multimedia application for accessing at least one function of the specified depacketizer defined by the at least one coded engine.

11. A multimedia system, comprising:

a multimedia framework; and

at least one processor operative to execute the multimedia framework and at least one multimedia application out of at least one memory,

wherein the multimedia framework includes an application layer, a framework utility layer, and a core engine layer, the core engine layer including at least one core engine, the multimedia framework defining a hierarchy of classes including a base class, at least one first subclass, and at least one lower level subclass, each of the first subclass and the lower level subclass having an associated class definition, the lower level subclass inheriting the class definition of the first subclass,

wherein the framework utility layer includes application programming interface and at least one extensible sub-framework, the extensible sub-framework including at least one first object, the first subclass corresponding to the first object,

wherein the first object is configured to export a first object interface to the application programming interface, such that the extensible sub-framework is decoupled from the multimedia application executing on the application layer via the first object interface,

wherein the first object is further configured to export a second object interface to the core engine via the lower level subclass, such that the extensible sub-framework is decoupled from the core engine via the second object interface, the class definition of the lower level subclass specifying one or more methods invokable by the multimedia application for accessing one or more functions of the core engine,

wherein the at least one extensible sub-framework includes an extensible video/text overlay sub-framework, wherein the at least one core engine includes at least one core rendering engine, and wherein the extensible video/text overlay sub-framework is operative to utilize the at least one core rendering engine while performing operations dictated by the respective multimedia application,

wherein the first object is a video/text overlay object, and

wherein at least a first one of the one or more methods specified by the class definition of the lower level subclass is invokable by the multimedia application for combining a plurality of raw multimedia data streams.

12. The multimedia system of claim 11, wherein the at least one core rendering engine defines at least one specified renderer, and

wherein at least a second one of the one or more methods specified by the class definition of the lower level subclass is invokable by the multimedia application for accessing at least one function of the specified renderer defined by the at least one core rendering engine.

13. The multimedia system of claim 11, wherein the application programming interface includes an overlay module containing at least one format definition corresponding to at least a second one of the one or more methods specified by the class definition of the lower level subclass.

14. A method of implementing a multimedia application within a multimedia system, the multimedia system including at least one processor and a packet interface, the at least one processor being operative to execute the multimedia application out of at least one memory, the method comprising the steps of:

receiving a first video packet sequence from a packet interface of the multimedia system;

executing the multimedia application on an application layer included in a multimedia framework, the multimedia framework further including a framework utility layer and a core engine layer, the core engine layer including at least one core engine, the multimedia framework defining a hierarchy of classes including a base class, at least one first subclass, and at least one lower level subclass, each of the first subclass and the lower level subclass having an associated class definition, the lower level subclass inheriting the class definition of the first subclass, the framework utility layer including an application programming interface, an extensible codec sub-framework, and an extensible packetization sub-framework, at least one of the extensible codec and packetization sub-frameworks including at least one first object, the first subclass corresponding to the first object;

exporting, by the at least one of the extensible codec and packetization sub-frameworks, a first object interface to the application programming interface, such that the respective extensible codec and packetization sub-frameworks are decoupled from the multimedia application via the first object interface;

exporting, by at least the extensible codec sub-framework, a second object interface to the core engine via the lower level subclass, such that the extensible codec sub-framework is decoupled from the core engine via the second object interface, the class definition of the lower level subclass specifying one or more methods invokable by the multimedia application for accessing one or more functions of the core engine, the executing of the multimedia application including;

responsive to invoking at least one method of the extensible packetization sub-framework, depacketizing the video packet sequence to obtain a first plurality of video frames compressed according to a first specified coding format;

responsive to invoking at least one of the methods specified by the class definition of the lower level subclass, decoding the first plurality of video frames to generate a raw data stream using the core codec engine via the second object interface;

responsive to invoking at least one of the methods specified by the class definition of the lower level subclass, encoding the raw data stream to generate a second plurality of video frames compressed according to a second specified coding format using the core codec engine via the second object interface;

responsive to invoking at least one method of the extensible packetization sub-framework, packetizing the second plurality of video frames to generate a second video packet sequence; and

providing the second video packet sequence to the packet interface of the multimedia system for transmission.

15. The method of claim 14 wherein the hierarchy of classes further includes at least one second subclass, and at least one second lower level subclass, each of the second subclass and the second lower level subclass having an associated class definition, the second lower level subclass inheriting the class definition of the second subclass, wherein the framework utility layer further includes an extensible video/text overlay sub-framework, the extensible video/text overlay sub-framework including at least one second object, the second subclass corresponding to the second object, wherein the core engine layer further includes at least one core rendering engine,

wherein the method further comprises:

exporting, by the extensible video/text overlay sub-framework, a third object interface to the core engine via the second lower level subclass, such that the extensible video/text overlay sub-framework is decoupled from the core engine via the third object interface, the class definition of the second lower level subclass specifying one or more methods invokable by the multimedia application for accessing one or more functions of the core rendering engine, and wherein the executing of the multimedia application further includes;

responsive to invoking at least one of the methods specified by the class definition of the second lower level subclass, combining the raw data stream with additional video or text data using the core rendering engine.