Flexible decomposition and recomposition of multimedia conferencing streams using real-time control information

• US 8,310,520 B2
• Filed: 08/19/2009
• Issued: 11/13/2012
• Est. Priority Date: 08/19/2009
• Status: Active Grant

• Current Assignee: Avaya Incorporated
• Original Assignee: Avaya Incorporated
• Inventors: Gopal, Rajat, Nicol, John
• Primary Examiner(s): NGUYEN, TUAN DUC
• Application Number:
  US12/543,998
• Publication Number:
  US 20110043600A1
• Time in Patent Office: 1,182 Days
• Field of Search: 348/14.01, 348/14.08, 348/14.09, 370/259, 370/260, 709/204, 375/240.08, 375/240.26
• US Class Current: 348/14.08
• CPC Class Codes: H04L 12/1822 Conducting the conference, e.g. admission, detection, selection or grouping of participants, correlating users to one or more conference sessions, prioritising transmission
• CPC Class Codes: H04L 12/1881 with schedule organisation, e.g. priority, sequence management
• CPC Class Codes: H04L 65/1083 In-session procedures
• CPC Class Codes: H04L 65/403 Arrangements for multi-party communication, e.g. for conferences data switching systems for conference H04L12/18; arrangements for connecting several subscribers to a common circuit, i.e. affording conference facilities H04M3/56; television conferencing systems H04N7/15
• CPC Class Codes: H04N 7/152 Multipoint control units therefor

Abstract

A multimedia conferencing system is presented that can modify the composite multimedia stream into separate multimedia streams. Two or more video conference participants send multimedia streams, capturing and encoding the conference at the participant'"'"'s location, to a multimedia server. The multimedia server includes a multipoint control unit that creates a composite multimedia display, which includes video regions corresponding to at least two of the multimedia streams from the participants. A composite multimedia stream describes the composite multimedia display. The ancillary server receives the composite multimedia stream from the multipoint control unit together with static and dynamic control information, which describes the format and layout of the composite multimedia display and the composite multimedia stream. Using the dynamic and static control information, the ancillary server decomposes the composite multimedia stream into two or more constituent streams. The two or more constituent streams may then be separately processed.

Claims

1. A method, comprising:

a processor receiving control information and a composite multimedia stream, the composite multimedia stream providing a composite multimedia display and the control information describing a characteristic of the composite multimedia stream;

the processor determining, from the control information, first and second regions of the composite multimedia display;

the processor determining, for a determined time and with the control information, a first constituent stream of the composite multimedia stream associated with the first region of the composite multimedia display; and

the processor extracting, from the composite multimedia stream, the first constituent stream of the composite multimedia stream.

2. The method as defined in claim 1, wherein the control information includes static control information and dynamic control information.

3. The method as defined in claim 1, further comprising:

for the predetermined time, the processor determining, with dynamic control information, a second constituent stream of the multimedia steam associated with a second region of the composite multimedia display; and

the processor extracting the second constituent stream of the multimedia stream from the composite multimedia stream.

4. The method as defined in claim 3, further comprising:

the processor sending the first constituent stream of the multimedia stream to a first user application; and

the processor sending the second constituent stream of the multimedia stream to a second user application.

5. The method as defined in claim 3, wherein the composite multimedia stream comprises at least three constituent streams, and further comprising:

the processor filtering out the first constituent stream of the composite multimedia stream; and

the processor sending at least one of the second constituent stream of the composite multimedia stream or a third constituent stream of the composite multimedia stream to a user application.

6. The method as defined in claim 5, wherein the user application is a multimedia content store.

7. The method as defined in claim 5, further comprising the user application recomposing the second constituent stream of the composite multimedia stream and the third constituent stream of the composite multimedia stream into a new composite multimedia stream, wherein the new multimedia stream provides a new composite multimedia display.

8. The method as defined in claim 1, wherein the control information describes a layout for the composite multimedia display.

9. The method as defined in claim 1, wherein the control information temporally describes which constituent stream of the composite multimedia stream is displayed in which region of the composite multimedia display.

10. An ancillary server comprising:

a processor operable to read and operable to execute two or more computer executable modules;

a memory in communication with the processor, the memory operable to store and operable to send the two or more computer executable modules to the processor; and

the two or more computer executable modules comprising;

a control information database operable to store control information that describes a format for a first composite multimedia stream;

a decomposer module in communication with the control information database, the decomposer module operable to decompose the first composite multimedia stream into two or more constituent streams of the first composite multimedia stream based on the control information; and

a recomposer module in communication with the decomposer module, the recomposer module operable to recompose the two or more constituent streams of the first composite multimedia stream into a second composite multimedia stream.

11. The ancillary server as defined in claim 10, wherein the recomposer module further comprises a filter module operable to filter out one of the constituent streams of the first composite multimedia stream to create the second composite multimedia stream.

12. The ancillary server as defined in claim 10, further comprising a preferences database in communication with the recomposer module, the preferences database operable to store a preference for the second composite multimedia stream, wherein the preference describes a desired format for the second composite multimedia stream.

13. The ancillary server as defined in claim 10, wherein the control information describes a format for a composite multimedia display provided by the first composite multimedia stream, wherein the format includes a layout for the composite multimedia display, the composite multimedia display layout having two or more regions.

14. The ancillary server as defined in claim 13, wherein the decomposer module is operable to receive dynamic control information, wherein the dynamic control information describes to which region of the composite multimedia display layout a constituent stream of the composite multimedia stream is assigned.

15. The ancillary server as defined in claim 14, wherein the dynamic control information changes in response to an assignment change to the composite multimedia display layout.

16. The ancillary server as defined in claim 10, further comprising a second ancillary server.

17. A non-transient computer program product including computer executable instructions stored onto a computer readable medium which, when executed by a processor of a computer, cause the computer to perform method for recomposing a composite multimedia stream, the instructions comprising:

instructions to receive static control information, wherein the static control information describes a layout for a composite multimedia display sent from a multipoint control unit, wherein the layout includes two or more regions defined on a multimedia display;

instructions to receive the composite multimedia stream, wherein the composite multimedia stream includes two or more constituent streams, wherein each constituent stream is a multimedia stream assigned to one of the regions of the display;

instructions to receive dynamic control information, wherein the dynamic control information includes information about which of the two or more constituent streams of the composite multimedia stream is an active speaker;

instructions to determine which region of the two or more regions defined on a multimedia display is displaying the constituent stream of an active speaker; and

instructions to extract the constituent stream of the composite multimedia stream assigned to the determined region.

18. The non-transient computer program product as defined in claim 17, wherein the region to which the multimedia stream is assigned changes periodically.

19. The non-transient computer program product as defined in claim 17, further comprising:

instructions to receive new dynamic control information, wherein the new dynamic control information describes a new region for the multimedia stream; and

instructions to extract the multimedia stream from the new region.

20. The non-transient computer program product as defined in claim 17, further comprising:

instructions to receive second static control information, wherein the second static control information describes a second layout for a second composite multimedia display sent from a multipoint control unit, wherein the second layout includes two or more regions defined on a second multimedia display;

instructions to receive a second composite multimedia stream, wherein the second composite multimedia stream includes two or more constituent streams, wherein each constituent stream is a multimedia stream assigned to one of the regions of the second multimedia display;

instructions to receive second dynamic control information, wherein the second dynamic control information includes second information about which of the two or more constituent streams of the second composite multimedia stream are using a microphone;

instructions to determine which region of the two or more regions defined on a second multimedia display is displaying the constituent stream of an active speaker;

instructions to extract the second multimedia stream assigned to the determined region; and

instructions to recompose the second multimedia stream with the multimedia stream into a third composite multimedia display.