Method and system for the authoring and playback of independent, synchronized media through the use of a relative virtual time code

• US 7,913,157 B1
• Filed: 04/17/2007
• Issued: 03/22/2011
• Est. Priority Date: 04/18/2006
• Status: Active Grant

• Current Assignee: Overcast Media Incorporated
• Original Assignee: Overcast Media Incorporated
• Inventors: Butler, Laura Janet, Matthews, Joseph H. III, Stoakley, Richard Wales
• Primary Examiner(s): Hong; Stephen S
• Assistant Examiner(s): YANG, I CHAN
• Application Number:
  US11/736,352
• Time in Patent Office: 1,435 Days
• Field of Search: 715/201, 715/202, 715/203, 715/204, 715/723, 386/12, 386/61, 386/65, 386/66, 725/135, 725/136
• US Class Current: 715/203
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/323 Time code signal, e.g. on a cue track as SMPTE- or EBU-time code
• CPC Class Codes: H04N 21/234318 by decomposing into objects, e.g. MPEG-4 objects
• CPC Class Codes: H04N 21/43072 of multiple content streams on the same device
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

A method and system is provided for the creation and playback of multiple independently produced and distributed media intended for synchronized playback. One embodiment of the invention overcomes variances in independently produced and distributed media that make accurate synchronization impossible today. The system utilizes both authoring and playback processes. During authoring, a relative virtual time code profile is generated based on the original source media in a defined associated media set. The system employs an extensible framework of multiple synchronization recognizers that analyze the source media to generate a relative virtual time code profile for the associated media set. During playback, the system'"'"'s client can access the relative virtual time code profile to coordinate the synchronized playback of an associated media set. The system generates an absolute time code using the available associated media and the original relative virtual time code profile. The system can overcome significant variances between the available associated media and the original associated media such as missing content, added content, resolution differences, format differences, etc.

Claims

1. A method, comprising:

enabling authoring of a media set;

enabling selection of a first independent media element, associated with a first entity, as a principal media element;

enabling selection of at least a second independent media element, associated with a second entity, as a supplemental media element;

extracting principal media metadata from the principal media element;

determining properties of the principal media element;

generating a principal media profile representing the principal media metadata and properties; and

extracting and analyzing content of the principal media element, using as inputs, the principal media element, the principal media profile, and at least some results of media compositing;

generating a principal media timeline, which is an outline representation of the extracted and analyzed content of the principal media element, including relationship information between the content; and

enabling presenting of the first and second media elements together as the media set in a synchronous manner using a relative virtual time code, wherein said relative virtual time code includes an intermediary data structure containing a collection of patterns and time-based relationships used to generate an absolute time code for variations of the media elements in the synchronous media set and wherein said absolute time code includes a fixed time code for specific instances of the media elements in the synchronous media set,

wherein said authoring of the media set includes;

selecting the first media element as said principal media element;

selecting or creating the second media element as said supplemental media element;

performing media compositing to relate presentation of the principal and supplemental media elements to each other;

generating a portable media project file that contains results of the media compositing;

generating a portable media compositing file from the portable media project file; and

publishing the portable media compositing file, the portable media compositing file being usable to allow presentation of the media set in the synchronous manner.

2. The method of claim 1 wherein said authoring or said presenting the media set includes using at least one recognition algorithm to identify landmarks in the first media element and to determine a manner in which to present the second media element synchronously with the first media element based on the landmarks.

3. The method of claim 1 wherein the portable media project file includes:

a link to the principal media element and a link to at least one of said supplemental media element;

said principal media metadata and supplemental media metadata;

said principal media timeline;

mixing data representing presentation relationships between the principal and supplemental media elements; and

a first absolute time code to match segments of the supplemental media element and mixing data with locations in the principal media element using the principal media timeline as reference.

4. The method of claim 3 wherein said performing the media compositing includes generating the first absolute time code, generating the mixing data, and obtaining the supplemental media metadata, which collectively comprise the results of the media compositing.

5. The method of claim 4 wherein said generating the portable media project file that contains the results of the media compositing includes:

said extracting the principal media metadata from the principal media element;

said determining properties of the principal media element;

said generating a principal media profile representing the principal media metadata and properties;

said extracting and analyzing content of the principal media element, using as inputs, the principal media element, the principal media profile, and at least some of the results of the media compositing;

said generating the principal media timeline, which is the outline representation of the extracted and analyzed content of the principal media, including relationship information between the content; and

saving the portable media project file.

6. The method of claim 5 wherein the portable media compositing file includes:

the link to at least one of said supplemental media element;

the mixing data;

the principal media metadata and supplemental media metadata; and

said relative virtual time code associated with the media set and having information usable to translate said first absolute time code, which is associated with an original principal media element, into a second absolute time code of a playback principal media element, thereby providing a correspondence between points of the original principal media element and points of the playback principal media element.

7. The method of claim 6 wherein said generating the portable media compositing file from the portable media project file includes:

using the principal media element as a first input;

using the portable media project file as a second input;

using the first and second inputs, generating landmark definitions associated with events and relationships between the principal and supplemental media elements;

generating the relative virtual time code based on the generated landmark definitions and the principal media timeline; and

saving the portable media compositing file.

8. The method of claim 1 wherein said enabling selection of the first media element or said enabling selection of at least the second media element respectively includes providing the first media element or providing the second media element from a network location.

9. The method of claim 1 wherein said enabling selection of the first media element or said enabling selection of at least the second media element respectively includes providing the first media element or providing the second media element from a portable storage device.

10. The method of claim 1 wherein said presenting the media set in the synchronous manner includes:

selecting the supplemental media element;

selecting a playback principal media element;

obtaining said published portable media compositing file;

performing media compositing to relate the selected principal media element and the selected supplemental media element to each other using the portable media compositing file; and

playing back the media set, having the composited principal media supplemental media elements, in the synchronous manner.

11. The method of claim 1 wherein said presenting the media set in the synchronous manner includes:

selecting a playback principal media element;

presenting a list of available supplemental media elements;

selecting at least one of the supplemental media elements from the list;

obtaining said published portable media compositing file;

performing media compositing to relate the selected principal media element and the selected supplemental media element to each other using the portable media compositing file; and

playing back the media set, having the composited principal media supplemental media elements, in the synchronized manner.

12. The method of claim 10, further comprising prior to said playing back the media set:

extracting the principal media metadata from the playback principal media element;

determining properties of the playback principal media element;

generating a playback principal media profile representing the playback principal media metadata and properties;

extracting and analyzing content of the playback principal media element, using as inputs, the playback principal media profile, the playback principal media element, and original principal media metadata that is contained in the portable media compositing file and that corresponds to an original principal media element involved in an authoring process; and

generating a playback principal media timeline, which is an outline representation of the extracted and analyzed content of the playback principal media, including relationship information between the content.

13. The method of claim 12, further comprising:

using the playback principal media element as a first input;

using the playback principal media profile as a second input;

using the playback principal media timeline as a third input;

using the relative virtual time code, contained in the portable media compositing file, as a fourth input;

using landmark definitions in the relative virtual time code of the fourth input to generate references to the playback principal media timeline of the third input, the landmark definitions corresponding to synchronous events and relationships between the playback principal media element and the supplemental media element as determined in the authoring process;

comparing the playback principal media profile with the original principal media metadata that is contained in the portable media compositing file to generate a new timeline and a presentation space;

associating segments of the playback principal media element and segments of the supplemental into the new timeline in a manner that fits the presentation space; and

using the references generated using the landmark definitions to obtain an absolute time code reference, the absolute time code reference including the new timeline, mixing data, and information describing relationships between elements of the playback principal media element and the supplemental media element in the media set.

14. The method of claim 13 wherein said playing back the media set includes:

compositing the playback principal media element and the supplemental media element using instructions in the absolute time code reference;

optionally introducing an additional media element into the composition; and

providing the composition to a display device or to a storage device.

15. The method of claim 14 wherein said introducing the additional media element into the composition includes introducing an advertisement into the composition.

16. The method of claim 1, further comprising prior to said presenting the synchronous media set:

providing a plurality of inputs that include said relative virtual time code and information associated with a playback version of said first media element; and

generating said absolute time code from said inputs to identify landmarks usable to present the playback first media element and the second media element as the synchronous media set, said identifying of landmarks including applying a threshold process and a best-guess process to said inputs.

17. The method of claim 1 wherein:

the first media element or the second media element is selectable from a source that includes at least one of;

a broadcast, an online source, local storage, a portable storage medium, or a DVD; and

the first and second media elements are dynamically synchronized from at least one incoming stream.

18. A system, comprising:

means for authoring a media set;

means for providing a first independent media element, associated with a first entity, as a principal media element;

means for providing at least a second independent media element associated with a second entity, as a supplemental media element;

means for extracting principal media metadata from the principal media element and for determining properties of the principal media element, and further for generating a principal media profile representing the principal media metadata and properties;

means for extracting and analyzing content of the principal media element, using as inputs, the principal media element, the principal media profile, and at least some results of media compositing, and for generating a principal media timeline, which is an outline representation of the extracted and analyzed content of the principal media element, including relationship information between the content; and

means for presenting the first and second media elements together as the media set in a synchronous manner using a relative virtual time code, wherein said relative virtual time code includes an intermediary data structure containing a collection of patterns and time-based relationships used to generate an absolute time code for variations of the media elements in the synchronous media set and wherein said absolute time code includes a fixed time code for specific instances of the media elements in the synchronous media set,

wherein the means for authoring the media set includes;

means for selecting the first media element as said principal media element;

means for selecting or creating the second media element as said supplemental media element;

means for performing media compositing to relate presentation of the principal and supplemental media elements to each other;

means for generating a portable media project file that contains results of the media compositing;

means for generating a portable media compositing file from the portable media project file; and

means for publishing the portable media compositing file, the portable media compositing file being usable to allow presentation of the media set in the synchronous manner.

19. The system of claim 18 wherein the means for presenting the media set in the synchronized manner includes:

means for selecting the supplemental media element;

means for selecting a playback principal media element;

means for obtaining the published portable media compositing file;

means for performing media compositing to relate the selected principal media element and the selected supplemental media element to each other using the portable media compositing file; and

means for playing back the media set, having the composited principal media supplemental media elements, in the synchronous manner.

20. The system of claim 19, further comprising:

means for generating a playback principal media timeline;

means for generating said absolute time code using at least the playback principal media timeline; and

means for compositing the playback principal media element and the supplemental media element for playback using the absolute time code.

21. The system of claim 18 wherein:

the first media element or the second media element is selectable from a source that includes at least one of;

a broadcast, an online source, local storage, a portable storage medium, or a DVD; and

the first and second media elements are dynamically synchronized from at least one incoming stream.

22. An authoring apparatus, comprising:

a computing device to provide a plurality of tools, the plurality of tools including;

a first tool to select a first independent media element, associated with a first entity, as a principal media element;

a second tool to select or create a second independent media element, associated with a second entity, as a supplemental media element;

a third tool to perform media compositing to relate presentation of the principal and supplemental media elements to each other;

a fourth tool to generate a portable media project file that contains results of the media compositing;

a fifth tool to generate a portable media compositing file from the portable media project file; and

a sixth tool to publish the portable media compositing file, the portable media compositing file being usable to allow presentation of the media set in a synchronous manner using a relative virtual time code, wherein said relative virtual time code includes an intermediary data structure containing a collection of patterns and time-based relationships used to generate an absolute time code for variations of the media elements in the synchronous media set and wherein said absolute time code includes a fixed time code for specific instances of the media elements in the synchronous media set,

wherein the fourth tool includes;

a principal media analyzer to extract principal media metadata from the principal media element and to determine properties of the principal media element, and to further generate a principal media profile representing the principal media metadata and properties; and

a principal media timeline analyzer operatively coupled to the principal media analyzer to extract and analyze content of the principal media element, using as inputs, the principal media element, the principal media profile, and at least some of the results of the media compositing, the principal media timeline analyzer being adapted to generate the principal media timeline, which is an outline representation of the extracted and analyzed content of the principal media element, including relationship information between the content.

23. The apparatus of claim 22 wherein the fifth tool includes:

a virtual time code generator having the principal media element as a first input and having the portable media project file as a second input; and

a landmark identification analyzer operatively coupled to the virtual time code generator to use the first and second inputs to generate landmark definitions associated with events and relationships between the principal and supplemental media elements, wherein the virtual time code generator is adapted to generate the relative virtual time code based on the generated landmark definitions and the principal media timeline.

24. The apparatus of claim 22 wherein:

the first media element or the second media element is selectable from a source that includes at least one of;

a broadcast, an online source, local storage, or a portable storage medium, or a DVD; and

the computing device is configured to enable dynamic synchronization of the first and second media elements, which are contained at least one incoming stream.

25. A presentation apparatus, comprising:

a computing device to provide a plurality of tools, the plurality of tools including;

a first tool to select a playback principal media element, the playback principal media element being an independent media element associated with a first entity;

a second tool to select at least one supplemental media element, the supplemental media element being an independent media element associated with a second entity;

a third tool to obtain a published portable media compositing file;

a fourth tool to perform media compositing to relate the selected principal media element and the selected supplemental media element to each other using the portable media compositing file; and

a fifth tool to control playback of the media set, having the composited principal media supplemental media elements, in a synchronous manner using a relative virtual time code, wherein said relative virtual time code includes an intermediary data structure containing a collection of patterns and time-based relationships used to generate an absolute time code for variations of the media elements in the synchronous media set and wherein said absolute time code includes a fixed time code for specific instances of the media elements in the synchronous media set;

a principal media analyzer to extract the principal media metadata from the playback principal media element and to determine properties of the playback principal media element, and to further generate a playback principal media profile representing the playback principal media metadata and properties; and

a principal media timeline analyzer to extract and analyze content of the playback principal media element, using as inputs, the playback principal media profile, the playback principal media element and original principal media metadata that is contained in the portable media compositing file and that corresponds to an original principal media element involved in an authoring process, the principal media timeline analyzer being adapted to generate a playback principal media timeline, which is an outline representation of the extracted and analyzed content of the playback principal media element, including relationship information.

26. The apparatus of claim 25, further comprising:

an absolute time code generator having the playback principal media element as a first input, the playback principal media profile as a second input, the playback principal media timeline as a third input, and the relative virtual time code contained in the portable media compositing file as a fourth input; and

a landmark identification analyzer, operatively coupled to the absolute time code generator, to use landmark definitions in the relative virtual time code of the fourth input to generate references to the playback principal media timeline of the third input, the landmark definitions corresponding to synchronous events and relationships between the playback principal media element and the supplemental media element as determined in the authoring process,

wherein the absolute time code generator is adapted to compare the playback principal media profile with the original principal media metadata that is contained in the portable media compositing file to generate a new timeline and a presentation space, to associate segments of the playback principal media element and segments of the supplemental into the new timeline in a manner that fits the presentation space, and to use the references generated using the landmark definitions to obtain said absolute time code, the absolute time code including the new timeline, mixing data, and information describing relationships between elements of the playback principal media element and the supplemental media element in the media set.

27. The apparatus of claim 25 wherein:

the first media element or the second media element is selectable from a source that includes at least one of;

a broadcast, an online source, local storage, a portable storage medium, or a DVD; and

the computing device is configured to enable dynamic synchronization of the first and second media elements, which are contained at least one incoming stream.