Method and system for the automatic generation of multi-lingual synchronized sub-titles for audiovisual data

• US 7,117,231 B2
• Filed: 11/27/2001
• Issued: 10/03/2006
• Est. Priority Date: 12/07/2000
• Status: Expired due to Fees

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Fischer, Uwe, Stenzel, Gerhard, Kriechbaum, Werner, Hoffmann, Stefan
• Primary Examiner(s): Pardo, Thuy N.
• Application Number:
  US09/994,544
• Publication Number:
  US 20020087569A1
• Time in Patent Office: 1,771 Days
• Field of Search: 348/589, 704/255, 704/278, 704/218, 707/10, 707/101, 707/102, 707/103.Y, 707/104.1, 707/203, 386/97
• US Class Current: 1/1
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/435 Processing of additional data, e.g. decrypting of additional data, reconstructing software from modules extracted from the transport stream
• CPC Class Codes: Y10S 707/99942 Manipulating data structure, e.g. compression, compaction, compilation
• CPC Class Codes: Y10S 707/99943 Generating database or data structure, e.g. via user interface
• CPC Class Codes: Y10S 707/99944 Object-oriented database structure
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing
• CPC Class Codes: Y10S 707/99954 Version management

Abstract

The present invention provides a method and system for computerized synchronization of an audio stream having a first synchronized textual representation usable for subtitling of the audio stream in the original language, with a second synchronized textual representation which can be used as an alternative subtitle including a transcription of the original language into another language. Time synchronous links can be built between the audio stream and, for instance, the textual representations of the words spoken in the audio stream. More particularly, the second representation can inherit the synchronization between the audio stream and the first representation using structure association information determined between the first and the second representation.

Claims

1. A computer-based method of synchronizing a realization of a media stream having at least one version of content and having a first representation synchronized with said realization, and at least one second representation, said method comprising:

determining structure information for said first representation and said at least one second representation;

determining structure association information between said first representation and said at least one second representation, wherein said structure association information includes semantic structure association information;

synchronizing said at least one second representation with said first synchronized representation and said realization using said semantic structure association information; and

aligning said at least one version of content with said first representation to produce a web of relations between a structural view of said at least one version of content and said first representation.

2. The method according to claim 1, said step of determining structure information further comprising:

analyzing said structure information of said first and said at least one second representation, and providing a stream of tree locators.

3. The method according to claim 2, further comprising:

aligning said determined structure information of said first representation and said at least one second representation using said semantic structure association information in a form lacking temporal information.

4. The method according to claim 3, further comprising:

aligning an audio stream specified by said media stream with an audio structure corresponding to said audio stream.

5. The method according to claim 3, further comprising:

aligning a text stream specified by said media stream with a text structure corresponding to said text stream.

6. The method according to claim 1, wherein the step of synchronizing said at least one second representation and said realization is done using only said semantic structure association information.

7. A system for synchronizing a realization of a media stream having at least one version of content and having a first representation synchronized with said realization, and at least one second representation, said system comprising:

a first structurer configured to determine structure information for said first representation;

at least a second structurer configured to determine structure information for said at least one second representation; and

a first aligner configured to align said structure information for said first representation and said at least one second representation;

wherein said first aligner aligns in part at least a semantic structure association information lacking temporal data forming a portion of said structure information for said first representation and said at least one second representation and produces a web of relations between a structural view of said at least one version of content and said first representation.

8. The system according to claim 7, further comprising:

at least one renderer configured to render said at least one second representation, after being synchronized, in a form suitable for displaying as an overlayed subtitle.

9. The system according to claim 8, wherein said realization specifies a media stream, said system further comprising:

a tree aligner configured to determine a tree structure for said media stream.

10. The system according to claim 9, further comprising:

means for detecting speech and non-speech boundaries.

11. The system according to claim 9, further comprising:

means for detecting transitions and speaker changes.

12. A machine-readable storage, having stored thereon a computer program having a plurality of code sections executable by a machine for causing the machine to perform the steps of:

determining structure information for a first representation being synchronized to a corresponding media stream and at least one second representation;

determining structure association information between said first representation and said at least one second representation, wherein said structure association information includes semantic structure association information lacking temporal information;

synchronizing said at least one second representation with said first synchronized representation and said realization using said semantic structure association information and;

aligning said determined structure information of said first representation and said at least one second representation to produce a web of relations between a structural view of at least one version of content of a realization and said first representation.

13. The machine-readable storage according to claim 12, said step of determining structure information further comprising:

analyzing said structure information of said first and said at least one second representation, and providing a stream of tree locators.

14. The machine-readable storage according to claim 12, further comprising:

aligning an audio stream specified by said media stream with an audio structure corresponding to said audio stream.

15. The machine readable storage according to claim 12, further comprising:

aligning a text stream specified by said media stream with a text structure corresponding to said text stream.

16. The machine-readable storage according to claim 12, said step of synchronizing said at least one second representation and said realization is done using only said semantic structure association information.