AUTOMATIC SYNCHRONIZATION OF SUBTITLES BASED ON AUDIO FINGERPRINTING

• US 20170188084A1
• Filed: 11/09/2016
• Published: 06/29/2017
• Est. Priority Date: 12/28/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: BARREIRA AVEGLIANO, Priscilla, CORREIA VILLA REAL, Lucas, LAIOLA GUIMARAES, Rodrigo, SANCHEZ GALLO, Diego
• Granted Patent: US 10,021,445 B2
• Time in Patent Office: Days
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/11 by using information not detectable on the record carrier
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 21/233 Processing of audio elementary streams monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/23424 involving splicing one content stream with another content stream, e.g. for inserting or substituting an advertisement
• CPC Class Codes: H04N 21/235 Processing of additional data, e.g. scrambling of additional data or processing content descriptors
• CPC Class Codes: H04N 21/4302 Content synchronisation processes, e.g. decoder synchronisation
• CPC Class Codes: H04N 21/4394 involving operations for analysing the audio stream, e.g. detecting features or characteristics in audio streams arrangements characterised by components specially adapted for monitoring, identification or recognition of audio in broadcast systems H04H60/58
• CPC Class Codes: H04N 21/44 Processing of video elementary streams, e.g. splicing a video clip retrieved from local storage with an incoming video stream or rendering scenes according to encoded video stream scene graphs
• CPC Class Codes: H04N 21/4856 for language selection, e.g. for the menu or subtitles
• CPC Class Codes: H04N 21/4884 for displaying subtitles
• CPC Class Codes: H04N 21/8126 involving additional data, e.g. news, sports, stocks, weather forecasts
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

Various mechanisms presented in this disclosure provide for dynamically adjusting the exhibition of subtitles. In one example, check-points are introduced into the subtitle file format that, once achieved by a multimedia player, provoke the rescheduling of the moment in which the texts contained in the subtitle file will be exhibited.

Claims

1. A computer-implemented method for synchronizing supplemental content to a multimedia stream, the method comprising:

obtaining, by a processor, the multimedia stream;

obtaining, by the processor, the supplemental content;

identifying by the processor, beginning at an anchor time start point in the multimedia stream, an audio fingerprint of a sound track of the multimedia stream; and

inserting by the processor into the supplemental content the audio fingerprint, wherein the audio fingerprint is inserted into the supplemental content in association with a time start point to match the audio fingerprint inserted into the supplemental content with the audio fingerprint of a sound track of the multimedia stream at a time period in the multimedia file beginning at the start point in the supplemental content,

wherein the multimedia stream comprises a multimedia file, and

wherein multiple audio fingerprints are identified at different time start points throughout the multimedia file, and the multiple audio fingerprints with corresponding time start points that match the respective different time start points in the multimedia file are inserted into the supplemental content, for allowing resynchronization of the supplemental content to the multimedia stream at the different time start points in the multimedia file that is played.

4. The method of claim 1, wherein the supplemental content comprises one of:

(a) captions in the language of the sound track of the multimedia stream;

(b) captions in a language different from the language of the sound track of the multimedia stream; and

(c) a combination thereof.

5. The method of claim 1, wherein the supplemental content is in the form of a file.

6. The method of claim 1, wherein the time start point associated with the audio fingerprint in the supplemental content is a time start point of a text entry in the supplemental content.

7. The method of claim 1, wherein each time start point associated with each respective audio fingerprint in the supplemental content is a time start point of a respective text entry in the supplemental content.

2. (canceled)

3. (canceled)

8. A computer readable storage medium storing a program of instructions executable by a machine to perform a method of synchronizing supplemental content to a multimedia file, the method comprising:

obtaining by a processor the multimedia file, wherein the multimedia file includes therein a soundtrack;

obtaining by the processor the supplemental content, wherein the supplemental content includes therein at least one audio fingerprint and wherein the audio fingerprint included in the supplemental content is associated with a time start point;

identifying by the processor, in the supplemental content, the audio fingerprint;

identifying by the processor, in the supplemental content, the time start point associated with the identified audio fingerprint;

determining by the processor whether, at a time period in the multimedia file beginning at a time corresponding to the identified time start point in the supplemental content, an audio fingerprint of the sound track of the multimedia file matches the identified audio fingerprint of the supplemental content;

responsive to determining that at the time period in the multimedia file beginning at the time corresponding to the identified time start point in the supplemental content the audio fingerprint of the sound track of the multimedia file does not match the identified audio fingerprint of the supplemental content;

searching, by the processor, the sound track of the multimedia file from a beginning of the multimedia file towards an end of the multimedia file to determine whether the sound track of the multimedia file contains an audio fingerprint that matches the identified audio fingerprint of the supplemental content; and

responsive to determining that the sound track of the multimedia file contains an audio fingerprint that matches the identified audio fingerprint of the supplemental content;

identifying, by the processor, a fingerprint time period in the sound track of the multimedia file where the multimedia file contains the audio fingerprint that matches the identified audio fingerprint of the supplemental content; and

determining, by the processor, an offset time difference between a start time of the fingerprint time period in the soundtrack of the multimedia file and the time start point in the supplemental content associated with the identified audio fingerprint.

9. The computer readable storage medium of claim 8, further comprising playing, by the processor, the multimedia file and the supplemental content.

10. The computer readable storage medium of claim 9, wherein the time start point associated with the audio fingerprint in the supplemental content is a time start point of a text entry in the supplemental content.

11. The computer readable storage medium of claim 10, wherein, during playing of the multimedia file and the supplemental content, the text entry in the supplement content is displayed at a timing that is corrected based upon the offset time difference.

12. The computer readable storage medium of claim 8, wherein the multimedia file comprises one of:

(a) a video clip;

(b) a television program; and

(c) a movie.

13. The computer readable storage medium of claim 8, wherein the supplemental content comprises one of:

(a) captions in the language of the sound track of the multimedia file;

(b) captions in a language different from the language of the sound track of the multimedia file;

(c) a sound track different from the sound track of the multimedia file; and

(d) a combination thereof.

14. The computer readable storage medium of claim 8, wherein the supplemental content is in the form of a file.

15. The computer readable storage medium of claim 8, wherein the supplemental content includes therein a plurality of audio fingerprints and wherein each audio fingerprint included in the supplemental content is associated with a respective time start point.

16. A computer readable storage medium storing a program of instructions executable by a machine to perform a method of synchronizing supplemental content to a multimedia broadcast, the method comprising:

obtaining by a processor the multimedia broadcast, wherein the multimedia broadcast includes therein a soundtrack;

obtaining by the processor the supplemental content, wherein the supplemental content includes therein at least one audio fingerprint and wherein the audio fingerprint included in the supplemental content is associated with a time start point;

identifying by the processor, in the supplemental content, the audio fingerprint;

identifying by the processor, in the supplemental content, the time start point associated with the identified audio fingerprint;

determining by the processor whether, at a time period in the multimedia broadcast beginning at a time corresponding to the identified time start point in the supplemental content, an audio fingerprint of the sound track of the multimedia broadcast matches the identified audio fingerprint of the supplemental content;

responsive to determining that at the time period in the multimedia broadcast beginning at the time corresponding to the identified time start point in the supplemental content the audio fingerprint of the sound track of the multimedia broadcast does not match the identified audio fingerprint of the supplemental content;

monitoring, by the processor, the sound track of the multimedia broadcast as the multimedia broadcast is obtained to determine whether the sound track of the multimedia broadcast contains an audio fingerprint that matches the identified audio fingerprint of the supplemental content; and

responsive to determining that the sound track of the multimedia broadcast contains an audio fingerprint that matches the identified audio fingerprint of the supplemental content;

identifying, by the processor, a fingerprint time period in the sound track of the multimedia broadcast where the multimedia broadcast contains the audio fingerprint that matches the identified audio fingerprint of the supplemental content; and

determining, by the processor, an offset time difference between a start time of the fingerprint time period in the soundtrack of the multimedia broadcast and the time start point in the supplemental content associated with the identified audio fingerprint.

17. The computer readable storage medium of claim 16, further comprising playing, by the processor, the supplemental content as the multimedia broadcast is obtained.

18. The computer readable storage medium of claim 17, wherein the time start point associated with the audio fingerprint in the supplemental content is a time start point of a text entry in the supplemental content.

19. The computer readable storage medium of claim 18, wherein, during playing of the supplemental content as the multimedia broadcast is obtained, the text entry in the supplement content is displayed at a timing that is corrected based upon the offset time difference.

20. The computer readable storage medium of claim 16, wherein the supplemental content includes therein a plurality of audio fingerprints and wherein each audio fingerprint included in the supplemental content is associated with a respective time start point.