TRANSCRIPT ALIGNMENT

• US 20100299131A1
• Filed: 05/21/2009
• Published: 11/25/2010
• Est. Priority Date: 05/21/2009
• Status: Abandoned Application

• Current Assignee: Nexidia Incorporated (Nice Ltd)
• Original Assignee: Nexidia Incorporated (Nice Ltd)
• Inventors: Watters, Daryl Kip, Gavalda, Marsal, Lanham, Drew
• Application Number:
  US12/469,916
• Publication Number:
  US 20100299131A1
• Time in Patent Office: Days
• US Class Current: 704/2
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G10L 15/10 using distance or distortion measures between unknown speech and reference templates

Abstract

Some general aspects relate to systems, software, and methods for media processing. In one aspect, a script associated with a multimedia recording is accepted, wherein the script includes dialogue, speaker indications and video event indications. A group of search terms are formed from the dialogue, with each search term being associated with a location within the script. Zero or more putative locations of each of the search terms are identified in a time interval of the multimedia recording. For at least some of the search terms, multiple putative locations are identified in the time interval of the multimedia recording. The time interval of the multimedia recording and the script are partially aligned using the determined putative locations of the search terms and one or more of the following: a result of matching audio characteristics of the multimedia recording with the speaker indications, and a result of matching video characteristics of the multimedia recording with the video event indications. Based on a result of the partial alignment, event-localization information is generated. Further processing of the generated event-localization information is enabled.

Claims

1. One or more processor readable storage devices having code embodied on said storage devices, said code for programming one or more processors to perform a method comprising:

accepting a script associated with a multimedia recording, wherein the script includes dialogue, speaker indications and video event indications;

forming a plurality of search terms from the dialogue, each search term associated with a location within the script;

determining zero or more putative locations of each of the search terms in a time interval of the multimedia recording, including for at least some of the search terms, determining multiple putative locations in the time interval of the multimedia recording;

partially aligning the time interval of the multimedia recording and the script using the determined putative locations of the search terms and one or more of the following;

a result of matching audio characteristics of the multimedia recording with the speaker indications, and a result of matching video characteristics of the multimedia recording with the video event indications;

using a result of the partial alignment to generate event-localization information; and

enabling further processing of the generated event-localization information.

2. The storage device of claim 1, wherein at least some of the dialogue included in the script is produced from the multimedia recording.

3. The storage device of claim 1 having code embodied thereon for applying a word spotting approach to determine one or more putative locations for each of the plurality of search terms.

5. One or more processor readable storage devices having code embodied on said storage devices, said code for programming one or more processors to perform a method comprising:

accepting a script associated with a multimedia recording, wherein the script includes dialogue-based script elements and non-dialogue-based script elements;

forming a plurality of search terms from the dialogue-based script elements, each search term associated with a location within the script;

determining zero or more putative locations of each of the search terms in a time interval of the multimedia recording, including for at least some of the search terms, determining multiple putative locations in the time interval of the multimedia recording;

generating a model that maps at least some of the script elements onto corresponding media elements of the multimedia recording based at least in part on the determined putative locations of the search terms; and

enabling localization of the multimedia recording using the generated model.

6. The storage device of claim 5, wherein at least some of the dialogue-based script elements are produced from the multimedia recording.

7. The storage device of claim 5 having code embodied thereon for applying a word spotting approach to determine one or more putative locations for each of the plurality of search terms.

8. The storage device of claim 5 having code embodied thereon for associating each of the putative locations with a score characterizing a quality of match of the search term and the corresponding putative location.

9. The storage device of claim 5 having code embodied thereon for enabling localization of the multimedia recording comprising:

receiving a user-specified text-based search term through a user interface;

using the generated model to identify one or more occurrences of the user-specified text-based search term within the multimedia recording; and

enabling navigation of the multimedia recording to one of the identified one or more occurrences of the user-specified text-based search term responsive to a user-specified selection received through the user interface.

10. The storage device of claim 5 having code embodied thereon for enabling localization of the multimedia recording comprising:

receiving a user-specified search criteria through a user interface;

associating at least one non-dialogue-based script element in the script with the user-specific search criteria;

using the generated model to identify one or more occurrences of the non-dialogue-based element associated with the search criteria within the multimedia recording;

enabling navigation of the multimedia recording to one of the identified one or more occurrences of the non-dialogue-based script element responsive to a user-specified selection received through the interface.

11. The storage device of claim 5, wherein the non-dialogue-based script elements include an element associated with speaker identifier.

12. The storage device of claim 5, wherein the non-dialogue-based script elements include an element associated with non-dialogue-based characteristics of segments of the multimedia recording.

13. The storage device of claim 5, wherein the non-dialogue-based script elements include an element associated with statistics on speaker turns.

14. The storage device of claim 5 having code embodied thereon for forming a specification of a time-aligned script having dialogue-based script elements arranged in an order corresponding to a time progression of the multimedia recording.

15. The storage device of claim 5 having code embodied thereon for forming a specification of a continuity script having both dialogue-based elements and non-dialogue-based elements arranged in an order corresponding to a time progression of the multimedia recording.

16. The storage device of claim 15 further having code embodied thereon for enabling the localization of the multimedia recording based on the non-dialogue-based elements in the continuity script.

17. One or more processor readable storage devices having code embodied on said storage devices, said code for programming one or more processors to perform a method comprising:

accepting a script that is at least partially aligned to a time interval of a multimedia recording, wherein the script includes a plurality of script segments each associated with a corresponding location in the time interval of the multimedia recording;

processing the script to segment the multimedia recording to form a plurality of multimedia recording segments, including associating each script segment with a corresponding multimedia recording segment; and

forming a visual representation of the script during a presentation of the multimedia recording that includes successive presentations of one or more multimedia recording segments, including, for each one of the successive presentations of one or more multimedia recording segments, forming a respective visual representation of the script segment associated with the corresponding multimedia recording segment.

4. The storage device of claim 19 having code embodied thereon for associating each of the putative locations with a score characterizing a quality of match of the search term and the corresponding putative location.

18. The storage device of claim 17 having code embodied thereon for forming a visual representation of the script comprising:

for each one of the successive presentations of one or more multimedia recording segments, determining a time onset of the visual representation of the script segment relative to a time onset of the presentation of the corresponding multimedia recording segment.

19. The storage device of claim 17 having code embodied thereon for forming a visual representation of the script comprising:

for each one of the successive presentations of one or more multimedia recording segments, determining visual characteristics of the visual representation of the script segment associated with the corresponding multimedia recording segment.

20. The storage device of claim 17 having code embodied thereon for processing the script to segment the multimedia recording comprising:

accepting an input from a source of a first identity; and

according to the input, processing the script to associate at least one script segment with a corresponding multimedia recording segment.

21. The storage device of claim 20 having code embodied thereon for processing the script to segment the multimedia recording comprising:

accepting a second input from a source of a second identity different from the first identity; and

according to the second input, processing the script to associate at least one script segment with a corresponding multimedia recording segment.

22. The storage device of claim 21, wherein the source of the first identity and the source of the second identity are members of a community.

23. The storage device of claim 17, wherein text of the visual representation of the script is in a first language, and audio of the presentation of the multimedia recording is in a second language.

24. The storage device of claim 23, wherein the first language is the same as the second language.

25. The storage device of claim 23, wherein the first language is different from the second language.

26. One or more processor readable storage devices having code embodied on said storage devices, said code for programming one or more processors to perform a method comprising:

accepting a script that is at least partially aligned to a time interval of a first multimedia recording, wherein the script includes a plurality of script segments each associated with a corresponding location in the time interval of the first multimedia recording;

accepting a second multimedia recording associated with the multimedia recording;

forming a plurality of search terms from the script elements in the script, each search term associated with a location within the script;

determining zero or more putative locations of each of the search terms in a time interval of the second multimedia recording, including for at least some of the search terms, determining multiple putative locations in the time interval of the second multimedia recording;

generating a model that maps at least some of the script elements onto corresponding media elements of the second multimedia recording based at least in part on the determined putative locations of the search terms;

associating at least one media element in the first multimedia recording with a corresponding media element in the second multimedia recording according to the generated model and the partial alignment of the script to the first multimedia recording.

27. The storage device of claim 26 having code embodied thereon for replacing said media element in the first multimedia with the associated media element in the second multimedia recording.

28. One or more processor readable storage devices having code embodied on said storage devices, said code for programming one or more processors to perform a method comprising:

accepting, from a source of a first identity, a first script that is at least partially aligned to a time interval of a multimedia recording;

accepting, from a source of a second identity different from the first identity, a second script that is at least partially aligned to the time interval of the multimedia recording;

comparing a quality of alignment of the first script to the multimedia recording with a quality of alignment of the second script to the multimedia recording; and

based on a result of the comparison, selecting one script from the first and the second script for use in a presentation of the multimedia recording.

29. The storage device of claim 28 having code embodied thereon for:

forming a visual representation of the selected script during the presentation of the multimedia recording.

30. One or more processor readable storage devices having code embodied on said storage devices, said code for programming one or more processors to perform a method comprising:

accepting a script that is at least partially aligned to a time interval of a multimedia recording, wherein the script includes a plurality of script segments each associated with a corresponding location in the time interval of the multimedia recording, and the multimedia recording includes a multimedia segment not represented in the script;

determining a sequential order of the plurality of script segments based on their corresponding locations in the time interval of the multimedia recording; and

identifying, in the sequential order of the plurality of script segments, a location associated with the multimedia not represented in the script, including, for each script element;

computing an actual time lapse from its immediate preceding script element based on their corresponding locations in the time interval of the multimedia recording; and

comparing the actual time lapse with an expected time lapse determined according to a voice characteristic.

31. The storage device of claim 30 wherein the multimedia segment not represented in the script includes a voice segment.

32. The storage device of claim 30 wherein the expected time lapse is determined based on a speed of utterance.