Methods and apparatus for use in sound replacement with automatic synchronization to images

• US 20050042591A1
• Filed: 10/28/2003
• Published: 02/24/2005
• Est. Priority Date: 11/01/2002
• Status: Active Grant

• Current Assignee: Synchro Arts Limited (LANDR Audio, Inc.)
• Original Assignee: Synchro Arts Limited (LANDR Audio, Inc.)
• Inventors: Bloom, Phillip Jeffrey, Ellwood, William John
• Granted Patent: US 8,009,966 B2
• Time in Patent Office: Days
• US Class Current: 434/307.A
• CPC Class Codes: G10L 2021/105 Synthesis of the lips movements from speech, e.g. for talking heads
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/036 Insert-editing
• CPC Class Codes: G11B 27/10 Indexing; Addressing; Timing or synchronising; Measuring tape travel
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

Digital audio and video files are created corresponding to selected scenes from a creative production and are provided with a processing system that enables dialog to be selected from a scene and replaced by a user'"'"'s dialog which is automatically synchronized with the original dialog so as to be in synchronism with lip movements displayed by the accompanying video display. The processing further includes a graphical user interface that presents the user with the video, the text of the dialog, and cues for rehearsal and recording of replacement dialog by the user. Replay of the user'"'"'s dialog is accompanied by the video and part of the original audio except that the original dialog corresponding to the user'"'"'s dialog is muted so that the user'"'"'s dialog is heard as a replacement. Singing or other sounds associated with visible action may also be replaced by the same processes.

Claims

1. An audio and video data processor, comprising:

a selector for selecting at least a portion of an audio data stream, said audio data stream being synchronized with a video data stream;

an audio feature analyser for abstracting from said selected portion of said audio data stream a stream of time-varying features and for abstracting corresponding time-varying features from an input audio data stream;

a timing analysis and waveform editing processor adapted to determine timing differences between said stream of time-varying features and said corresponding time-varying features and to utilize said timing differences to edit said input audio data stream; and

a playback control module adapted to control running of said synchronized audio data and video data streams with said edited input audio data stream replacing said selected portion.

2. A data processing system for audio and video data, comprising:

digitized audio and video data for providing an audio data stream synchronized with a video data stream;

timing data representative of a plurality of selected times in a running of said synchronized audio and video data streams;

audio feature data for providing a data stream of time-varying features abstracted from at least a selected portion of said audio data stream;

an audio feature analyser for abstracting a corresponding stream of time-varying features from an input audio data stream;

a timing analysis and waveform editing processor adapted to determine timing differences between said streams of time-varying features and to utilize said timing differences to edit said input audio data stream and produce edited input audio data; and

a playback control module adapted to control running said synchronized audio data and video data streams with said edited input audio data replacing said selected portion.

3. A data processing system according to claim 2, further comprising cueing data representative of timing of said selected portion of said audio data stream.

4. A data processing system according to claim 2, further comprising additional digitized audio data for providing a further audio data stream synchronized with said video data stream.

5. A process for providing audio and video data, comprising th steps of:

providing an audio data stream and a video data stream synchronized together;

selecting at least a portion of said audio data stream;

analysing said selected portion to abstract therefrom a stream of time-varying features; and

providing control data relating said selected portion to said stream of time-varying features.

11. A process according to claim 5, wherein more than one portion of said audio data stream is selected.

6. A method of providing a processing system for audio and video data, comprising the steps of:

storing digitized audio and video data for providing an audio data stream synchronized with a video data stream;

storing timing data representative of a plurality of selected times in a running of said synchronized audio and video data streams;

selecting at least a portion of said audio data stream;

abstracting from the selected portion of said audio data stream audio feature data for providing a data stream of time-varying features;

storing the abstracted audio feature data;

storing an audio feature analyser for abstracting a corresponding stream of time-varying features from an input audio data stream;

storing a timing analysis and waveform editing processor adapted to determine timing differences between said data stream of time-varying features and corresponding features abstracted from an input audio data stream; and

storing a playback control module for controlling running said synchronized audio data and video data streams with edited input audio data from said processor replacing said selected portion.

7. A method according to claim 6, further comprising the step of:

storing cueing data representative of timing of said selected portion of said audio data stream.

8. A method according to claim 6, further comprising the step of:

storing additional digitized audio data for providing a further audio data stream synchronized with said video data stream.

9. A method according to claim 6, wherein said timing data further comprises gain control data adapted to control audio gain at selected times during a running of said synchronized audio and video data streams.

10. A method of processing audio data, comprising the steps of:

providing an original audio data stream synchronized with a video data stream;

selecting at least a portion of said original audio data stream;

storing an input audio data stream substantially in synchronization with a portion of said video data stream corresponding to the selected portion of said original audio data stream;

abstracting from said input audio data stream a stream of time-varying features of the input audio data stream;

comparing the abstracted stream of time-varying features with a corresponding stream of time-varying features abstracted from said selected portion of said original audio data stream and determining timing differences between said streams of time-varying features;

utilizing said timing differences to edit said input audio data stream and produce edited input audio data; and

running said synchronized original audio data stream and video data stream with said edited input audio data replacing said selected portion.

12. A method according to claim 10, wherein more than one portion of said original audio data stream is selected.

13. Apparatus for processing audio data, comprising:

means for deriving from audio data feature data representative of audible time-varying acoustic features of the audio data;

means for comparing first feature data derived from first audio data synchronously associated with video data with second feature data derived from second audio data and determining timing differences between the first and second feature data;

means for editing the second audio data in dependence upon said timing difference such as to provide edited second audio data in a synchronous relation to said first audio data; and

means for synchronously outputting said video data and said edited second audio data while muting said first audio data.

14. Apparatus for processing audio data, comprising:

means for deriving from audio data feature data representative of audible time-varying acoustic features of the audio data;

means for selecting from data representing synchronously streamable video and audio data data representing a portion of a stream of the streamable data and measuring durations of and intervals containing audible time-varying acoustic features of the audio data; and

means for populating a database with data and measurements provided by said selecting and measuring means.

15. Apparatus according to claim 14, further comprising means for populating said database with text related to said data and measurements provided by said selecting and measuring means.

16. Audio and video data processing software comprising:

a feature analysis program adapted to derive from audio data feature data representative of audible time-varying acoustic features of the audio data;

a comparison and timing program adapted to compare first feature data derived from first audio data synchronously associated with video data with second feature data derived from second audio data and to determine timing differences between the first and second feature data;

an editing program adapted to edit the second audio data in dependence upon said timing differences such as to provide edited second audio data in a synchronous relation to said first audio data; and

a streaming program adapted to synchronously output said video data and said edited second audio data while muting said first audio data.

17. Audio and video data processing software comprising:

a feature analysis program adapted to derive from audio data feature data representative of audible time-varying acoustic features of the audio data;

a selection and measuring program adapted to select from data representing synchronously streamable video and audio data data representing a portion of a stream of the streamable data and to measure durations of and intervals containing audible time-varying acoustic features of the audio data; and

a database program adapted to populate a database with data and measurements provided by said selection and measuring program.

18. Audio and video data processing software according to claim 17, wherein said database program is further adapted to enable population of said database with text related to said data and measurements provided by said selection and measuring program.

19. Apparatus for processing audio and video data, comprising:

means for selecting from data representing synchronously streamable video and audio data scene data representing a portion of a stream of the streamable data and measuring durations of and intervals containing audible time-varying acoustic features of audio data within said data; and

means for populating a database with scene data and measurements provided by said selecting and measuring means.

20. Apparatus according to claim 19, further comprising means for populating said database with text related to said scene data and measurements.

21. Apparatus according to claim 19, further comprising means for populating said database with still data representative of static video data extractable from said scene data.

22. Audio and video data processing software comprising:

a selection and measuring program adapted to select from data representing synchronously streamable video and audio data scene data representing a portion of a stream of the streamable data and to measure duration of an intervals containing audible time-varying acoustic features of audio data within said scene data; and

a database program adapted to populate a database with scene data and measurements provided by said selection and measuring program.

23. Audio and video data processing software according to claim 22, wherein said database program is further adapted to populate said database with text related to said scene data and measurements.

24. Audio and video data processing software according to claim 22, wherein said database program is further adapted to populate said database with still data representative of static video data extractable from said scene data.

25. A method of processing audio data comprising the steps of:

deriving from first audio data first feature data representative of audible time-varying acoustic features of the first audio data;

deriving from second audio data second feature data representative of audible time-varying acoustic features of the second audio data;

comparing said first and second feature data and determining timing differences between the first and second feature data;

editing the second audio data in dependence upon said timing differences such as to provide edited second audio data having a synchronous relation to said first audio data; and

outputting synchronously said edited second audio data with video data having a synchronous relation to said first audio data, while muting said first audio data.

26. A method of processing audio data, comprising the steps of:

selecting from data representing synchronously streamable video and audio data scene data representing a portion of a stream of the streamable data;

measuring durations of and intervals containing audible time-varying acoustic features of the audio data; and

populating a database with scene data and measurements selected from and measured in the scene data.

27. A method according to claim 26, further comprising deriving from the audio data in the scene data feature data representative of audible time-varying acoustic features of the audio data;

and populating the database with said feature data.

28. A method according to claim 26, further comprising creating text data related to said scene data and measurements and populating said database with said text data.

29. A method according to claim 26, further comprising extracting still data representative of static video data from said scene data, and populating said database with said still data.

30. Graphical user interface software comprising:

a video and graphics display program adapted to control a display screen to display moving pictures in response to a stream of video data and to display a plurality of graphically defined control areas on said screen;

a control module adapted to detect selection of a said control area by coincidence of cursor positioning and actuation of a pointing device and to generate respective control signals in response to such selection; and

an output program adapted to respond to said control signals by outputting selected synchronized streams of video data and audio data, and to record an input audio stream provided during the said synchronized streams.