Audio/visual editing tool

• US 7,945,142 B2
• Filed: 06/15/2006
• Issued: 05/17/2011
• Est. Priority Date: 06/15/2006
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Mercer, Ian C., Finkelstein, Charles A.
• Primary Examiner(s): Nguyen; Huy T
• Application Number:
  US11/424,509
• Publication Number:
  US 20070292106A1
• Time in Patent Office: 1,797 Days
• Field of Search: 386/46, 386/52, 386/54, 386/282, 386/287
• US Class Current: 386/285
• CPC Class Codes: G11B 27/28 by using information signals recorded by the same method as the main recording G11B27/22 takes precedence
• CPC Class Codes: H04N 5/91 Television signal processing therefor
• CPC Class Codes: H04N 9/8205 involving the multiplexing of an additional signal and the colour video signal

Abstract

A method for analyzing data sequences, such as video and audio segments, and extracting metadata from the data sequences for providing information related to events of the data sequences. The metadata extracted is used for making editing decisions, such as for an audio-visual production comprised of a collection of video and audio segments. The metadata identifies events of importance in the video and audio segments, which may then be applied to the editing process. A user interface including the extracted metadata in a timeline format facilitates editing the audio-visual production.

Claims

1. A method of editing media objects based on events, said method comprising:

analyzing a visual data sequence according to at least one characteristic of said visual data sequence;

extracting metadata associated with said visual data sequence according to said at least one characteristic of said visual data sequence;

analyzing an audio data sequence substantially corresponding to said visual data sequence according to at least one characteristic of said audio data sequence;

extracting metadata associated with the audio data sequence according to at least one characteristic of said audio sequence data;

determining intersections of metadata from two or more of said extracted metadata occurring at substantially the same time;

aggregating the extracted metadata from said visual data sequence, said audio data sequence, and said determined intersections of metadata; and

associating metadata and metadata intersections with a timeline associated with the data sequences.

2. A method as set forth in claim 1 further comprising analyzing a second audio data sequence according to at least one characteristic of said second audio data sequence;

extracting metadata associated with the second audio data sequence according to at least one characteristic of said second audio data sequence;

wherein said analyzing the visual data sequence according to the at least one characteristic of said visual data sequence comprises analyzing the visual data sequence according to at least one visual characteristic of said visual data sequence;

wherein said extracting metadata associated with said visual data sequence according to said at least one characteristic of said visual data sequence comprises extracting metadata associated with said visual data sequence according to said at least one visual characteristic of said visual data sequence;

wherein said analyzing the audio data sequence according to the at least one characteristic of said audio data sequence comprises analyzing the audio data sequence according to at least one audio characteristic of said audio data sequence;

wherein said extracting metadata associated with the audio data sequence according to said at least one characteristic of said audio data sequence comprises extracting metadata associated with said audio data sequence according to said at least one audio characteristic of said audio data sequence;

wherein said analyzing the second audio data sequence according to the at least one characteristic of said second audio data sequence comprises analyzing the second audio data sequence according to at least one audio characteristic of said second audio data sequence; and

wherein said extracting metadata associated with the second audio data sequence according to at least one characteristic of said second audio data sequence comprises extracting metadata associated with the second audio data sequence according to said at least one audio characteristic of said second audio data sequence.

3. A method as set forth in claim 2 wherein said at least one audio characteristic of said audio data sequence comprises at least one of music beats, audio dynamic peaks, speech characteristics, selected recorded sounds, word boundary detection, and word and phrase detection;

and

wherein said at least one visual characteristic of said video data sequence comprises at least one of video dynamic peaks, dynamic image changes, and image pattern recognition.

4. A method as set forth in claim 1 further comprising rendering a user interface depicting a timeline with the data sequences, the metadata, and the intersections of metadata, receiving user input regarding changes in one or more of said data sequences via said user interface;

and

repeating said analyzing and extracting metadata associated with each of said data sequences, said aggregating, said determining, said associating, and said rendering.

5. A method as set forth in claim 1 further comprising performing the first analyzing, the first extracting, the second analyzing, the second extracting, the determining, the aggregating, and the associating automatically without user direction and rendering a resultant audio/visual work.

6. A method as set forth in claim 5 further comprising allowing the user to modify the resultant audio/visual work.

7. A system for editing media objects based on events, said system comprising:

an analysis engine for analyzing a visual data sequence according to at least one characteristic of said visual data sequence and for analyzing an audio data sequence substantially corresponding to said visual data sequence according to at least one characteristic of said audio data sequence, said analysis engine further extracting metadata associated with said data sequences for providing information related to events of the data sequences, said analysis engine comprising,

a metadata extraction module for extracting metadata associated with said visual data sequence according to said at least one characteristic of said visual data sequence and for extracting metadata associated with the audio data sequence according to at least one characteristic of said audio sequence data, wherein an intersection of metadata from two or more of said extracted metadata occurs at substantially the same time and wherein each metadata and the intersection of metadata is associated with a timeline location corresponding to a timeline relating the data sequences,

a metadata aggregation module for aggregating the extracted metadata from the visual data sequence, the audio data sequence, and said intersection of metadata,

a user interface metadata association module for providing information related to said aggregated metadata to a user via a user interface, and

an analysis storage area for storing raw and extracted application data for analysis by the analysis engine, said raw application data comprising at least one visual data sequence stored as application data and at least one audio data sequence stored as application data, said extracted application data comprising metadata associated with said visual data sequence extracted by the metadata extraction module, metadata associated with said audio data sequence extracted by the metadata extraction module, and aggregated metadata aggregated by the metadata aggregation module.

8. A system as set forth in claim 7 wherein said extracted metadata associated with said visual data sequence comprises at least one of face patterns, color entropy characteristics, chroma values and patterns, and luma values and patterns, and wherein said extracted metadata associated with said audio data sequence comprises at least one of pauses between phrases of an audio data sequence comprising speech, music, and recorded sounds.

9. A system as set forth in claim 7 wherein said raw application data further comprises at least one second audio data sequence stored as application data, said extracted application data comprising metadata associated with said second audio data sequence extracted by the metadata extraction module.