System and method for media production

• US 8,006,186 B2
• Filed: 12/22/2000
• Issued: 08/23/2011
• Est. Priority Date: 12/22/2000
• Status: Expired due to Fees

• Current Assignee: Muvee Technologies Pte Ltd.
• Original Assignee: Muvee Technologies Pte Ltd.
• Inventors: Kellock, Peter Rowan, Altman, Edward James
• Primary Examiner(s): Huynh; Ba
• Application Number:
  US10/451,041
• Publication Number:
  US 20040027369A1
• Time in Patent Office: 3,896 Days
• Field of Search: 715/201, 715/206, 715/835, 715/840, 715/854, 715716-732
• US Class Current: 715/728
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

An editing system is proposed for automatically, or semi-automatically, editing input data to generate output production. The input material is annotated by, or analyzed to derive, a set of media descriptors which describe the input material and which are derived from the input material. The style of editing is controlled using style data which is optionally derived from a user. The input material may include any or more of motion video, still images, music, speech, sound effects, animated graphics and text. The style data and the descriptors are used to generate a set of operations which, when carried out on the input data, generate an edited output production.

Claims

1. A computer-implemented method of constructing output data constituting a music-based audiovisual media production for display on an audiovisual display device, from input data including both visual media and musical media, the method including the steps of:

(a) performing signal analysis upon the musical media to generate a series of music descriptor values, said signal analysis being performed upon the musical media at a micro timescale including beats and sub-beats, each of the music descriptor values indicating the presence of a perceptually-significant characteristic of the musical media at a corresponding time in the music, said music descriptor values including edit hints, each edit hint corresponding to a certain time in the input music data, said edit hints including edit hints corresponding to respective times between said beats;

(b) receiving from a user an input indicative of the values of a set of editing parameters;

(c) defining segments of the visual media according to said series of music descriptor values and said set of editing parameters; and

(d) concatenating a plurality of said segments of the visual media.

2. A method according to claim 1, further including a pre-selection step wherein elements of said input data are manually identified and selected by a user for one or more of the following actions:

compulsory inclusion in the output data;

optional inclusion in the output data;

compulsory exclusion from the output data.

3. A method according to claim 1, further including a step wherein said output data is subjected to manual editing comprising one or more of the following treatments:

replacing a selected portion of the visual media within a segment of said output data with an alternative portion; and

overlaying text or graphics upon said output data.

4. A method according to claim 1 including a step of supplementing the at least one music descriptor value with additional pre-generated descriptors received from an external source, said additional descriptors being used in a step of generating a set of editing decisions for constructing the output data from the input data.

5. A method according to claim 4 wherein said additional descriptors include descriptors generated by instrumentation at a time of recording the input data.

6. A method according to claim 4 wherein said additional descriptors include descriptors generated manually.

7. A method according to claim 4 wherein said additional descriptors include music descriptors generated during music production.

8. A method according to claim 1 in which the method includes applying transformation operations that include modification of the color of one or more parts of an image defined by said input visual media.

9. A method according to claim 1 wherein the method includes applying transformation operations that include modification of the playback speed of one or more parts of said input data.

10. A method according to claim 1 in which in said step of concatenating the plurality of segments, the plurality of segments are concatenated with neighboring segments partially overlapping, and in which the method includes applying combination operations that include video transitions.

11. A method according to claim 1 in which said editing parameters include one or more target values for respective descriptors associated with the visual media, and the method includes selecting, for inclusion in the output data, one of more of the plurality of segments of said input data according to a calculation the proximity of a) said target value or values and b) the descriptors for each said segment.

12. A method according to claim 11 in which the descriptors include visual descriptor values, each value representing a measure of a visual properly of a respective segment of the input data, wherein the method includes:

setting a respective visual descriptor target value for at least one visual property of each of a plurality of temporal segments of the media production; and

selecting, for each temporal segment of the media production, of the portions of the input data for inclusion in the segment based on a calculation of the closeness of the value to the target value for the segment.

13. A method according to claim 1 in which the order of the selected segments of the visual media is equal to, or at least correlated with, the order of these segments within the input data whereby the output data comprises a summary of the input data in which the original sequence is preserved.

14. A method according to claim 1 in which said input indicates said set of editing parameters by specifying one of a plurality of pre-defined sets of editing parameter values.

15. A method according to claim 14 in which said set of editing parameters is composed of two or more sub-sets of editing parameters each sub-set being associated with a respective plurality of the segments, whereby variation is introduced into an editing style of said output data.

16. A method according to claim 15 in which said sub-sets of the editing parameters include target music descriptor values, and are associated with sections of said visual media by matching said generated music descriptor values to said target descriptor values, whereby variations in an editing style of said output data are synchronized with variations in said music data.

17. A method according to claim 1 in which said editing parameters include preferred segment duration information, the preferred segment duration information is varied periodically with temporal location in the output data, whereby a pattern of cyclic variation is introduced into an editing style of said output data.

18. A method according to claim 1 in which at least some random variation is introduced into the setting of preferred segment durations whereby variation is introduced into said editing decisions.

19. A method according to claim 1, further including receiving from a user a manual input identifying one or more portions of said input data for inclusion in the output data and specifying, for each of said portions, one or more aspects of transitions and effects to be applied to said input data in order to generate said output data.

20. A method according to claim 3 in which segments of the output data are manually replaced with alternative segments generated from said input data, further comprising using descriptors associated with the visual media to suggest to a user of the input data resembling or contrasting with portions in the segments to be replaced, whereby the user is assisted in the process of manually replacing segments.

21. A method according to claim 3 in which, following said generation of said descriptors, said descriptors are stored, said manual replacement step employing said stored descriptors.

22. A method according to claim 1 in which, subsequent to generating output data, said output data may be progressively refined by recursively applying the following steps:

viewing said output data,

indicating parts of said output data for which further modification is prevented, and

generating a new version of said output data in which parts which have not been thus identified are re-generated in a modified form.

23. A method according to claim 22 in which, following said generation of said descriptors, said descriptors are stored, said step of generating a new version of the output data employing said stored descriptors.

24. A method according to claim 12 in which the visual descriptor values include a human-probability descriptor value for each of a plurality of the portions of the input data, the human-probability descriptor value representing a probability that a human being is present in the respective portion of said input data, wherein portions of the input data for which the human-probability descriptor value is high are more frequently incorporated into the output data than portions for which the human-probability descriptor value is low.

25. A method according to claim 24 in which said human-probability descriptor value is a face probability descriptor value representing a probability that a human face is present in each portion of said input data.

26. A method according to claim 12 in which said visual descriptor values include at least one camera motion descriptor value for each of a plurality of moving image portions of the input data which represent moving image data, the camera-motion descriptor value representing for each respective element a degree to which the camera which collected that portion was moving when that portion was collected, wherein portions of the input data for which the camera motion descriptor value is low are more frequently incorporated into the output data than portions for which the camera motion descriptor value is high.

27. A method according to claim 1 in which said output data includes at least one overlay, said overlay comprising at least one of text and graphics.

28. A method according to claim 27 in which an overlay is animated.

29. A method according to claim 28 in which the animation of said overlay is determined by the music descriptor value.

30. A method according claim 1 which is performed by a distributed system and each of at least two steps of the method is performed at a different respective location.

31. A method according to claim 1 which is performed by a distributed system and in which step (a) is performed at one location and the remaining steps are performed at a different location.

32. A method according to claim 1 which is performed by a distributed system and:

steps (a) to (d) are performed at a first location operated by a first user;

resulting information representing the media production is transmitted to a second location operated by a second user, the second user having access to the input data or a copy thereof; and

step (e) is performed at the second location operated by the second user;

whereby the second user may view a media production created by the first user without the need to transmit the input data from said first user to said second user.

33. A method according to claim 1 in which at least some of the music descriptor values have a value indicative of the strength of the perceptually-characteristic feature of the music at the corresponding time.

34. A method according to claim 1 in which the music descriptor values are related to the timing of musical beats or other prominent musical events, the music descriptor values being used to modify the duration of segments to synchronize transitions or cuts between said concatenated segments with said musical beats or other prominent musical events.

35. A method according to claim 34 in which the modification of the segment duration is determined by applying a time-dependent threshold to the at least one music descriptor value.

36. A method according to claim 1 in which the music descriptor values are related to the timing of musical beats or other prominent musical events in said musical media and the method includes including the generation of flashes.

37. A method according to claim 1 in which

further comprising generating motion descriptors from the visual media, and

said musical event descriptor values and said motion descriptors are used to control the timing of segments of visual media in said audiovisual production such that prominent visual events are synchronized with prominent features of the music.

38. A method according to claim 1 in which the input data includes a soundtrack associated with motion video, and the method mixes parts of said soundtrack with the musical media in accordance with at least one of:

selecting said parts of soundtrack according to their audio characteristics,

determining when to mix in said parts of soundtrack according to the value of music descriptors, and

lowering the volume of said music media when said parts of soundtrack are mixed in.

39. A method according to claim 1 and further including the step of establishing a user interface having two or more of the following controls:

a control allowing a user to select visual media,

a control allowing a user to select musical media,

a control allowing a user to add text data,

a control allowing a user to select or define said editing parameters, and

a control which causes the output data to be created.

40. A method according to claim 1 including the step of displaying a graphical user interface including visual thumbnails.

41. A recording medium carrying program instructions which are readable by a computer and which cause the computer to perform a method according to claim 1.

42. A computer system comprising a processor and a data storage device,

the data storage device storing program instructions operable by the processor to construct output data constituting a music-based audiovisual media production for display on an audiovisual display device from input data including both visual media and musical media, by performing the steps of:

(a) performing signal analysis upon the musical media to generate a series of music descriptor values, said signal analysis being performed upon the musical media at a micro timescale including beats and sub-beats, each of the music descriptor values indicating the presence of a perceptually-significant characteristic of the musical media at a corresponding time in the music, said music descriptor values including edit hints, each edit hint corresponding to a certain time in the input music data, said edit hints including edit hints corresponding to respective times between said beats;

(b) receiving from a user an input indicative of the values of a set of editing parameters;

(c) defining segments of the visual media according to said series of music descriptor values and said set of editing parameters; and

(d) concatenating a plurality of said segments of the visual media.

43. A computer-implemented method for editing input data including input visual data and input music data to form a music-based audiovisual media production for display on an audiovisual display device and including output visual data, the method including the steps of:

analyzing the input music data to derive one or more descriptors including music descriptors, said music descriptors characterizing each of a plurality of portions of the input music data at a micro timescale including beats and sub-beats;

defining edit preferences for controlling the editing of the input data;

using said descriptors and said edit preferences to generate a set of editing decisions, the set of editing decisions specifying a set of editing operations to be performed on said input data; and

generating said music-based audiovisual media production by performing said set of operations upon said input data;

wherein said music descriptors include edit hints, each hint corresponding to a certain time in the input music data and indicating a perceptually-significant characteristic of a portion of the input music data, said edit hints including edit hints corresponding to respective times between said beats.