System and method for media production

• US 20040027369A1
• Filed: 06/18/2003
• Published: 02/12/2004
• Est. Priority Date: 12/22/2000
• Status: Active Grant

• Current Assignee: Muvee Technologies Pte Ltd.
• Original Assignee: Muvee Technologies Pte Ltd.
• Inventors: Kellock, Peter Rowan, Altman, Edward
• Granted Patent: US 8,006,186 B2
• Time in Patent Office: Days
• US Class Current: 345/716
• CPC Class Codes: G11B 27/034 on discs G11B27/036, G11B27/038 take precedence
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D

Abstract

An editing system is proposed for automatically, or semi-automatically, editing input data to generate output production. The input material is annotated by, or analyzed to derive, a set of media descriptors which describe the input material and which are derived from the input material. The style of editing is controlled using style data which is optionally derived from a user. The input material may include any or more of motion video, still images, music, speech, sound effects, animated graphics and text. The style data and the descriptors are used to generate a set of operations which, when carried out on the input data, generate an edited output production.

Claims

1. A method for editing input data to form output data, said input data and output data both including visual data, the method including the steps of:

analyzing said input data to generate one or more descriptors characterizing each of a plurality of portions of the input data;

defining style information for controlling the editing of the input data;

using said descriptors and said style information to generate a set of editing decisions, the set of editing decisions specifying a set of editing operations to be performed on said input data; and

generating said output data by performing said set of operations upon said input data.

2. A method according claim 1 in which said input data comprises visual data and music data and said output data is an audiovisual production which comprises portions of both said music data and said visual data.

3. A method according to claim 2 in which

one or more of said descriptors are derived from said music data, and

said descriptors are used in conjunction with said style information to determine the selection, processing and timing of a set of visual editing operations, and

said visual editing operations are applied to said visual data,

whereby a music-based production is generated in which the selection, processing and timing of visual elements is controlled or influenced by the characteristics and timing of said music data.

4. A method according to any preceding claim, further including a pre-selection step wherein elements of said input data are manually identified and selected for one or more of the following actions:

compulsory inclusion in the output data;

optional inclusion in the output data according to a certain probability;

compulsory exclusion from the output data.

5. A method according to any preceding claim, further including a touch-up step wherein said output data is subjected to manual editing comprising one or more of the following treatments:

replacing segments of said output data with alternative segments from said input data;

altering transitions between segments in said output data;

applying special effects to said output data;

overlaying text or graphics upon said output data.

6. A method according to any preceding claim in which said step of using said descriptors and said style information to generate a set of editing decisions comprises the application of the following sub-steps to a set of times corresponding to locations in said output data:

Acquiring a value for an editing preference from said style information

Acquiring a value for an editing hint derived from said one or more descriptors

Combining said editing preference and said editing hint to derive an editing decision.

7. A method according to claim 1 including a step of supplementing said descriptors with additional pre-generated descriptors received from an external source, said additional descriptors being used in said step of generating said set of decisions.

8. A method according to claim 7 wherein said additional descriptors include descriptors generated by instrumentation at a time of recording the input data.

9. A method according to claim 7 or claim 8 wherein said additional descriptors include descriptors generated manually.

10. A method according to claim 7, claim 8 or claim 9 wherein said additional descriptors include music descriptors generated during music production.

11. A method according to any preceding claim in which said set of editing operations include operations of at least one of the following types:

segmentation, selective inclusion, sequencing, transformation or combination.

12. A method according to claim 11 in which said input data includes visual data, and said transformation operations include modification of the color of one or more parts of an image defined by said input data.

13. A method according to claim 11 or claim 12 in which said transformation operations include modification of the playback speed of one or more parts of said input material.

14. A method according to any of claims 9 to 11 in which said combination operations include video transitions.

15. A method according to any preceding claim in which the step of defining the style information is performed by selecting one of a plurality of predefined sets of style information.

16. A method according to any preceding claim in which said style information includes a preferred segment duration parameter which influences the duration of segments of the input data incorporated into the output data.

17. A method according to any preceding claim in which said style information includes one or more target values for respective descriptors, and said step of generating the set of operations comprises selecting, for inclusion in the output data, one of more of the plurality of portions of said input data according to a calculation the proximity of a) said target value or values and b) the descriptors for each said portion.

18. A method according to any preceding claim in which the order of portions of the output data is equal to, or at least correlated with, the order within the input data of corresponding portions of the input data whereby the output data comprises a summary of the input data in which the original sequence is preserved.

19. A method according to any preceding claim in which said style information is composed of two or more sub-styles, each sub-style comprising a variant of said style information, and said two or more sub-styles are associated with a plurality of sections of said output data such that they control the editing of said sections, whereby variation is introduced into said editing style of said output data.

20. A method according to claim 19 in which said input data includes music data and said sub-styles are linked to sections of said music data by matching descriptor values characterizing said sections of music data to target descriptor values in said sub-styles, whereby variations in the editing style of said output data are synchronized with variations in said music data.

21. A method according to any preceding claim which employs at least one parameter which varies as a function of location within said output data, said parameter acting to influence edit decisions, whereby variation is introduced into the editing style of said output data.

22. A method according to claim 21 in which said at least one parameter varies periodically with location in the output data, whereby a pattern of cyclic variation is introduced into the editing style of said output data.

23. A method according to any preceding claim in which said style information includes data which defines a probability distribution and in which said step of generating a set of editing decisions is influenced by one or more random values selected according to said probability distribution, whereby limited variation is introduced into said editing decisions.

24. A method according to any preceding claim, further including receiving from a user a manual input identifying one or more elements of said input data and specifying, for each of said elements, one or more aspects of transitions and effects to be applied to said input data in order to generate said output data.

25. A method according to claim 5 in which segments of the output data are manually replaced with alternative segments from said input data, further comprising using said descriptors to suggest to a user segments of the input data resembling or contrasting with said segments of the output data to be replaced, whereby the user is assisted in the process of manually replacing segments.

26. A method according to any preceding claim in which, subsequent to generating output data, said output data may be progressively refined by recursively applying the following steps:

viewing said output data,

indicating portions of said output data for which further modification is prevented, and

generating a new version of said output data in which portions which have not been thus identified are re-generated in a modified form.

27. A method according to any preceding claim in which said descriptors include a human-probability descriptor for each of a plurality of elements of the input data, the human-probability descriptor representing a probability that a human being is present in each element of said input material, and said step of generating a set of operations generates operations for which the elements of the input data for which the value of said human-probability descriptor is high are more frequently incorporated into the output data than elements for which the human-probability descriptor is low.

28. A method according to claim 27 in which said human-probability descriptor is a face probability descriptor representing a probability that a human face is present in each element of said input material.

29. A method according to any preceding claim in which said descriptors include at least one camera motion descriptor for each of a plurality of moving image elements of the input data which represent moving image data, the camera-motion descriptor representing for each respective element a degree to which the camera which collected that element was moving when that element was collected, and said step of generating a set of operations generates operations for which the elements of the input data for which the value of said camera motion descriptor is low are more frequently incorporated into the output data than elements for which the camera motion descriptor is high.

30. A method according to any preceding claim in which said output data includes at least one overlay, said overlay comprising at least one of text and graphics.

31. A method according to claim 30 in which an overlay is animated.

32. A method according to claim 31 in which said input data includes music and at least one parameter of the animation of said overlay is determined by a music descriptor representing a characteristic of said music.

33. A method according any preceding claim in which at least two of said steps of defining said style information, generating said set of operations, and generating said output data, are initiated by different, spatially separated users.

34. A method according to any preceding claim in which said steps of defining said style information and generating said set of decisions are performed by a first user, and said sets of decisions are transmitted to a second user operating an apparatus with access to the input data, or a copy thereof, the second user initiating said step of generating said output data using said set, whereby the second user may inspect output data created by said first user without the need to transmit media data from said first user to said second user.

35. A method according to claim 3 in which said descriptors derived from music data include a musical energy descriptors derived from loudness or musical tempo and said musical energy descriptor is used to determine or influence the rapidity of said visual editing operations.

36. A method according to claim 3 in which said descriptors derived from music data include one or more edit hint descriptors indicating the timing of musical beats and said visual editing operations include the generation of transitions between visual segments, said edit hint descriptors being used in conjunction with said style information to synchronize said transitions with said musical beats.

37. A method according to claim 3 in which said descriptors derived from music data include one or more edit hint descriptors related to the timing of musical beats or other prominent events in said music data and said visual editing operations include the generation of flashes, said edit hint descriptors being used in conjunction with said style information to synchronize said flashes with said musical beats or other prominent events.

38. A method according to claim 3 in which

said descriptors include musical event descriptors derived from music data, and

said descriptors include motion descriptors derived from visual input data, and

said descriptors are used to control the timing of segments of visual data in said audiovisual production such that prominent visual events are synchronized with prominent features of the music.

39. A method according to claim 36 or claim 37 in which one or more of said visual editing operations are determined by applying a time-dependent threshold governed by said style information to said edit hint descriptors.

40. A method according to any preceding claim in which the input data includes a soundtrack associated with motion video, and music, and said set of operations mixes portions of said soundtrack with said music in accordance with at least one of:

selecting said portions of soundtrack according to their audio characteristics,

determining when to mix in said portions of soundtrack according to the value of music descriptors, and

lowering the volume of said music when said portions of soundtrack are mixed in.

41. A computer program product, such as a recording medium, carrying program instructions which are readable by a computer apparatus and which cause the computer apparatus to perform a method according to any preceding claim.

42. An editing system comprising stored program means which cause the editing system to perform a method according to any preceding claim and further comprising a a user interface having two or more of the following controls:

a control allowing a user to select visual input data,

a control allowing a user to select input music data,

a control allowing a user to add text data,

a control allowing a user to select or define a style, and

a control which causes the output data to be created.

43. An editing system for editing input data to form output data, said input data and output data both including visual data, the system including:

analysis means for analyzing said input data to generate one or more descriptors characterizing each of a plurality of portions of the input data;

style definition means for defining style information for controlling the editing of the input data;

construction means for using said descriptors and said style information to generate a set of one or more editing decisions specifying editing operations to be performed on said input data; and

rendering means for generating said output data by performing said set of operations on said input data.

44. A system according to claim 43 in which said input data comprises visual data and music data and said output data is an audiovisual production which comprises portions of both said music data and said visual data.

45. A system according to claim 44 in which

one or more of said descriptors are derived from said music data, and

said descriptors are used in conjunction with said style information to determine the selection, processing and timing of a set of visual editing operations, and

said visual editing operations are applied to said visual data,

whereby a music-based production is generated in which the selection, processing and timing of visual elements is controlled or influenced by the characteristics and timing of said music data.

46. A media player for playing input media data in a modified form, said input media data including visual data, the media player including:

analysis means for analyzing said input media data to generate one or more descriptors characterizing each of a plurality of portions of the input media data;

style definition means for defining style information for controlling modifications to be applied to said input media data;

construction means for using said descriptors and said style information to generate a set of one or more editing decisions specifying the modification operations to be applied to said input media data; and

rendering and playback means for playing said input media data in modified form by performing said set of modification operations on said input media data and delivering the modified input media data to an audiovisual display.

47. A system for editing input data comprising motion video data and music to form output data comprising motion video data with an associated sound track, the system including

analysis means for analyzing said music to generate one or more descriptors characterizing beats in said music,

style definition means for defining style information for controlling the editing of said motion video data, comprising means for a user to select a style from a plurality of styles,

construction means for using said descriptors and said style information, to generate a set of editing decisions specifying editing operations to be performed on said motion video data, where at least one of said editing operations is a cut or transition whose timing is influenced by the beat descriptors;

and

rendering means for generating said output data by performing said editing operations on said input data.

48. A system according to claim 47 further comprising means for displaying a graphical user interface including visual thumbnails.

49. A method for editing input data comprising motion video data and music to form output data comprising motion video data with an associated sound track and the method includes the steps of

analyzing said music to generate one or more descriptors characterizing beats in said music,

defining style information for controlling the editing of said motion video data by a user selecting a style from a plurality of styles,

using said descriptors and said style information, to generate a set of editing decisions specifying editing operations to be performed on said motion video data, where at least one of said editing operations is a cut or transition whose timing is influenced by the beat descriptors;

and

generating said output data by performing said editing operations on said input data.

50. A method according to claim 49 including the step of displaying a graphical user interface including visual thumbnails.