Method of and system for spotting digital media objects and event markers using musical experience descriptors to characterize digital music to be automatically composed and generated by an automated music composition and generation engine

• US 10,672,371 B2
• Filed: 12/13/2018
• Issued: 06/02/2020
• Est. Priority Date: 09/29/2015
• Status: Active Grant

• Current Assignee: Shutterstock Incorporated
• Original Assignee: Amper Music, Inc. (Shutterstock Incorporated)
• Inventors: Silverstein, Andrew H.
• Primary Examiner(s): Warren, David S
• Application Number:
  US16/219,299
• Publication Number:
  US 20190279606A1
• Time in Patent Office: 537 Days
• Field of Search: 84609, 84600
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G10H 1/00 Details of electrophonic musical instruments
• CPC Class Codes: G10H 1/0025 Automatic or semi-automatic music composition, e.g. producing random music, applying rules from music theory or modifying a musical piece automatically producing a series of tones G10H1/26
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/021 Background music, e.g. for video sequences, elevator music musical accompaniment G10H2210/005
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/105 Composing aid, e.g. for supporting creation, edition or modification of a piece of music
• CPC Class Codes: G10H 2210/111 Automatic composing, i.e. using predefined musical rules
• CPC Class Codes: G10H 2210/115 using a random process to generate a musical note, phrase, sequence or structure using a random process to build a rhythm pattern G10H2210/356; random rhythm pattern selection G10H2210/366
• CPC Class Codes: G10H 2210/341 Rhythm pattern selection, synthesis or composition Rhythm analysis G10H2210/071; accompaniment G10H2210/005
• CPC Class Codes: G10H 2220/101 for graphical creation, edition or control of musical data or parameters
• CPC Class Codes: G10H 2240/081 Genre classification, i.e. descriptive metadata for classification or selection of musical pieces according to style analysis genre G10H2210/036
• CPC Class Codes: G10H 2240/085 Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
• CPC Class Codes: G10H 2240/131 Library retrieval, i.e. searching a database or selecting a specific musical piece, segment, pattern, rule or parameter set
• CPC Class Codes: G10H 2240/305 Internet or TCP/IP protocol use for any electrophonic musical instrument data or musical parameter transmission purposes
• CPC Class Codes: G10H 2250/311 Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation musical analysis G10H2210/031; neural networks G06N3/02
• CPC Class Codes: G10L 25/15 the extracted parameters being formant information

Abstract

An automated music composition and generation system and process for scoring a selected media object or event marker, with one or more pieces of digital music, by spotting the selected media object or event marker with musical experience descriptors selected and applied to the selected media object or event marker by the system user during a scoring process, and using said selected musical experience descriptors to drive an automated music composition and generation engine to automatically compose and generate the one or more pieces of digital music.

Claims

1. An automated music composition and generation system for spotting a digital media object or event marker with one or more musical experience descriptors during a scoring process using musical experience descriptors to characterize one or more pieces of digital music to be automatically composed and generated by an automated music composition and generation engine, for use in scoring said digital media object or event marker, said automated music composition and generation system comprising:

an automated music composition and generation engine configured for receiving, as inputs, musical experience descriptors being selected by a system user while spotting a digital media object or event marker during a scoring process, and producing, as output, one or more pieces of digital music automatically composed and generated by said automated music composition and generation engine based on said selected musical experience descriptors supplied to said automated music composition and generation engine; and

a system user interface, operably connected to said automated music composition and generation engine, and configured for (i) selecting a digital media object or event marker to be scored with pieces of digital music automatically composed and generated by said automated music composition and generation engine, (ii) selecting said musical experience descriptors from a group consisting of emotion-type musical experience descriptors, style-type musical experience descriptors, timing-type musical experience descriptors, and accent-type musical experience descriptors, (iii) applying the selected musical experience descriptors to said selected digital media object or event marker, and (iv) providing the selected musical experience descriptors to said automated music composition and generation engine and automatically composing and generating one or more pieces of digital music characterized by said selected musical experience descriptors, and for musically scoring said selected digital media object or event marker to produce a musically-scored digital media object or event marker;

wherein after selecting said digital media object or event marker, and during spotting of said selected digital media object or event marker,

(i) selecting and applying one or more of said musical experience descriptors to said selected digital media object or event marker to indicate when and how particular musical events should occur in said one or more pieces of digital music automatically composed and generated by said automated music composition and generation engine, for musically scoring said selected digital media object or event marker;

(ii) providing said selected and applied musical experience descriptors to said automated music composition and generation engine, and

(iii) said automated music composition and generation engine automatically composing and generating one or more pieces of digital music characterized by said selected and applied musical experience descriptors.

2. The automated music composition and generation system of claim 1, wherein said system user interface, operably connected to said automated music composition and generation engine, is provided and configured for selecting and applying said musical experience descriptors to said selected digital media object or event marker during spotting.

3. The automated music composition and generation system of claim 1, wherein said digital media object is selected from the group consisting of a video, a podcast, an audio-recording, a digital image, a photograph, a slideshow, an event-marker, and other events.

4. The automated music composition and generation system of claim 1, wherein said timing-type musical experience descriptors and said accent-type musical experience descriptors indicate when musical events occur in the pieces of digital music, and include one or more parameters, commands and/or markers selected from the group consisting of (i) a parameter indicating the length of the piece of digital music, (ii) a marker indicating the timing location of a start in the piece of digital music, (iii) a marker indicating the timing location of a stop in the piece of digital music, (iv) a marker indicating the timing location of an instrument hit in the piece of digital music, (v) a marker indicating the timing location of a fade in the piece of digital music, (vi) a marker indicating the timing location of a fade out in the piece of digital music, (vii) a marker indicating the timing location of a modulation in the piece of digital music, (viii) a marker indicating the timing location of an increase in volume in the piece of digital music, and (ix) a marker indicating the timing location of a particular accent in the piece of digital music, (x) a marker indicating the timing location of a new emotion or mood to be conveyed by the piece of digital music, (xi) a marker indicating the timing location of a change in style in the piece of digital music, (xii) a marker indicating the timing location of a change in instrumentation in the piece of digital music, and (xiii) a marker indicating the timing location of a change in the structure of the piece of digital music.

5. The automated music composition and generation system of claim 1, wherein said musical experience descriptors have a graphical-icon format and/or linguistic format.

6. The automated music composition and generation system of claim 1, wherein said system user interface is an interface selected from the group consisting of a text keyboard, a manual data entry device, a speech recognition interface, a graphical user interface (GUI), and a touch-screen graphical user interface (GUI).

7. An automated music composition and generation process supported within an automated music composition and generation system for spotting a digital media object or event marker with one or more musical experience descriptors during a scoring process using musical experience descriptors to characterize one or more pieces of digital music to be automatically composed and generated by an automated music composition and generation engine, for use in musically scoring said digital media object or event marker, said automated music composition and generation process comprising the steps of:

(a) selecting a digital media object or event marker to be spotted by a system user with musical experience descriptors, and scored with one or more pieces of digital music automatically generated by an automated music composition and generation system;

(b) during spotting of the selected digital media object or event marker, selecting one or more musical experience descriptors to be applied to said selected digital media object or event marker, and applying the selected musical experience descriptors to said selected digital media object or event marker to indicate when and how particular musical events should occur in said one or more pieces of digital music automatically composed and generated by said automated music composition and generation engine, for musically scoring said selected digital media object or event marker; and

(c) providing said selected and applied musical experience descriptors to said automated music composition and generation engine and automatically composing and generating the one or more pieces of digital music characterized by said selected and applied musical experience descriptors, for use in scoring said selected digital media object or event marker with said one or more pieces of digital music.

8. The automated music composition and generation process of claim 7, wherein step (a) comprises providing a system user interface, operably connected to said automated music composition and generation engine, and configured for (i) selecting said digital media object or event marker to be scored with pieces of digital music automatically composed and generated by said automated music composition and generation engine.

9. The automated music composition and generation process of claim 7, wherein step (b) comprises providing a system user interface, operably connected to said automated music composition and generation engine, and configured for selecting and applying said musical experience descriptors to said selected digital media object or event marker during spotting.

10. The automated music composition and generation process of claim 7, wherein step (b) comprises selecting said musical experience descriptors selected from a group consisting of emotion-type musical experience descriptors, style-type musical experience descriptors, timing-type musical experience descriptors, and accent-type musical experience descriptors.

11. The automated music composition and generation process of claim 10, wherein said timing-type musical experience descriptors and said accent-type musical experience descriptors indicate when musical events occur in the pieces of digital music, and include one or more parameters, commands and/or markers selected from the group consisting of (i) a parameter indicating the length of the piece of digital music, (ii) a marker indicating the timing location of a start in the piece of digital music, (iii) a marker indicating the timing location of a stop in the piece of digital music, (iv) a marker indicating the timing location of an instrument hit in the piece of digital music, (v) a marker indicating the timing location of a fade in the piece of digital music, (vi) a marker indicating the timing location of a fade out in the piece of music, (vii) a marker indicating the timing location of a modulation in the piece of digital music, (viii) a marker indicating the timing location of an increase in volume in the piece of digital music, and (ix) a marker indicating the timing location of a particular accent in the piece of digital music, (x) a marker indicating the timing location of a new emotion or mood to be conveyed by the piece of digital music, (xi) a marker indicating the timing location of a change in style in the piece of digital music, (xii) a marker indicating the timing location of a change in instrumentation in the piece of digital music, and (xiii) a marker indicating the timing location of a change in the structure of the piece of digital music.

12. The automated music composition and generation process of claim 7, which further comprises step (d) combining said one or more pieces of digital music with said selected digital media object or event marker, so as to produce a musically-scored digital media object or event marker.

13. The automated music composition and generation process of claim 7, wherein said system user interface is an interface selected from the group consisting of a text keyboard, a manual data entry device, a speech recognition interface, a graphical user interface (GUI), and a touch-screen graphical user interface (GUI).

14. An automated music composition and generation process for spotting a digital media object or event marker with one or more musical experience descriptors during a scoring process using musical experience descriptors to characterize one or more pieces of digital music to be automatically composed and generated by an automated music composition and generation engine, for use in musically scoring said digital media object or event marker, said automated music composition and generation process comprising the steps of:

(a) a system user accessing a system user interface, and selecting a digital media object or event marker to be scored with one or more pieces of digital music automatically composed and generated by said automated music composition and generation engine;

(b) during spotting of said digital media object or event marker, the system user selecting and applying one or more of said musical experience descriptors to said selected digital media object or event marker to indicate when and how particular musical events should occur in said one or more pieces of digital music automatically composed and generated by said automated music composition and generation engine, for musically scoring said selected digital media object or event marker;

(c) providing said selected and applied musical experience descriptors to said automated music composition and generation engine, and said automated music composition and generation engine automatically composing and generating said one or more pieces of digital music characterized by said selected and applied musical experience descriptors, and for use in musically scoring said selected digital media object or event marker; and

(d) combining said one or more pieces of digital music with said selected digital media object or event marker so as to create a digital file representing a musically-scored digital media object or event marker, for display and review by said system user.

15. The automated music composition and generation process of claim 14, which further comprises:

(e) reviewing and assessing said digital file and making modifications to one or more selected musical experience descriptors;

(f) providing the modified musical experience descriptors to said automated music composition and generation engine; and

(g) initiating said automated music composition and generation engine to compose and generate a new digital file for display and review.

16. The automated music composition and generation process of claim 14, wherein said digital media object is selected from the group consisting of a video, a podcast, an audio-recording, a digital image, a photograph, a slideshow, an event-marker, and other events.

17. The automated music composition and generation process of claim 14, wherein said timing-type musical experience descriptors and said accent-type musical experience descriptors indicate when musical events occur in the pieces of digital music, and include one or more parameters, commands and/or markers selected from the group consisting of (i) a parameter indicating the length of the piece of digital music, (ii) a marker indicating the timing location of a start in the piece of digital music, (iii) a marker indicating the timing location of a stop in the piece of digital music, (iv) a marker indicating the timing location of an instrument hit in the piece of digital music, (v) a marker indicating the timing location of a fade in the piece of digital music, (vi) a marker indicating the timing location of a fade out in the piece of digital music, (vii) a marker indicating the timing location of a modulation in the piece of digital music, (viii) a marker indicating the timing location of an increase in volume in the piece of digital piece, and (ix) a marker indicating the timing location of a particular accent in the piece of digital music, (x) a marker indicating the timing location of a new emotion or mood to be conveyed by the piece of digital music, (xi) a marker indicating the timing location of a change in style in the piece of digital music, (xii) a marker indicating the timing location of a change in instrumentation in the piece of digital music, and (xiii) a marker indicating the timing location of a change in the structure of the piece of digital music.

18. The automated music composition and generation process of claim 14, wherein said musical experience descriptors have a graphical-icon and/or linguistic format.

19. The automated music composition and generation process of claim 14, wherein said system user interface is an interface selected from the group consisting of a text keyboard, a manual data entry device, a speech recognition interface, a graphical user interface (GUI), and a touch-screen graphical user interface (GUI).

20. The automated music composition and generation process of claim 14, wherein said digital media object is a video, and wherein the system user (i) selects a video from a video library maintained within a storage device, (ii) selects and applies one or more musical experience descriptors to the selected video, and (iii) provides the musical experience descriptors to said automated music composition and generation engine, for use in automatically composing and generating pieces of digital music for scoring said selected video.