Automated music composition and generation system for spotting digital media objects and event markers using emotion-type, style-type, timing-type and accent-type musical experience descriptors that characterize the digital music to be automatically composed and generated by the system

• US 10,467,998 B2
• Filed: 04/17/2017
• Issued: 11/05/2019
• Est. Priority Date: 09/29/2015
• Status: Active Grant

• Current Assignee: Shutterstock Incorporated
• Original Assignee: Amper Music, Inc. (Shutterstock Incorporated)
• Inventors: Silverstein, Andrew H.
• Primary Examiner(s): Warren, David S
• Application Number:
  US15/489,701
• Publication Number:
  US 20170263226A1
• Time in Patent Office: 932 Days
• Field of Search: 84600, 84609
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G10H 1/00 Details of electrophonic musical instruments
• CPC Class Codes: G10H 1/0025 Automatic or semi-automatic music composition, e.g. producing random music, applying rules from music theory or modifying a musical piece automatically producing a series of tones G10H1/26
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/021 Background music, e.g. for video sequences or elevator music
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/105 Composing aid, e.g. for supporting creation, edition or modification of a piece of music
• CPC Class Codes: G10H 2210/111 Automatic composing, i.e. using predefined musical rules
• CPC Class Codes: G10H 2210/115 using a random process to generate a musical note, phrase, sequence or structure
• CPC Class Codes: G10H 2210/341 Rhythm pattern selection, synthesis or composition
• CPC Class Codes: G10H 2220/101 for graphical creation, edition or control of musical data or parameters
• CPC Class Codes: G10H 2240/081 Genre classification, i.e. descriptive metadata for classification or selection of musical pieces according to style
• CPC Class Codes: G10H 2240/085 Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
• CPC Class Codes: G10H 2240/131 Library retrieval, i.e. searching a database or selecting a specific musical piece, segment, pattern, rule or parameter set
• CPC Class Codes: G10H 2240/305 Internet or TCP/IP protocol use for any electrophonic musical instrument data or musical parameter transmission purposes
• CPC Class Codes: G10H 2250/311 Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation
• CPC Class Codes: G10L 25/15 the extracted parameters being formant information

Abstract

An autonomous music composition and performance system employing an automated music composition and generation engine configured to receive musical signals from a set of a real or synthetic musical instruments being played by a group of human musicians. The system buffers and analyzes musical signals from the set of real or synthetic musical instruments, composes and generates music in real-time that augments the music being played by the band of musicians, and/or records, analyzes and composes music recorded for subsequent playback, review and consideration by the human musicians.

Claims

1. An automated music composition and generation system for scoring a selected media object or event marker, with one or more custom pieces of digital music, by spotting the selected media object or event marker with musical experience descriptors selected and applied to the selected media object or event marker by the system user during a scoring process, and using said selected musical experience descriptors to drive an automated music composition and generation engine to automatically compose and generate the one or more custom pieces of digital music, said automated music composition and generation system comprising:

an automated music composition and generation engine configured for receiving, as inputs, musical experience descriptors being selected by a system user while spotting a selected media object or event marker during a scoring process, and producing, as output, one or more custom pieces of digital music automatically composed and generated by said automated music composition and generation engine based on said selected musical experience descriptors supplied to said automated music composition and generation engine; and

a system user interface, operably connected to said automated music composition and generation engine, and configured for (i) selecting a media object or event marker to be scored by the system user with custom pieces of digital music automatically composed and generated by said automated music composition and generation engine, (ii) selecting said musical experience descriptors from a group consisting of emotion-type musical experience descriptors, style-type musical experience descriptors, timing-type musical experience descriptors, and accent-type musical experience descriptors, and (iii) applying the selected musical experience descriptors to said selected media object or event marker, and providing said selected and applied musical experience descriptors to said automated music composition and generation engine for use in automatically composing and generating custom pieces of digital music characterized by said selected musical experience descriptors, and for scoring said selected media object or event marker to produce a musically-scored media object or event marker;

wherein after the system user selects a media object or event marker during spotting, the system user selects and applies one or more of said musical experience descriptors to said selected media object or event marker;

wherein said musical experience descriptors specify what, when and how musical events occur in the custom pieces of digital music;

wherein said automated music composition and generation engine automatically generates custom pieces of digital music for scoring the selected media object or event marker, based on and using said musical experience descriptors (i) selected by the system user from said system user interface, (ii) applied to the selected media object or event marker during spotting, and (iii) provided to said automated music composition and generation engine; and

wherein said automated music composition and generation engine, once initiated by the system user, automatically transforms said selected musical experience descriptors into a set of music-theoretic system operating parameters, which are used by said automated music composition and generation engine to automatically compose and generate said custom pieces of digital music for scoring said selected media object or event marker.

2. The automated music composition and generation system of claim 1, wherein said media object is selected from the group consisting of a video, a podcast, an audio-recording, a digital image or photograph, a slideshow, an event-marker, and other events.

3. The automated music composition and generation system of claim 1, which further comprises an editing subsystem for (i) reviewing and assessing said digital file, (ii) making modifications to one or more selected musical experience descriptors, (iii) providing the modified musical experience descriptors to said automated music composition and generation engine, and (iv) then initiating said automated music composition and generation engine to compose and generate a new digital file for display and review;

wherein said automated music composition and generation engine, once re-initiated by the system user, automatically transforms said modified musical experience descriptors into a new set of music-theoretic system operating parameters, which are used by said automated music composition and generation engine to automatically compose and generate one or more modified custom pieces of digital music for scoring said selected digital media object or event marker to produce said musically-scored media object or event marker.

4. The automated music composition and generation system of claim 1, wherein said timing-type musical experience descriptors and said accent-type musical experience descriptors indicate when musical events occur in the custom pieces of digital music, and include one or more parameters, commands and/or markers selected from the group consisting of (i) a parameter indicating the length of the custom piece of digital music, (ii) a marker indicating the timing location of a start in the custom piece of digital music, (iii) a marker indicating the timing location of a stop in the custom piece of digital music, (iv) a marker indicating the timing location of an instrument hit in the custom piece of digital music, (v) a marker indicating the timing location of a fade in the custom piece of digital music, (vi) a marker indicating the timing location of a fade out in the custom piece of digital music, (vii) a marker indicating the timing location of a modulation in the custom piece of digital music, (viii) a marker indicating the timing location of an increase in volume in the custom piece of digital piece music, and (ix) a marker indicating the timing location of a particular accent in the custom of digital music, (x) a marker indicating the timing location of a new emotion or mood to be conveyed by the custom piece of digital music, (xi) a marker indicating the timing location of a change in style in the custom piece of digital music, (xii) a marker indicating the timing location of a change in instrumentation in the custom piece of digital music, and (xiii) a marker indicating the timing location of a change in the structure of the custom piece of digital music.

5. The automated music composition and generation system of claim 1, wherein said musical experience descriptors has have a graphical-icon format and/or a linguistic format.

6. The automated music composition and generation system of claim 1, wherein said media object or event marker is scored with said one or more custom pieces of digital music composed and generated using a virtual instrument music synthesis method and said selected and applied musical experience descriptors.

7. The automated music composition and generation system of claim 1, wherein said system user interface is comprises an interface selected from the group consisting of a text keyboard, a manual data entry device, a speech recognition interface, a graphical user interface (GUI), and a touch-screen graphical user interface (GUI).

8. The automated music composition and generation system of claim 7, wherein said selection mechanism allows system users to spot selected digital media objects or event markers with one or more custom pieces of digital music automatically composed and generated by said automated music composition and generation engine using musical experience descriptors selected using said interface.

9. The automated music composition and generation system of claim 1, wherein said automated music composition and generation engine employs a virtual-instrument music synthesis method during automated music composition and generation.

10. An automated music composition and generation process supported within an automated music composition and generation system for scoring a selected media object or event marker, with one or more custom pieces of digital music, by spotting the selected media object or event marker with musical experience descriptors selected and applied to the selected media object or event marker by the system user during a scoring process, and using said selected musical experience descriptors to drive an automated music composition and generation engine to automatically compose and generate the one or more custom pieces of digital music, said automated music composition and generation process comprising the steps of:

(a) providing a system user interface, operably connected to said automated music composition and generation engine, and configured for (i) selecting a media object or event marker to be scored by the system user with custom pieces of digital music automatically composed and generated by said automated music composition and generation engine, (ii) selecting said musical experience descriptors selected from a group consisting of emotion-type musical experience descriptors, style-type musical experience descriptors, timing-type musical experience descriptors, and accent-type musical experience descriptors, and (iii) applying the selected musical experience descriptors to said selected media object or event marker, and providing the selected musical experience descriptors to said automated music composition and generation engine for use in automatically composing and generating custom pieces of digital music characterized by said selected musical experience descriptors, and for scoring said selected media object or event marker to produce a musically-scored media object or event marker;

wherein said automated music composition and generation engine is configured for receiving, as inputs, musical experience descriptors being selected by a system user while spotting during a scoring process, and producing, as output, one or more custom pieces of digital music automatically composed and generated by said automated music composition and generation engine based on said selected musical experience descriptors supplied to said automated music composition and generation engine;

(b) a system user accessing said system user interface, and selecting a a media object or event marker to be scored with custom pieces of digital music composed and generated by said automated music composition and generation engine;

(c) during spotting, the system user selecting and applying one or more of said musical experience descriptors to said selected media object or event marker;

wherein said musical experience descriptors indicate what, when and how musical events occur in the custom pieces of digital music;

(d) providing said selected musical experience descriptors to said automated music composition and generation engine;

(e) initiating said automated music composition and generation engine, to automatically transform said selected musical experience descriptors into a set of music-theoretic system operating parameters, which are used by said automated music composition and generation engine to automatically compose and generate custom pieces of digital music for scoring scenes in said selected media object or event marker; and

(f) said automated music composing and generation system combining the composed custom pieces of digital music with the selected media object or event marker so as to create a digital file, representing the musically-scored media object or event marker, for display and review by said system user.

11. The automated music composition and generation process of claim 10, which further comprises:

(g) reviewing and assessing said digital file and making modifications to one or more selected musical experience descriptors;

(h) providing the modified musical experience descriptors to said automated music composition and generation engine; and

(i) initiating said automated music composition and generation engine to compose and generate a new digital file for display and review;

wherein said automated music composition and generation engine, once re-initiated by the system user, automatically transforms said modified musical experience descriptors into a new set of music-theoretic system operating parameters, used by said automated music composition and generation engine to automatically compose and generate one or more modified custom pieces of digital music for scoring said selected media object or event marker.

12. The automated music composition and generation process of claim 10, wherein said media object is selected from the group consisting of a video, a podcast, an audio-recording, a digital image or photograph, a slideshow, an event-marker, and other events.

13. The automated music composition and generation process of claim 10, wherein said timing-type musical experience descriptors and said accent-type musical experience descriptors indicate when musical events occur in the custom pieces of digital music, and include one or more parameters, commands and/or markers selected from the group consisting of (i) a parameter indicating the length of the custom piece of digital music, (ii) a marker indicating the timing location of a start in the custom piece of digital music, (iii) a marker indicating the timing location of a stop in the custom piece of digital music, (iv) a marker indicating the timing location of an instrument hit in the custom piece of digital music, (v) a marker indicating the timing location of a fade in the custom piece of digital music, (vi) a marker indicating the timing location of a fade out in the custom piece of digital music, (vii) a marker indicating the timing location of a modulation in the custom piece of digital music, (viii) a marker indicating the timing location of an increase in volume in the custom piece of digital piece music, and (ix) a marker indicating the timing location of a particular accent in the custom piece of digital music, (x) a marker indicating the timing location of a new emotion or mood to be conveyed by the custom piece of digital music, (xi) a marker indicating the timing location of a change in style in the custom piece of digital music, (xii) a marker indicating the timing location of a change in instrumentation in the custom piece of digital music, and (xiii) a marker indicating the timing location of a change in the structure of the custom piece of digital music.

14. The automated music composition and generation process of claim 10, wherein said musical experience descriptors has have a graphical-icon and/or linguistic format.

15. The automated music composition and generation process of claim 10, wherein said media object or event marker is scored with said one or more custom pieces of digital music composed and generated using a virtual instrument music synthesis method and said selected and applied musical experience descriptors.

16. The automated music composition and generation process of claim 10, wherein said system user interface is selected from the group consisting of a text keyboard, a manual data entry device, a speech recognition interface, a graphical user interface (GUI), and a touch-screen graphical user interface (GUI).

17. The automated music composition and generation process of claim 10, wherein said system user interface allows system users to spot selected digital media objects or event markers, with one or more custom pieces of digital music said musical experience descriptors selected and applied using said system user interface.

18. The automated music composition and generation process of claim 10, wherein said automated music composition and generation engine employs a virtual-instrument music synthesis method during automated music composition and generation.

19. The automated music composition and generation process of claim 10, wherein said media object is a video, and wherein the system user (i) selects a video from a video library maintained within a storage device, (ii) selects and applies one or more musical experience descriptors to the selected video, and (iii) provides said selected and applied musical experience descriptors to said automated music composition and generation engine, for use in automatically composing and generating custom pieces of digital music for scoring in said selected video.