AUTOMATED MUSIC COMPOSITION SYSTEM AND METHOD DRIVEN BY LYRICS AND EMOTION AND STYLE TYPE MUSICAL EXPERIENCE DESCRIPTORS

• US 20170263228A1
• Filed: 04/17/2017
• Published: 09/14/2017
• Est. Priority Date: 09/29/2015
• Status: Active Grant

• Current Assignee: Shutterstock Incorporated
• Original Assignee: Amper Music, Inc. (Shutterstock Incorporated)
• Inventors: Silverstein, Andrew H.
• Granted Patent: US 10,311,842 B2
• Time in Patent Office: Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G10H 1/00 Details of electrophonic musical instruments
• CPC Class Codes: G10H 1/0025 Automatic or semi-automatic music composition, e.g. producing random music, applying rules from music theory or modifying a musical piece automatically producing a series of tones G10H1/26
• CPC Class Codes: G10H 1/368 displaying animated or moving pictures synchronized with the music or audio part
• CPC Class Codes: G10H 2210/021 Background music, e.g. for video sequences, elevator music musical accompaniment G10H2210/005
• CPC Class Codes: G10H 2210/066 for pitch analysis as part of wider processing for musical purposes, e.g. transcription, musical performance evaluation; Pitch recognition, e.g. in polyphonic sounds; Estimation or use of missing fundamental
• CPC Class Codes: G10H 2210/105 Composing aid, e.g. for supporting creation, edition or modification of a piece of music
• CPC Class Codes: G10H 2210/111 Automatic composing, i.e. using predefined musical rules
• CPC Class Codes: G10H 2210/115 using a random process to generate a musical note, phrase, sequence or structure using a random process to build a rhythm pattern G10H2210/356; random rhythm pattern selection G10H2210/366
• CPC Class Codes: G10H 2210/341 Rhythm pattern selection, synthesis or composition Rhythm analysis G10H2210/071; accompaniment G10H2210/005
• CPC Class Codes: G10H 2220/101 for graphical creation, edition or control of musical data or parameters
• CPC Class Codes: G10H 2240/081 Genre classification, i.e. descriptive metadata for classification or selection of musical pieces according to style analysis genre G10H2210/036
• CPC Class Codes: G10H 2240/085 Mood, i.e. generation, detection or selection of a particular emotional content or atmosphere in a musical piece
• CPC Class Codes: G10H 2240/131 Library retrieval, i.e. searching a database or selecting a specific musical piece, segment, pattern, rule or parameter set
• CPC Class Codes: G10H 2240/305 Internet or TCP/IP protocol use for any electrophonic musical instrument data or musical parameter transmission purposes
• CPC Class Codes: G10H 2250/311 Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation musical analysis G10H2210/031; neural networks G06N3/02
• CPC Class Codes: G10L 25/15 the extracted parameters being formant information

Abstract

An automated music composition and generation process within an automated music composition and generation system driven by lyrical musical experience descriptors. The process involves the system user accessing said automated music composition and generation system, employing an automated music composition and generation engine having a system user interface. The system user interface is used to select and provide musical experience descriptors, including lyrics, to the automated music composition and generation engine for processing by said automated music composition and generation engine. The system user initiates the automated music composition and generation engine to compose and generate music based on the provided musical descriptors including lyrics.

Claims

1. An automated music composition and generation system comprising:

a system user interface for receiving musical experience descriptors and lyrical word descriptions from a system user;

a real-time pitch event analyzing subsystem for analyzing said lyrical word description and generating corresponding pitch events; and

an automated music composition and generation engine operably connected to said system user interface, and said real-time pitch event analyzing subsystem;

wherein said musical experience descriptors and/or said pitch events being used by said automated music composition and generation engine to automatically compose and generate a piece of music for emotionally scoring media or an event marker with said piece of music.

2. The automated music composition and generation system of claim 1, wherein said musical experience descriptors and said lyrical word descriptions are produced using a keyboard and/or a speech recognition interface operably connected to said system user interface.

3. The automated music composition and generation system of claim 1, wherein said media is selected from the group consisting of video recordings, slide-shows, audio recordings, and event markers.

4. The automated music composition and generation system of claim 1, wherein said automated music composition and generation engine uses said musical experience descriptors and/or said pitch events to generate said piece of music.

5. The automated music composition and generation system of claim 4, wherein said automated piece of music comprises a piece of musically-scored media, a music file and/or hard-copy sheet music.

6. The automated music composition and generation system of claim 1, wherein said system user interface further comprises transmitting typed, spoken or sung speech or lyrical input to said real-time pitch event analyzing subsystem, for real-time pitch event, rhythmic and prosodic analysis to generate different pitch-event streams for typed, spoken and sung lyrics, respectively, which are subsequently used to compose and generate music in said automated music composition and generation system during the music composition and generation process.

7. The automated music composition and generation system of claim 1, wherein said real-time pitch event analyzing subsystem comprises a vowel formant analyzer.

8. An automated music composition and generation process within a music composing and generation system driven by lyrical input, said automated music composition and generation process comprising the steps of:

(a) the system user accesses said automated music composition and generation engine, and then selects media to be scored with music generated by an automated music composition and generation system;

(b) the system user provides lyrical input to said automated music composition and generation system, and selects media to be musically-scored;

(c) said automated music composition and generation engine using a pitch and rhythm extraction subsystem to automatically extracting pitch information from said lyrical input to extract pitch events that are provided to said automated music composition and generation engine to automatically compose and generate music based on said lyrical input; and

(d) said system combining the composed music with the selected media so as to create a composite media file for display and enjoyment.

9. The automated music composition and generation process of claim 8, wherein said pitch and rhythm extraction subsystem is used to capture real-time pitch and rhythm information from system user supplied lyrics or music which is then provided to a lyrical/musical responsive parameter value selection mechanism supported in each subsystem.

10. A method of composing and generating music in an automated manner using a real-time pitch event analyzing subsystem in an automated music composition and generation system, said method comprises the following sequence of steps:

(a) providing musical experience descriptors to a system user interface of said automated music composition and generation system;

(b) providing lyrical input, in typed, spoken or sung format, to said system-user interface, for one or more scenes in a video or media object to be scored with music composed and generated by said automated music composition and generation system;

(c) using said real-time pitch event analyzing subsystem to extract pitch events, rhythmic information and/or prosodic information from the analyzed lyrical input, and coding with timing information on when such detected events occurred;

(d) encoding extracting pitch events, rhythmic information and prosodic information to precisely indicate when such detected events occurred along a time line; and

(e) providing the extracted pitch event, rhythmic and prosodic information to said automated music composition and generation system for use in constraining the system operating parameters employed in the various subsystems of said automated music composition and generation system.

11. The method of claim 10, wherein said musical experience descriptors including emotion-type musical experience descriptors, and style-type musical experience descriptors.

12. The method of claim 10, wherein said musical experience descriptors are supplied by a device selected from the group consisting of a keyboard data entry, speech recognition, and devices known in the data entry and handling arts.

13. The method of claim 10, wherein the system user speaks or sings lyrics for the intended media piece or section, for which the lyrics are intended to sent a tone, rhythm and melody for at least a limited number of notes in the music to be composed and generated by said automated music composition and generation system.

14. The method of claim 10, wherein speech signals corresponding to spoken or sung lyrics or words are digitalized and processed using a digital signal processing (DSP) chip programmed to carryout real-time rhythmic, pitch event, and/or prosodic analysis of typed/spoken/sung lyrics or words.

15. The method of claim 10, wherein said digital signal processing (DSP) chip employs vowel formant analysis to ascertain the occurrence of vowels in the lyrics, and wherein said vowels are used to generate notes of corresponding pitch to obtain a sense of melody from the lyrical input supplied to said automated music composition and generation system.

16. The method of claim 10, wherein a system input output interface allows the system user to transmit lyrical input to said automated music composition and generation system in the form of typed words, spoken words and/or sung speech, in any natural language supported by said automated music composition and generation system.

17. The method of claim 10, wherein said real-time pitch event analyzing subsystem comprises:

a lyrical input handler for handling the different forms of lyrical input supplied by the system user;

a pitch-event output handler for handling the different pitch event output streams generated by said real-time pitch event analyzing subsystem;

a lexical dictionary for storing linguistic information and models on each word in the language supported by said real-time pitch event analyzing system;

a vowel-format analyzer for analyzing the vowel-formants contained in processed lyrical input; and

a mode controller for controlling the lyrical input mode of said real-time pitch event analyzing subsystem.

18. An automated music composition and generation process within an automated music composition and generation system driven by lyrical musical experience descriptors, said automated music composition and generation process comprising the steps:

(a) the system user accessing said automated music composition and generation system, employing an automated music composition and generation engine having a system user interface;

(b) the system user using said system user interface to select and provide musical experience descriptors, including lyrics, to said automated music composition and generation engine for processing by said automated music composition and generation engine; and

(c) the system user initiating said automated music composition and generation engine to compose and generate music based on said provided musical descriptors including said lyrics.

19. The automated music composition and generation process of claim 18, wherein said automated music composition and generation engine comprising the steps of:

(a) receiving text-based lyrical input as a string of graphemes or morphemes;

(b) automatically transcribing said string of graphemes or morphemes into a phonetic string of phonemes, making use of a dictionary;

(c) based on the phonemes in the phonetic string, automatically transforming the vowels present in the phoneme string generates, into a string of vowel formats;

(d) automatically transforming the detected vowel formats into a string of pitch events representing a string of notes, without rhythm information; and

(e) transmitting said string of pitch events to said automated music composition and generation system.

20. The automated music composition and generation process of claim 18, wherein said automated music composition and generation engine comprising the steps of:

(a) receiving spoken lyrical input as an acoustical signal;

(b) automatically processing the acoustical signal using signal processing techniques to generate a phonetic equivalent string, making use of a dictionary and speech recognition methods;

(c) based on said phonemes in the phonetic string, automatically transforming the vowels present in said phoneme string, into a string of vowel formats;

(d) then automatically transforming the detected vowel formats into a string of pitch events with rhythm information;

(e) generating a string of pitch events with rhythm data, from said string of vowel formants; and

(f) transmitting the pitch event and rhythm data to said automated music composition and generation system.

21. The automated music composition and generation process of claim 18, wherein said automated music composition and generation engine comprising the steps of:

(a) receiving at said the sung lyrical input as an acoustical signal that is continuously buffered and processed;

(b) automatically processes said acoustical signal, using signal processing techniques, so as to produce a phonetic equivalent string, making use of a dictionary;

(c) based on said phonemes in the phonetic string, automatically transforming the vowels present in the phoneme string, into a string of vowel formats;

(d) automatically transforming the detected vowel formats into a string of pitch events with rhythm information; and

(e) automatically transmitting the pitch event and rhythm data relating to said automated music composition and generation system.