Electronic musical instrument, electronic musical instrument control method, and storage medium

• US 10,629,179 B2
• Filed: 06/20/2019
• Issued: 04/21/2020
• Est. Priority Date: 06/21/2018
• Status: Active Grant

• Current Assignee: Casio Computer Company Limited
• Original Assignee: Casio Computer Company Limited
• Inventors: Danjyo, Makoto, Ota, Fumiaki, Setoguchi, Masaru, Nakamura, Atsushi
• Primary Examiner(s): Donels, Jeffrey
• Application Number:
  US16/447,630
• Publication Number:
  US 20190392807A1
• Time in Patent Office: 306 Days
• Field of Search: 84602, 84610
• CPC Class Codes: G10H 1/0008 Associated control or indicating means
• CPC Class Codes: G10H 1/125 using a digital filter
• CPC Class Codes: G10H 1/36 Accompaniment arrangements
• CPC Class Codes: G10H 1/366 with means for modifying or correcting the external signal, e.g. pitch correction, reverberation, changing a singer's voice
• CPC Class Codes: G10H 2210/005 Musical accompaniment, i.e. complete instrumental rhythm synthesis added to a performed melody, e.g. as output by drum machines background music G10H2210/021
• CPC Class Codes: G10H 2210/091 for performance evaluation, i.e. judging, grading or scoring the musical qualities or faithfulness of a performance, e.g. with respect to pitch, tempo or other timings of a reference performance
• CPC Class Codes: G10H 2210/191 Tremolo, tremulando, trill or mordent effects, i.e. repeatedly alternating stepwise in pitch between two note pitches or chords, without any portamento between the two notes other common forms of tremolo, e.g. same note repetition, bisbigliando, amplitude tremolo, tremulants, percussion roll G10H2210/161 or G10H2210/205
• CPC Class Codes: G10H 2210/201 Vibrato, i.e. rapid, repetitive and smooth variation of amplitude, pitch or timbre within a note or chord discontinuities, note sequences or separate note onsets during the variation G10H2210/161; tremolo, i.e. stepwise pitch alternation G10H2210/191
• CPC Class Codes: G10H 2210/231 Wah-wah spectral modulation, i.e. tone color spectral glide obtained by sweeping the peak of a bandpass filter up or down in frequency, e.g. according to the position of a pedal, by automatic modulation or by voice formant detection; control devices therefor, e.g. wah pedals for electric guitars
• CPC Class Codes: G10H 2220/011 Lyrics displays, e.g. for karaoke applications
• CPC Class Codes: G10H 2250/015 Markov chains, e.g. hidden Markov models [HMM], for musical processing, e.g. musical analysis or musical composition
• CPC Class Codes: G10H 2250/311 Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation musical analysis G10H2210/031; neural networks G06N3/02
• CPC Class Codes: G10H 2250/455 Gensound singing voices, i.e. generation of human voices for musical applications, vocal singing sounds or intelligible words at a desired pitch or with desired vocal effects, e.g. by phoneme synthesis formant synthesis G10H2250/481; parcor synthesis G10H2250/505; modulation effects G10H2210/195; ensemble effects G10H2210/245; speech synthesis in general G10L13/00
• CPC Class Codes: G10H 2250/625 Interwave interpolation, i.e. interpolating between two different waveforms, e.g. timbre or pitch or giving one waveform the shape of another while preserving its frequency or vice versa
• CPC Class Codes: G10H 7/008 Means for controlling the transition from one tone waveform to another
• CPC Class Codes: G10L 13/033 Voice editing, e.g. manipulating the voice of the synthesiser

Abstract

An electronic musical instrument includes: a memory that stores a machine-learning trained acoustic model mimicking voice of a singer and at least one processor. When a vocoder mode is on, prescribed lyric data and pitch data corresponding to a user operation of an operation element of the musical instrument are inputted to the trained acoustic model, and inferred singing voice data that infers a singing voice of the singer is synthesized on the basis of acoustic feature data output by the trained acoustic model and on the basis of instrument sound waveform data that are synthesized in accordance with the pitch data corresponding to the user operation of the operation element. When the vocoder mode is off, the inferred singing voice data is synthesized based on the acoustic feature data without using the sound waveform data.

Claims

1. An electronic musical instrument comprising:

a plurality of operation elements respectively corresponding to mutually different pitch data;

a memory that stores a trained acoustic model obtained by performing machine learning on training musical score data including training lyric data and training pitch data, and on training singing voice data of a singer corresponding to the training musical score data, the trained acoustic model being configured to receive lyric data and prescribed pitch data and output acoustic feature data of a singing voice of the singer in response to the received lyric data and pitch data; and

at least one processor in which a first mode and a second mode are interchangeably selectable,

wherein in the first mode, the at least one processor;

in accordance with a user operation on an operation element in the plurality of operation elements, inputs prescribed lyric data and pitch data corresponding to the user operation of the operation element to the trained acoustic model so as to cause the trained acoustic model to output the acoustic feature data in response to the inputted prescribed lyric data and the inputted pitch data, and

digitally synthesizes and outputs inferred singing voice data that infers a singing voice of the singer on the basis of at least a portion of the acoustic feature data output by the trained acoustic model in response to the inputted prescribed lyric data and the inputted pitch data, and on the basis of instrument sound waveform data that are synthesized in accordance with the pitch data corresponding to the user operation of the operation element, and

wherein in the second mode, the at least one processor;

in accordance with a user operation on an operation element in the plurality of operation elements, inputs prescribed lyric data and pitch data corresponding to the user operation of the operation element to the trained acoustic model so as to cause the trained acoustic model to output the acoustic feature data in response to the inputted prescribed lyric data and the inputted pitch data, and

digitally synthesizes and outputs inferred singing voice data that infers a singing voice of the singer on the basis of the acoustic feature data output by the trained acoustic model in response to the inputted prescribed lyric data and the inputted pitch data, without using instrument sound waveform data that are synthesized in accordance with the pitch data corresponding to the user operation of the operation element.

2. The electronic musical instrument according to claim 1, wherein the at least one processor switches between the first mode and the second mode based on a user operation of a mode selection operation element provided in the electronic musical instrument.

3. The electronic musical instrument according to claim 1,

wherein the memory contains melody pitch data indicating operation elements that a user is to operate, singing voice output timing data indicating output timings at which respective singing voices for pitches indicated by the melody pitch data are to be output, and lyric data respectively corresponding to the melody pitch data, and

wherein in the first mode, the at least one processor:

when a user operation for producing a singing voice is performed at an output timing indicated by the singing voice output timing data, inputs pitch data corresponding to the user-operated operation element and lyric data corresponding to said output timing to the trained acoustic model, and outputs, at said output timing, inferred singing voice data that infers the singing voice of the singer on the basis of the at least a portion of the acoustic feature data output by the trained acoustic model in response to the input, and

when a user operation for producing a singing voice is not performed at the output timing indicated by the singing voice output timing data, inputs melody pitch data corresponding to said output timing and lyric data corresponding to said output timing to the trained acoustic model, and outputs, at said output timing, inferred singing voice data that infers the singing voice of the singer on the basis of the at least a portion of the acoustic feature data output by the trained acoustic model in response to the input.

4. The electronic musical instrument according to claim 1,

wherein the acoustic feature data of the singing voice of the singer includes spectral data that models a vocal tract of the singer and sound source data that models vocal cords of the singer, and

wherein in the second mode, the at least one processor synthesizes the inferred singing voice data that infers the singing voice of the singer on the basis of the spectral data and the sound source data.

5. The electronic musical instrument according to claim 1, further comprising a selection operation element that, from a plurality of instrument sounds including at least one of a brass sound, a string sound, an organ sound, or an animal cry, specifies one of the instrument sounds in response to a user operation, and

wherein in the first mode, the instrument sound waveform data corresponds to the instrument sound specified by the selection operation element.

6. The electronic musical instrument according to claim 1,

wherein the acoustic feature data of the singing voice of the singer includes spectral data that models a vocal tract of the singer and sound source data that models vocal cords of the singer, and

wherein in the first mode, the at least one processor synthesizes the inferred singing voice data that infers the singing voice of the singer on the basis of the sound source data by applying an acoustic feature of the spectral data to the instrument sound waveform data without using the sound source data of the acoustic feature data.

7. The electronic musical instrument according to claim 1, wherein the trained acoustic model has been trained via machine learning using at least one of a deep neural network or a hidden Markov model.

8. The electronic musical instrument according to claim 1,

wherein the plurality of operation elements include a first operation element as the operation element that was operated by the user and a second operation element that meets a prescribed condition with respect to the first operation element, and

wherein in both of the first and second modes, the at least one processor applies an acoustic effect to the inferred singing voice data when the second operation element is operated while the first operation element is being operated.

9. The electronic musical instrument according to claim 8, wherein the at least one processor changes a depth of the acoustic effect in accordance with a difference in pitch between a pitch corresponding to the first operation element and a pitch corresponding to the second operation element.

10. The electronic musical instrument according to claim 8, wherein the second operation element is a black key.

11. The electronic musical instrument according to claim 8, wherein the acoustic effect includes at least one of a vibrato effect, a tremolo effect, or a wah-wah effect.

12. A method performed by at least one processor in an electronic musical instrument that includes, in addition to the at least one processor:

a plurality of operation elements respectively corresponding to mutually different pitch data; and

a memory that stores a trained acoustic model obtained by performing machine learning on training musical score data including training lyric data and training pitch data, and on training singing voice data of a singer corresponding to the training musical score data, the trained acoustic model being configured to receive lyric data and prescribed pitch data and output acoustic feature data of a singing voice of the singer in response to the received lyric data and pitch data, a first mode and a second mode being interchangeably selectable in the at least one processor, the method comprising, via the at least one processor;

in the first mode;

in accordance with a user operation on an operation element in the plurality of operation elements, inputting prescribed lyric data and pitch data corresponding to the user operation of the operation element to the trained acoustic model so as to cause the trained acoustic model to output the acoustic feature data in response to the inputted prescribed lyric data and the inputted pitch data, and

digitally synthesizing and outputting inferred singing voice data that infers a singing voice of the singer on the basis of at least a portion of the acoustic feature data output by the trained acoustic model in response to the inputted prescribed lyric data and the inputted pitch data, and on the basis of instrument sound waveform data that are synthesized in accordance with the pitch data corresponding to the user operation of the operation element, and

in the second mode;

in accordance with a user operation on an operation element in the plurality of operation elements, inputting prescribed lyric data and pitch data corresponding to the user operation of the operation element to the trained acoustic model so as to cause the trained acoustic model to output the acoustic feature data in response to the inputted prescribed lyric data and the inputted pitch data, and

digitally synthesizing and outputting inferred singing voice data that infers a singing voice of the singer on the basis of the acoustic feature data output by the trained acoustic model in response to the inputted prescribed lyric data and the inputted pitch data, without using instrument sound waveform data that are synthesized in accordance with the pitch data corresponding to the user operation of the operation element.

13. The method according to claim 12, wherein the method includes, via the at least one processor, switching between the first mode and the second mode based on a user operation of a mode selection operation element provided in the electronic musical instrument.

14. The method according to claim 12,

wherein the memory contains melody pitch data indicating operation elements that a user is to operate, singing voice output timing data indicating output timings at which respective singing voices for pitches indicated by the melody pitch data are to be output, and lyric data respectively corresponding to the melody pitch data, and

wherein in the first mode, the method include, via the at least one processor:

when a user operation for producing a singing voice is performed at an output timing indicated by the singing voice output timing data, inputting pitch data corresponding to the user-operated operation element and lyric data corresponding to said output timing to the trained acoustic model, and outputting, at said output timing, inferred singing voice data that infers the singing voice of the singer on the basis of the at least a portion of the acoustic feature data output by the trained acoustic model in response to the input, and

when a user operation for producing a singing voice is not performed at the output timing indicated by the singing voice output timing data, inputting melody pitch data corresponding to said output timing and lyric data corresponding to said output timing to the trained acoustic model, and outputting, at said output timing, inferred singing voice data that infers the singing voice of the singer on the basis of the at least a portion of the acoustic feature data output by the trained acoustic model in response to the input.

15. The method according to claim 12,

wherein the acoustic feature data of the singing voice of the singer includes spectral data that models a vocal tract of the singer and sound source data that models vocal cords of the singer, and

wherein the method includes, in the second mode, causing the at least one processor to synthesize the inferred singing voice data that infers the singing voice of the singer on the basis of the spectral data and the sound source data.

16. The method according to claim 12,

wherein the electronic musical instrument further includes a selection operation element that, from a plurality of instrument sounds including at least one of a brass sound, a string sound, an organ sound, or an animal cry, specifies one of the instrument sounds in response to a user operation, and

wherein in the first mode, the instrument sound waveform data corresponds to the instrument sound specified by the selection operation element.

17. The method according to claim 12,

wherein the acoustic feature data of the singing voice of the singer includes spectral data that models a vocal tract of the singer and sound source data that models vocal cords of the singer, and

wherein in the first mode, the inferred singing voice data that infers the singing voice of the singer is synthesized on the basis of the sound source data by applying an acoustic feature of the spectral data to the instrument sound waveform data without using the sound source data of the acoustic feature data.

18. A non-transitory computer-readable storage medium having stored thereon a program executable by at least one processor in an electronic musical instrument that includes, in addition to the at least one processor:

a plurality of operation elements respectively corresponding to mutually different pitch data; and

a memory that stores a trained acoustic model obtained by performing machine learning on training musical score data including training lyric data and training pitch data, and on training singing voice data of a singer corresponding to the training musical score data, the trained acoustic model being configured to receive lyric data and prescribed pitch data and output acoustic feature data of a singing voice of the singer in response to the received lyric data and pitch data, a first mode and a second mode being interchangeably selectable in the at least one processor, the program causing the at least one processor to perform the following;

in the first mode;

in accordance with a user operation on an operation element in the plurality of operation elements, inputting prescribed lyric data and pitch data corresponding to the user operation of the operation element to the trained acoustic model so as to cause the trained acoustic model to output the acoustic feature data in response to the inputted prescribed lyric data and the inputted pitch data, and

digitally synthesizing and outputting inferred singing voice data that infers a singing voice of the singer on the basis of at least a portion of the acoustic feature data output by the trained acoustic model in response to the inputted prescribed lyric data and the inputted pitch data, and on the basis of instrument sound waveform data that are synthesized in accordance with the pitch data corresponding to the user operation of the operation element, and

in the second mode;

in accordance with a user operation on an operation element in the plurality of operation elements, inputting prescribed lyric data and pitch data corresponding to the user operation of the operation element to the trained acoustic model so as to cause the trained acoustic model to output the acoustic feature data in response to the inputted prescribed lyric data and the inputted pitch data, and

digitally synthesizing and outputting inferred singing voice data that infers a singing voice of the singer on the basis of the acoustic feature data output by the trained acoustic model in response to the inputted prescribed lyric data and the inputted pitch data, without using instrument sound waveform data that are synthesized in accordance with the pitch data corresponding to the user operation of the operation element.