Method and apparatus of synthesizing plucked string instruments using recurrent neural networks

• US 6,292,791 B1
• Filed: 06/16/1998
• Issued: 09/18/2001
• Est. Priority Date: 02/27/1998
• Status: Expired due to Fees

• Current Assignee: Transpacific IP Group Limited
• Original Assignee: Industrial Technology Research Institute
• Inventors: Chung, Tien-Ho, Liang, Sheng-fu, Su, Wen-yu
• Primary Examiner(s): Powell, Mark
• Assistant Examiner(s): STARKS, WILBERT L
• Application Number:
  US09/097,739
• Time in Patent Office: 1,190 Days
• Field of Search: 846/01, 846/03, 846/04, 706/23
• US Class Current: 706/23
• CPC Class Codes: G06N 3/02 Neural networks
• CPC Class Codes: G10H 2250/311 Neural networks for electrophonic musical instruments or musical processing, e.g. for musical recognition or control, automatic composition or improvisation
• CPC Class Codes: G10H 7/00 Instruments in which the tones are synthesised from a data store, e.g. computer organs

Abstract

A “virtual string” is generated for synthesizing sound produced by plucked-string instruments using recurrent neural networks. The disclosed recurrent neural network, called a Scattering Recurrent Network (SRN), is based on the physics of waves traveling in the string. Vibration measured from a plucked string is used as the training data for the SRN. The trained SRN is a virtual model capable of generating tones similar to the tones generated by the physical string. As with a real string, the “virtual string” corresponding to the SRN responds differently to different types of string “plucking” motions.

Claims

1. A method of synthesizing sounds produced by a plucked string comprising the steps of:

generating an input waveform corresponding to an initial plucking of the string, wherein waveform values are assigned to a plurality of predetermined points on the string and interpolating further waveform values based on the assigned values;

initializing nodes of a neural network with values based on the input waveform;

iteratively changing the node values based on weights associated with the nodes;

outputting a waveform based on selected ones of the node values at a plurality of the iterations; and

generating sound based on a sequence of the output waveforms, the generated sound simulating the sound made by the plucked string.

2. The method of claim 1, wherein the step of generating the input waveform further includes the sub-step of assigning waveform values to a plurality of predetermined points on the string based on a mathematical function.

3. The method of claim 1, wherein the step of initializing the neural network further includes the sub-step of setting displacement nodes of the neural network to values of the input waveform.

4. The method of claim 1, wherein the neural network is a recurrent neural network.

5. The method of claim 1, wherein the step of iteratively changing the node values further includes, for each iteration, the sub-steps of:

calculating values of arrival nodes in the neural network based on values of departure nodes in the neural network at a previous iteration;

calculating values of displacement nodes in the neural network based on the calculated values of the arrival nodes; and

calculating values of the departure nodes in the neural network based on the calculated values of the displacement nodes and arrival nodes.

6. The method of claim 5, wherein the sub-step of calculating the values of the arrival nodes in the neural network further includes the step of evaluating the equation:

$\{\begin{array}{c}{\mathrm{\varphi <br><br>}}_{i,i-1}<br><br>\left(t+1\right)=a<br><br>\left[{\mathrm{net}}_{i,i-1}^{\mathrm{\varphi <br><br>}}<br><br>\left(t\right)\right]=a<br><br>\left[w_{i,i-1}·<br><br>f_{i,i-1}<br><br>\left(t\right)\right]\\ {\mathrm{\varphi <br><br>}}_{i,i+1}<br><br>\left(t+1\right)=a<br><br>\left[{\mathrm{net}}_{i,i+1}^{\mathrm{\varphi <br><br>}}<br><br>\left(t\right)\right]=a<br><br>\left[w_{i,i+1}·<br><br>f_{i,i+1}<br><br>\left(t\right)\right]\end{array}.$

7. The method of claim 5, wherein the sub-step of calculating the values of the displacement nodes in the neural network further includes the step of evaluating the equations:

$y_i<br><br>\left(t+1\right)=\{\begin{array}{cc}a<br><br>\left[{\mathrm{net}}_i^y<br><br>\left(t+1\right)\right],& i=2,\mathrm{\dots <br><br>}<br><br>\text{\hspace{1em}<br><br>},N-1\\ 0,& i=1<br><br>\text{\hspace{1em}<br><br>}<br><br>\mathrm{or}<br><br>\text{\hspace{1em}<br><br>}<br><br>i=N\end{array}$ 

net_i^y(t+1)=r<sub>i,i−

1</sub>·

φ

<sub>i,i−

1</sub>(t+1)+r_i,i+1·

φ

_i,i+1(t+1).

8. The method of claim 5, wherein the sub-step of calculating the values of the departure nodes in the neural network further includes the step of evaluating the equation:

$\{\begin{array}{c}{f}_{i+1,i}<br><br>\left(t+1\right)=a<br><br>\left[{\mathrm{net}}_{i+1,i}^f<br><br>\left(t+1\right)\right]=a<br><br>\left[y_i<br><br>\left(t+1\right)-{\mathrm{\varphi <br><br>}}_{i,i+1}<br><br>\left(t+1\right)\right]\\ f_{i-1,i}<br><br>\left(t+1\right)=a<br><br>\left[{\mathrm{net}}_{i-1,i}^f<br><br>\left(t+1\right)\right]=a<br><br>\left[y_i<br><br>\left(t+1\right)-{\mathrm{\varphi <br><br>}}_{i,i-1}<br><br>\left(t+1\right)\right]\end{array}.$

9. A computer readable medium containing instructions for execution on a computer that synthesizes sounds produced by a plucked string, the instructions, when executed, causing the computer to perform the steps of:

generating an input waveform corresponding to an initial plucking of the string, wherein waveform values are assigned to a plurality of predetermined points on the string and interpolating further waveform values based on the assigned values;

initializing nodes of a neural network with values based on the input waveform;

iteratively changing the node values based on weights associated with the nodes;

outputting a waveform based on selected ones of the node values at a plurality of the iterations; and

generating sound based on a sequence of the output waveforms, the generated sound simulating the sound made by the plucked string.