Method of setting optimum-partitioned classified neural network and method and apparatus for automatic labeling using optimum-partitioned classified neural network

• US 7,444,282 B2
• Filed: 03/01/2004
• Issued: 10/28/2008
• Est. Priority Date: 02/28/2003
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Choo, Ki-hyun, Lee, Jae-won, Kim, Jeong-su, Lee, Ki-seung
• Primary Examiner(s): Smits; Talivaldis Ivars
• Assistant Examiner(s): Kovacek; David
• Application Number:
  US10/788,301
• Publication Number:
  US 20040172238A1
• Time in Patent Office: 1,702 Days
• Field of Search: 704200-202, 704/211, 704213-218, 704231-240, 704243-245, 704258-277, 706 15- 20, 706/25, 706 45- 48
• US Class Current: 704/202
• CPC Class Codes: G10L 15/04 Segmentation; Word boundary detection

Abstract

A method of automatic labeling using an optimum-partitioned classified neural network includes searching for neural networks having minimum errors with respect to a number of L phoneme combinations from a number of K neural network combinations generated at an initial stage or updated, updating weights during learning of the K neural networks by K phoneme combination groups searched with the same neural networks, and composing an optimum-partitioned classified neural network combination using the K neural networks of which a total error sum has converged; and tuning a phoneme boundary of a first label file by using the phoneme combination group classification result and the optimum-partitioned classified neural network combination, and generating a final label file reflecting the tuning result.

Claims

1. A method of automatic labeling to tune a phoneme boundary of a first label file generated by performing automatic labeling of a manual label file, the method comprising:

searching for neural networks having minimum errors with respect to a number of L phoneme combinations from a number of K neural network combinations generated at an initial stage or updated;

updating weights during learning of the K neural networks by K phoneme combination groups searched with the same neural networks;

composing an optimum-partitioned classified neural network combination using the K neural networks of which a total error sum has converged;

tuning a phoneme boundary of a first label file using a phoneme combination group classification result and the optimum-partitioned classified neural network combination from the composing of the optimum-partitioned classified neural network combination; and

generating a final label file reflecting the tuning result,

wherein a phoneme boundary tuning field in the tuning of the phoneme boundary or the first label file and the generating a final label file reflecting the tuning result is set to a predetermined field of a duration time of left and right phonemes of the phoneme combination.

2. The method of claim 1, further comprising setting an output value of the neural network to 1 for the part applicable to a boundary between phonemes, setting the output value for the part not applicable to a boundary between phonemes to 0, and setting the output value for the part of 1 frame left or right apart from a phoneme boundary to 0.5.

3. The method of claim 1, further comprising setting the predetermined field to a length which divides the duration time of the left and right phonemes into three equal parts and segments one-third each to the left and right near each phonemic boundary of the first label file.

4. The method of claim 1, further comprising using a computer readable medium having recorded thereon a computer readable program code to tune the phoneme boundary of the first label file generated by performing automatic labeling of the manual label file.

5. An apparatus for automatic labeling using an optimum-partitioned classified neural network, comprising:

a labeling unit to generate a first label file by performing automatic labeling for a manual label file;

an optimum-partitioned classified neural network composing unit searching neural networks having minimum errors with respect to a number of L phoneme combinations from a number of K neural network combinations generated at an initial stage or updated, updating weights during learning of the K neural networks by K phoneme combination groups searched with the same neural networks, and composing an optimum-partitioned classified neural network combination using the K neural networks of which a total error sum has converged; and

a phoneme boundary tuning unit tuning a phoneme boundary of the first label file by using a phoneme combination group classification result and the optimum-partitioned classified neural network combination supplied from the optimum-partitioned classified neural network composing unit, and generating a final label file reflecting the tuning result,

wherein the phoneme boundary tuning field of the phoneme boundary tuning unit is set to a predetermined field of a duration time of left and right phonemes of the phoneme combination.

6. The apparatus of claim 5, wherein the optimum-partitioned classified neural network composing unit comprises:

a training corpus storing input variables, wherein the input variables include acoustic feature variables, additional variables, and a manual label file;

a minimum error classifying unit generating L phoneme combinations realized with names of left and right phonemes by using a phoneme boundary obtained from input variables and a manual label file stored in the training corpus, searching a neural network having minimum errors with respect to the L phoneme combinations from K neural network combinations generated or updated at an initial time, and classifying the L phoneme combinations into K phoneme combination groups searched with the same neural networks; and

a re-training unit updating weights until individual errors of the neural networks have converged during learning with applicable learning data for the K neural networks by the K phoneme combination groups classified in the minimum error classifying unit and re-training the neural networks until a total error of the K neural networks, of which individual errors have converged, has converged.

7. The apparatus of claim 5, further comprising setting an output value of the neural network to 1 for the part applicable to a boundary between phonemes, setting the output value for the part not applicable to a boundary between phonemes to 0, and setting the output value for the part of 1 frame left or right apart from a phoneme boundary to 0.5.

8. The apparatus of claim 5, wherein the predetermined field is set to a length which divides the duration time of the left and right phonemes into three equal parts and segments one third each to the left and right near each phonemic boundary of the first label file.

9. An apparatus for automatic labeling using an optimum-partitioned classified neural network, comprising:

a labeling unit to perform automatic labeling of a manual label file and generate a first label file;

an optimum-partitioned classified neural network composing unit to receive input variables, segment phoneme combinations into partitions applicable to neural networks, and compose optimum-partitioned classified neural networks of Multi-Layer Perceptron-type from re-learned partitions; and

a phoneme boundary tuning unit to tune a phoneme boundary of the first label file supplied from the labeling unit and to generate a final label file reflecting the tuning result,

wherein the phoneme boundary tuning unit tunes the phoneme boundary by using the optimum-partitioned classified neural networks composed after completing learning in the optimum-partitioned classified neural network composing unit and judges the phoneme boundary according to whether an output of a neural network is 1 or 0 after applying the same input variable as the input variable used during the learning.

10. The apparatus of claim 9, wherein if there is a nonlinear clustering of neural networks of a Multi-Layer Perceptron-type, weights are determined by an iterative modification, wherein the iterative modification is performed by a back-propagation algorithm.

11. The apparatus of claim 9, wherein the optimum-partitioned classified neural network composing unit comprises:

a training corpus to store input variables, wherein the input variables comprise acoustic feature variables, additional variables, and a manual label file;

a minimum error classifying unit to generate phoneme combinations of names of left and right phonemes by using a phoneme boundary obtained from the input variables and manual label file stored in the training corpus, search an optimum neural network having a minimum error level relating to the phoneme combinations from neural network combinations of a Multi-Layer Perceptron-type, and classify the phoneme combinations into phoneme combination groups with the same neural networks; and

a re-training unit to update weights of the neural networks by learning with applicable learning data for the neural networks as many as the predetermined number of iterations with respect to each of the phoneme combination groups classified in the minimum error classifying unit and converge a total error by adapting the updated weights to an applicable neural network in the neural network combinations.

12. A method of composing the optimum neural network combination minimizing a total error sum, comprising:

preparing an initial neural network combination using input variables;

searching for the optimum neural network having minimum errors with respect to the phoneme combinations from the initial neural network combination and classifying phoneme combinations with optimum neural networks;

merging phoneme combinations with the same neural network and classifying the merged phoneme combinations into new partitions;

updating each neural network and learning each neural network according to the partitions generated by the classifying and the merging;

determining whether all neural networks are converged, wherein if all neural networks are converged, composing the neural network combination as an optimum-partitioned classified neural network combination, and if all neural networks are not converged, then re-learning the optimum neural network having minimum errors by repeating the classifying, the merging, and the updating operations until all neural networks are converged.

13. The method of claim 12, wherein the preparing the initial neural network combination further comprises setting learning data for neural network learning and setting a first and a second threshold value.

14. The method of claim 13, further comprising repeatedly performing the updating of each neural network via a procedure that calculates the error using an updated neural network parameter, wherein the updating ends when all neural networks are converged.

15. The method of claim 14, wherein all neural networks are converged when an error-changing rate is less than the first threshold value.

16. The method of claim 12, wherein the setting learning data for neural network learning and the preparing the initial neural network combination further comprises setting an iteration times index to 0, setting an initial error sum to infinity, and preparing a position value of the phoneme boundary obtained by manual labeling.

17. The method of claim 12, further comprising calculating the total error for the classified phoneme combinations with optimum neural networks, wherein the total error is a sum of square errors between a target output and an output obtained when inputting all learning data of the phoneme combination to the neural network.

18. The method of claim 12, further comprising calculating the total error for the new partition.

19. The method of claim 12, further comprising calculating a weight update value to update each neural network.

20. The method of claim 12, further comprising updating the neural networks by setting the learning gain with a small value and setting the first threshold value for convergence investigation with a comparatively small value.

21. The method of claim 12, further comprising setting an output value of the neural network to 1 for the part applicable to a boundary between phonemes, setting the output value for the part not applicable to a boundary between phonemes to 0, and setting the output value for the part of 1 frame left or right apart from a phoneme boundary to 0.5.

22. The method of claim 12, further comprising assigning all input variables an applicable phoneme combination by seeking a nearest phoneme boundary from positions of input variables and deciding which name of two phonemes is connected to the boundary.

23. The method of claim 12, further comprising segmenting all input variables such that the number of total partitions is the square of the number of used phonemes.

24. The method of claim 23, further comprising setting the number of individual neural networks to the same value as or less than the number of partitions by a phoneme combination.

25. The method of claim 12, further comprising classifying the optimum neural network having minimum errors using the following equation:

$c_i<br><br>\left(P_j\right)=\arg <br><br><br><br>{\min }_k<br><br>\left[\underset{P_c<br><br>\left(m\right)=j}{\sum <br><br>}<br><br><br><br>\underset{n\in <br><br>W_m}{\sum <br><br>}<br><br><br><br>{<br><br>y^d<br><br>\left(n\right)-F<br><br>\left({\mathrm{\Phi <br><br>}}_k^i,X<br><br>\left(n\right)\right)<br><br>}^2\right]$

wherein c_i(P_j) is an optimum neural network index in an i^th iteration for a j^th phoneme combination (P_j), W_m is a section in which input variables included in the m^th phoneme boundary are selected.

26. The method of claim 25, further comprising selecting the input variables included in the m^th phoneme boundary of the W_m section according to the following equation:

$W_m=[n<br><br><br><br>\frac{t_{m-1}+t_m}{2}\le <br><br>n\le <br><br>\frac{t_m+t_{m+1}}{2}]$

wherein t_m is a nearest frame index from a position of the m^th phoneme boundary.

27. The method of claim 25, wherein the total error of a k^th neural network is given as a sum of square errors between a target output and an output obtained in the case of inputting all learning data included in the phoneme combination to the k^th neural network.

28. The method of claim 12, further comprising determining a total error for the new partitions using the following equation:

$D_{i+1}<br><br>\left(S^i\right)=\underset{m=1}{\overset{M}{\sum <br><br>}}<br><br>\underset{P_j\in <br><br>s_k^i}{\sum <br><br>}<br><br>\min <br><br>\left[\underset{P_c<br><br>\left(m\right)=j}{\sum <br><br>}<br><br>\underset{n\in <br><br>W_m}{\sum <br><br>}<br><br>{<br><br>y^d<br><br>\left(n\right)-F<br><br>\left({\mathrm{\Phi <br><br>}}_k^i,X<br><br>\left(n\right)\right)<br><br>}^2\right]$

wherein i is an iteration times index and Sⁱ is a partition at an i^th iteration.

29. The method of claim 28, further comprising determining the Sⁱ partition at the i^th iteration using the following equation:

Sⁱ=s₁ⁱUs₂ⁱ. . . Us_kⁱ.

30. The method of claim 12, further comprising updating for individual neural networks and learning neural networks according to partitions generated by the classifying and the merging and determining weight update values using the following equation:

${\mathrm{\Delta <br><br>\Phi <br><br>}}_k^i=\mathrm{\eta <br><br>}<br><br>\underset{s_i^k}{\sum <br><br>}<br><br>\underset{P_c<br><br>\left(m\right)=j}{\sum <br><br>}<br><br>\underset{n\in <br><br>W_m}{\sum <br><br>}<br><br><br><br>y^d<br><br>\left(n\right)-F<br><br>\left({\mathrm{\Phi <br><br>}}_k^i,X<br><br>\left(n\right)\right)<br><br>{<br><br>}^2].$

31. The method of claim 12, further comprising determining whether all neural networks have converged by confirming whether there is a difference between a total error sum obtained from the current number of iterations and a total error sum obtained from the previous number of iteration using the following equation:

Δ

D=|D_i+1(C_i)−

D_i(C_i)|

wherein if a changing rate of a total error sum is smaller than the second threshold value, the learning is finished.

32. A method of learning and updating a neural network, comprising:

preparing initial neural network combinations composing neural networks of multi-layer perceptron-type;

searching a multi-layer perceptron index having a minimum error in the initial neural network combinations of all phoneme combinations;

classifying merged phoneme combinations into new partitions by merging phoneme combinations with the same multi-layer perceptron index if the multi-layer perceptron index having a minimum error for all phoneme combinations is searched; and

re-training neural networks to update weights by learning data applicable to each partition, wherein the re-training procedure of individual neural networks calculates errors using the updated weights and repeating the re-training until a changing rate of errors becomes smaller than a first threshold value.

33. The method of claim 32, further comprising repeating the searching, classifying, and re-training operations until the changing rate of a total error sum is smaller than a second threshold value when the total error sum obtained from the number of present iterations is compared to an error sum obtained from the number of previous iterations.

34. The method of claim 33, wherein the partition segmentation of the phoneme combination is performed without relation to linguistic knowledge.