Operator interactions for developing phoneme recognition by neural networks

• US 5,867,816 A
• Filed: 02/28/1997
• Issued: 02/02/1999
• Est. Priority Date: 04/24/1995
• Status: Expired due to Fees

• Current Assignee: Ericsson Messaging Systems Incorporated (Telefonaktiebolaget LM Ericsson)
• Original Assignee: Ericsson Messaging Systems Incorporated (Telefonaktiebolaget LM Ericsson)
• Inventors: Nussbaum, Paul A.
• Primary Examiner(s): Knepper, David D.
• Assistant Examiner(s): SAX, ROBERT L
• Application Number:
  US08/807,778
• Time in Patent Office: 704 Days
• Field of Search: 704/232, 704/231, 704/259
• US Class Current: 704/232
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 2015/0638 Interactive procedures

Abstract

An automated speech recognition system converts a speech signal into a compact, coded representation that correlates to a speech phoneme set. A number of different neural network pattern matching schemes may be used to perform the necessary speech coding. An integrated user interface guides a user unfamiliar with the details of speech recognition or neural networks to quickly develop and test a neural network for phoneme recognition. To train the neural network, digitized voice data containing known phonemes that the user wants the neural network to ultimately recognize are processed by the integrated user interface. The digitized speech is segmented into phonemes with each segment being labelled with a corresponding phoneme code. Based on a user selected transformation method and transformation parameters, each segment is transformed into a series of multiple dimension vectors representative of the speech characteristics of that segment. These vectors are iteratively presented to a neural network to train/adapt that neural network to consistently distinguish and recognize these vectors and assign an appropriate phoneme code to each vector. Simultaneous display of the digitized speech, segments, vector sets, and a representation of the trained neural network assist the user in visually confirming the acceptability of the phoneme training set. A user may also selectively audibly confirm the acceptability of the digitization scheme, the segments, and the transform vectors so that satisfactory training data are presented to the neural network. If the user finds a particular step or parameter produces an unacceptable result, the user may modify one or more of the parameters and verify whether the modification effected an improvement in performance. The trained neural network is also automatically tested by presenting a test speech signal to the integrated user interface and observing both audibly and visually automatic segmentation of the speech, transformation into multidimensional vectors, and the resulting neural network assigned phoneme codes. A method of decoding such phoneme codes using the neural network is also disclosed.

Claims

1. A method, comprising the steps of:

digitizing a set of speech signals,

dividing the digitized signals into individual segments, each segment representing one or more speech phonemes;

transforming each segment into one or more vectors having plural dimensions;

assigning a code to each of the vectors, each code corresponding to a phoneme;

developing a speech recognition system including training a neural network using the coded vectors; and

an operator modifying the transforming step based on the acceptability of the trained neural network.

2. A method according to claim 1, further comprising the steps of:

digitizing another set of speech signals;

dividing the digitized signals into individual segments;

transforming each segment into one or more vectors having plural dimensions; and

using the trained neural network to automatically recognize phonemes in the other set of speech signals and assign a code to each of the vectors, each code corresponding to a phoneme recognized by the neural network.

3. A method according to claim 1, further comprising:

audibly evaluating the developed speech recognition system.

4. A method according to claim 1, further comprising:

visually evaluating the developed speech recognition system.

5. A method according to claim 1, further comprising:

audibly and visually evaluating the developed speech recognition system.

6. A method according to claim 1, further comprising:

displaying the segments, vectors, and assigned codes along with an indicator of an internal state of the trained neural network on the display screen.

7. The method of claim 6, wherein the internal state of the trained neural network includes an estimated centroid vector corresponding to an exemplary, multiple-dimension vector representing a particular phoneme.

8. The method according to claim 1, further comprising:

audibly reproducing the digitized signals.

9. The method according to claim 8, further comprising:

modifying digitization parameters based on the acceptability of the audibly reproduced signals.

10. The method according to claim 1, further comprising:

audibly reproducing one or more of the segments.

12. The method according to claim 1, further comprising:

reverse transforming the one or more vectors and audibly reproducing the reverse transformed vectors.

13. The method according to claim 12, further comprising:

modifying the transforming step based on the acceptability of the audibly reproduced signals.

14. The method according to claim 1, further comprising:

decoding an automatically assigned code to generate an estimate of a vector representing a corresponding phoneme.

15. The method according to claim 14, wherein the decoding step is based on an internal structure of the neural network.

16. The method of claim 15, wherein the internal structure corresponds to a centroid vector.

17. The method according to claim 14, further comprising:

reverse transforming the estimate vector into a time varying signal, and

audibly reproducing the time varying signal to evaluate the performance of the speech recognition system.

18. The method according to claim 14, wherein the estimate vector is a centroid of plural vectors having plural dimensions corresponding to the code generated while training the neural network.

19. The method according to claim 16, further comprising:

reverse transforming the centroid vector and audibly determining if the centroid vector sounds like the phoneme the one centroid vector is to represent.

20. The method according to claim 19, further comprising:

modifying the coding step based on the acceptability of the audibly determining step.

21. The method according to claim 1, wherein the dividing step includes automatically dividing the digitized audio signals into segments, each segment corresponding to one or more phonemes.

22. The method according to claim 16, further comprising:

displaying corresponding digitized audio signals, segments, vectors, and centroids on a display screen.

23. The method according to claim 1, further comprising:

visually representing each vector as having multiple dimensions on a two-dimensional display screen.

24. The method according to claim 1, wherein the dividing step includes automatically dividing the digitized signals into segments, where each segment may have a different number of vectors, and further comprising the step of:

presenting the vectors to the neural network being developed and generating an output, and

assigning a most popular code to the segment based on the output of the neural network.

25. The method according to claim 1, wherein the coding step includes:

assigning more than one code to a segment of vectors, and

using the neural network being trained to determine automatically a code for the segment of vectors.

26. The method according to claim 1, wherein the transforming step includes transforming segments into the frequency domain using a Fourier transform.

27. The method according to claim 1, wherein the transforming step includes transforming segments using a linear predictive coding technique.

28. The method according to claim 1, further comprising:

displaying vectors resulting from the transforming step, and

confirming that the vectors corresponding to a segment vary smoothly over time.

11. A method, comprising the steps of:

digitizing a set of speech signals,

dividing the digitized signals into individual segments, each segment representing one or more speech phonemes;

transforming each segment into one or more vectors having plural dimensions;

assigning a code to each of the vectors, each code corresponding to a phoneme; and

developing a speech recognition system including training a neural network using the coded vectors,

wherein the dividing step is performed either manually under the control of an operator or using an automatic segmenting procedure and the operator modifies the segments either by manually changing boundaries defining ones of the segments or by changing automatic segmenting procedure parameters.

29. A speech signal processing system, comprising:

an analog to digital converter for converting speech signals to be analyzed to digital form;

a data processor connected to the analog to digital converter and performing the following data processing tasks;

analyzing the digitized signals as speech segments, each segment representing one or more speech phonemes;

transforming each segment into one or more vectors;

assigning a phoneme to each of the vectors;

an operator selecting a neural network from a plurality of different types of neural networks; and

training the selected neural network using the vectors,

wherein the trained neural network is used to recognize speech.

30. A speech signal processing system according to claim 29, further comprising:

a speaker,

wherein the trained neural network is audibly evaluated by a human operator using one or more speech representative signals generated by the trained neural network being audibly reproduced using the speaker.

31. A speech signal processing system according to claim 30, wherein the trained neural network is modified based on the audible evaluation.

32. A speech signal processing system according to claim 29, further comprising:

a display,

wherein the trained neural network is visibly evaluated by a human operator with one or more speech representative signals generated by the trained neural network being displayed on the display.

33. A speech signal processing system according to claim 32, wherein the trained neural network is modified based on the visible evaluation.

34. A speech signal processing system according to claim 32, wherein the speech segments and corresponding vectors are displayed on the display along with an indicator of the internal state of the trained neural network.

35. A speech signal processing system according to claim 33, wherein the internal state of the trained neural network is a centroid or an estimated centroid corresponding to an exemplary multiple dimension vector representing a particular speech phoneme.

36. A speech signal processing system according to claim 29, wherein the different types of neural networks include two or more of UCL, SCL, auto-associative, and back propagation neural networks.