Machine training for native language and fluency identification

• US 10,431,203 B2
• Filed: 09/05/2017
• Issued: 10/01/2019
• Est. Priority Date: 09/05/2017
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Millen, David, Real Coelho, Livy Maria
• Primary Examiner(s): Sirjani, Fariba
• Application Number:
  US15/695,209
• Publication Number:
  US 20190073996A1
• Time in Patent Office: 756 Days
• Field of Search: None
• CPC Class Codes: G06F 40/216 using statistical methods
• CPC Class Codes: G06F 40/242 Dictionaries
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/04 Architecture, e.g. interconnection topology
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/16 using artificial neural networks
• CPC Class Codes: G10L 15/1822 Parsing for meaning understanding
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 25/27 characterised by the analysis technique
• CPC Class Codes: G10L 25/51 for comparison or discrimination

Abstract

Training a machine by a machine learning technique for recognizing speech utterance to determine language fluency level of a user. Native speaker recorded data and language specific dictionary of heteronyms may be retrieved. The native speaker recorded data may be parsed and the heteronyms from the native speaker recorded data may be isolated. Linguistic features from the native speaker recorded data including at least linguistic features associated with the heteronyms may be extracted, and a language dependent machine learning model is generated based on the linguistic features.

Claims

1. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause a device to:

train a machine by a machine learning technique for recognizing speech utterance to determine language fluency level of a user,

the training comprising at least;

receiving native speaker recorded data from a database of recorded speech of at least one native speaker,

receiving a language specific dictionary of heteronyms,

parsing the native speaker recorded data and isolating the heteronyms from the native speaker recorded data,

extracting linguistic features from the native speaker recorded data including at least linguistic features associated with the heteronyms, the linguistic features associated with the heteronyms including at least phonetics, and

generating a language dependent machine learning model based at least on the linguistic features, wherein the language dependent machine learning model is trained to output a score indicating language fluency;

generate a test corpus of sentences, wherein each sentence in the test corpus includes at least one pair of heteronyms, wherein heteronyms are words spelled identically but having different pronunciations and meanings from one another;

cause presenting of a sentence from the test corpus to the user on a user interface display;

receive a test speech utterance of the user uttering the presented sentence;

execute the language dependent machine learning model operating on the test speech utterance to obtain user pronunciation of the presented sentence including the at least two heteronyms;

evaluate a language fluency level of the user based on the obtained user pronunciation; and

output a score representing the language fluency level of the user.

2. The computer program product of claim 1, wherein the linguistic features comprise phoneme duration, intonation, timing, and loudness.

3. The computer program product of claim 1, wherein the language dependent machine learning model comprises a deep learning model.

4. The computer program product of claim 1, wherein the language dependent machine learning model comprises a naï

ve Bayes model.

5. The computer program product of claim 1, wherein the language dependent machine learning model comprises a random forest model.

6. The computer program product of claim 1, wherein the device is further caused to automatically retrain the language dependent machine learning model based on detecting a new set of heteronyms.

7. The computer program product of claim 1, wherein the receiving of the native speaker recorded data from a database of recorded speech of the native speaker comprises at least retrieving data from call center recordings.

8. A system of training a machine that recognizes native speech utterance, comprising:

a hardware processor;

a storage device communicatively coupled to the hardware processor and storing native speaker recorded data;

the hardware processor executing a machine learning technique to train the hardware processor to recognize speech utterance to determine language fluency level of a user, the training comprising the hardware processor;

receiving native speaker recorded data from the storage device;

receiving language specific dictionary of heteronyms;

parsing the native speaker recorded data and identifying the heteronyms from the native speaker recorded data;

extracting linguistic features from the native speaker recorded data including at least linguistic features associated with the heteronyms, the linguistic features associated with the heteronyms including at least phonetics; and

generating a language dependent machine learning model based on the linguistic features,

wherein the language dependent machine learning model is trained to output a score indicating language fluency;

the hardware processor further performing;

generating a test corpus of sentences, wherein each sentence in the test corpus includes at least one pair of heteronyms, and wherein heteronyms are words that are spelled identically but having different pronunciations and meanings from one another;

causing presenting of a sentence from the test corpus to the user;

receiving a test speech utterance of the user uttering the presented sentence;

executing the language dependent machine learning model operating on the test speech utterance to obtain user pronunciation of the presented sentence including the at least two heteronyms;

evaluating a language fluency level of the user based on the obtained user pronunciation; and

outputting a score representing the language fluency level of the user.

9. The system of claim 8, further comprising a user interface display coupled to the hardware processor, wherein the sentences from the test corpus of words are presented to the user via the user interface display for the user to utter.

10. The system of claim 8, wherein the linguistic features comprise phoneme duration, intonation, timing, and loudness.

11. The system of claim 8, wherein the language dependent machine learning model comprises a deep learning model.

12. The system of claim 8, wherein the language dependent machine learning model comprises a nave Bayes model.

13. The system of claim 8, wherein the language dependent machine learning model comprises a random forest model.

14. The system of claim 8, wherein the hardware processor automatically retraining the language dependent machine learning model based on automatically detecting a new set of heteronyms.

15. The computer system of claim 8, wherein the hardware processor receiving the native speaker recorded data from the storage device comprises at least retrieving data from call center recordings.

16. A computer program product comprising a computer readable storage medium having program instructions embodied therewith, the program instructions executable by a device to cause the device to:

generate a test corpus of sentences in a single dialect of a single language, each sentence in the corpus including at least two heteronyms spelled identically but having different pronunciations and meanings from one another;

cause displaying of a sentence from the test corpus to a user;

receive, at a language dependent machine learning model, data representing a test speech utterance of the user uttering the displayed sentence;

execute the language dependent machine learning model operating on the data representing the test speech utterance to obtain user pronunciation of the displayed sentence including the at least two heteronyms,

wherein the language dependent machine learning model is trained using at least linguistic features extracted from native speaker recorded data uttering the heteronyms present in the sentences of the test corpus, the linguistic features including at least phonetics associated with heteronyms, wherein the language dependent machine learning model is trained to output a score indicating a language fluency level of a user by evaluating user pronunciation of the at least two heteronyms in at least one of the sentences of the test corpus based on feature parameters associated with said at least one sentence and indicating different pronunciations of the two heteronyms in the sentence; and

output a score representing the language fluency level of the user based on the data representing the test speech utterance.