SYSTEM AND METHOD FOR CROWDSOURCING OF WORD PRONUNCIATION VERIFICATION

• US 20150095031A1
• Filed: 09/30/2013
• Published: 04/02/2015
• Est. Priority Date: 09/30/2013
• Status: Abandoned Application

• Current Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Original Assignee: AT&T Intellectual Property I LP (AT&T, Inc.)
• Inventors: CONKIE, Alistair D., MISHRA, Taniya, GOLIPOUR, Ladan
• Application Number:
  US14/041,768
• Publication Number:
  US 20150095031A1
• Time in Patent Office: Days
• US Class Current: 704/254
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams

Abstract

Disclosed herein are systems, methods, and computer-readable storage media for crowdsourcing verification of word pronunciations. A system performing word pronunciation crowdsourcing identifies spoken words, or word pronunciations in a dictionary of words, for review by a turker. The identified words are assigned to one or more turkers for review. Assigned turkers listen to the word pronunciations, providing feedback on the correctness/incorrectness of the machine made pronunciation. The feedback can then be used to modify the lexicon, or can be stored for use in configuring future lexicons.

Claims

1. A method comprising:

identifying a spoken word in a dictionary of words for review;

assigning a plurality of turkers to review the spoken word;

receiving, from the plurality of turkers, a plurality of word scores, wherein each word score in the plurality of word scores represents an evaluation of a pronunciation of the spoken word by a respective turker in the plurality of turkers;

determining an average word score based on the plurality of word scores;

comparing the average word score to a required score, to yield a comparison; and

when the comparison indicates the pronunciation of the spoken word is incorrect;

assigning the spoken word to an expert turker for review, to yield expert feedback; and

assigning turker performance scores to each respective turker in the plurality of turkers based on the word score the each respective turker provided, the comparison, and the expert feedback.

2. The method of claim 1, further comprising, after assigning the turker performance scores, assigning additional turkers to review a second spoken word, wherein the assigning of the additional turkers is based on the turker performance scores.

3. The method of claim 2, further comprising modifying a grapheme-to-phoneme pronunciation model used to generate the dictionary of words based on the average score, the comparison, and the expert feedback.

4. The method of claim 1, wherein the plurality of turkers have an expertise in one of an accent and a subject matter.

5. The method of claim 1, wherein the dictionary of words is generated using a grapheme-to-phoneme model.

6. The method of claim 5, further comprising modifying the grapheme-to-phoneme model based on the average word score.

7. The method of claim 1, wherein the average word score is calculated using the plurality of word scores and a weight associated with a reliability of each respective turker in the plurality of turkers.

8. A system, comprising:

a processor; and

a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising;

identifying a spoken word in a dictionary of words for review;

assigning a plurality of turkers to review the spoken word;

receiving, from the plurality of turkers, a plurality of word scores, wherein each word score in the plurality of word scores represents an evaluation of a pronunciation of the spoken word by a respective turker in the plurality of turkers;

determining an average word score based on the plurality of word scores;

comparing the average word score to a required score, to yield a comparison;

when the comparison indicates the pronunciation of the spoken word is incorrect;

assigning the spoken word to an expert turker for review, to yield expert feedback; and

assigning turker performance scores to each respective turker in the plurality of turkers based on the word score the each respective turker provided, the comparison, and the expert feedback.

9. The system of claim 8, the computer-readable storage medium having additional instructions which result in the operations further comprising, after assigning the turker performance scores, assigning additional turkers to review a second spoken word, wherein the assigning of the additional turkers is based on the turker performance scores.

10. The system of claim 9, the computer-readable storage medium having additional instructions which result in the operations further comprising modifying a grapheme-to-phoneme pronunciation model used to generate the dictionary of words based on the average score, the comparison, and the expert feedback.

11. The system of claim 8, wherein the plurality of turkers have an expertise in one of an accent and a subject matter.

12. The system of claim 8, wherein the dictionary of words is generated using a grapheme-to-phoneme model.

13. The system of claim 12, the computer-readable storage medium having additional instructions stored which result in the operations further comprising modifying the grapheme-to-phoneme model based on the average word score.

14. The system of claim 8, wherein the average word score is calculated using the plurality of word scores and a weight associated with a reliability of each respective turker in the plurality of turkers.

15. A computer-readable storage device having instructions stored which, when executed by the processor, cause a computing device to perform operations comprising:

identifying a spoken word in a dictionary of words for review;

assigning a plurality of turkers to review the spoken word;

receiving, from the plurality of turkers, a plurality of word scores, wherein each word score in the plurality of word scores represents an evaluation of a pronunciation of the spoken word by a respective turker in the plurality of turkers;

determining an average word score based on the plurality of word scores;

comparing the average word score to a required score, to yield a comparison;

when the comparison indicates the pronunciation of the spoken word is incorrect;

assigning the spoken word to an expert turker for review, to yield expert feedback; and

assigning turker performance scores to each respective turker in the plurality of turkers based on the word score the each respective turker provided, the comparison, and the expert feedback.

16. The computer-readable storage device of claim 15, the computer-readable storage device having additional instructions which result in the operations further comprising, after assigning the turker performance scores, assigning additional turkers to review a second spoken word, wherein the assigning of the additional turkers is based on the turker performance scores.

17. The computer-readable storage device of claim 16, the computer-readable storage device having additional instructions which result in the operations further comprising modifying a grapheme-to-phoneme pronunciation model used to generate the dictionary of words based on the average score, the comparison, and the expert feedback.

18. The computer-readable storage device of claim 15, wherein the plurality of turkers have an expertise in one of an accent and a subject matter.

19. The computer-readable storage device of claim 15, wherein the dictionary of words is generated using a grapheme-to-phoneme model.

20. The computer-readable storage device of claim 19, the computer-readable storage medium having additional instructions stored which result in the operations further comprising modifying the grapheme-to-phoneme model based on the average word score.