System and method for transcription of spoken words using multilingual mismatched crowd unfamiliar with a spoken language

• US 10,269,353 B2
• Filed: 03/31/2017
• Issued: 04/23/2019
• Est. Priority Date: 08/30/2016
• Status: Active Grant

• Current Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Original Assignee: TATA Consultancy Services Limited (Tata Sons Pvt Ltd.)
• Inventors: Radadia, Purushotam Gopaldas, Kalra, Kanika, Kumar, Rahul, Sriraman, Anand, Sirigireddy, Gangadhara Reddy, Joshi, Shrikant, Karande, Shirish Subhash, Lodha, Sachin Premsukh
• Primary Examiner(s): Kazeminezhad, Farzad
• Application Number:
  US15/476,505
• Publication Number:
  US 20180061417A1
• Time in Patent Office: 753 Days
• CPC Class Codes: G06F 16/24552 Database cache management
• CPC Class Codes: G06F 3/0238 Programmable keyboards key guide holders G06F3/0224
• CPC Class Codes: G06F 40/47 Machine-assisted translation, e.g. using translation memory
• CPC Class Codes: G10L 15/005 Language recognition
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/26 Speech to text systems G10L15/08 takes precedence
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals

Abstract

The disclosure generally relates to transcription of spoken words, and more particularly to a system and method for transcription of spoken words using multilingual mismatched words. The process comprises collection of multi-scripted noisy transcriptions of the spoken word obtained from workers of the multilingual mismatched crowd unfamiliar with the spoken language. The collected words are mapped to a phoneme sequence in the source language using script specific graphemes to phoneme model. Further, it builds a multi-scripted transcription script specific, worker specific and a global insertion-deletion-substitution (IDS) channel. Furthermore, the disclosure also determines reputation of workers to allocate the transcription task. Determination of reputation is based on word belief. The word belief is determined by taking ratio of likelihood probability of mapped phoneme sequence of transcriptions given the current estimate of word to the sum of likelihood probabilities of mapped phoneme sequences of the transcriptions given the phoneme sequence of each dictionary word.

Claims

1. A computer implemented method for transcribing one or more spoken word utterances of a source language using a multilingual mismatched crowd unfamiliar with the source language, the method comprises:

collecting, at a word transcription table, a plurality of multi-scripted noisy transcriptions of the spoken word obtained from a plurality of workers of the multilingual mismatched crowd, wherein the word transcription table is configured to store transcription responses of the spoken word segments presented to the plurality of workers, audio chunk id, each of the plurality of workers id, and the plurality of workers transcription text;

mapping each of the collected plurality of multi-scripted transcriptions to a phoneme sequence in the source language using script specific graphemes to phoneme model;

building worker specific insertion-deletion-substitution (IDS) channel model, multi-scripted transcription script specific IDS channel model and a global IDS channel model from the multi-scripted transcriptions;

filtering out a set of workers of the plurality of workers based on the reputation of the workers, estimated by simulating IDS channel for worker specific on dictionary words using worker reputation module;

allocating the transcription tasks to the set of workers such that required number of transcriptions per word are minimized; and

decoding, at a transcription decoding module, the plurality of multi-scripted transcriptions are combined to decode the transcription in source script, wherein the decoding comprises steps of;

finding likelihood probability of the mapped phoneme sequences of the multi-scripted mismatched crowd transcriptions with each of the dictionary words phoneme sequence using insertion-deletion-substitution channel parameters and voting the dictionary word that maximizes above likelihood; and

determining word belief by taking ratio of the likelihood probability of the mapped phoneme sequences of transcriptions given current estimate of word to sum of the likelihood probabilities of mapped phoneme sequences of the transcriptions given the phoneme sequence of each dictionary word.

2. The method of claim 1, wherein the multi-scripted transcriptions of words, whose ground truth phoneme sequences are known in the source language, are used to train worker script'"'"'s grapheme to source language'"'"'s phoneme mapping models using expectation maximization algorithm.

3. The method of claim 2, further wherein with the help of ground truth phoneme sequences, the worker specific, the transcription script specific and the global IDS channel models are trained using the expectation maximization algorithm.

4. The method of claim 1, wherein the building of the worker specific IDS channel model using gold standard transcription text.

5. The method of claim 1, wherein the multilingual mismatched crowd includes the plurality of workers who are having different scripts for the same word, unfamiliar to the source language and transcribing the given words of the source language in their own language.

6. The method of claim 1, wherein the multi-script transcriptions of words whose ground truths are not known are first mapped to the phoneme sequence using the graphemes to phoneme (G2P) model, and with the help of phoneme sequences of the dictionary words the worker specific, the transcription script specific and the global IDS channel models are trained in iterative fashion using expectation maximization algorithm.

7. The method of claim 1, wherein estimation of worker reputation is based on transcribed words whose ground truth is known and task allocation follows the estimated worker reputation.

8. The method of claim 1, wherein the task allocation utilizes bipartite matching algorithm to allocate the tasks to worker such that average word belief is maximized.

9. The method of claim 1, wherein the likelihood probability of a mapped phoneme sequence of the plurality of worker'"'"'s transcription of the spoken word given the phoneme sequence of the dictionary word is obtained by aligning both phoneme sequences using linearly combined IDS channel parameters of worker, his/her script and global.

10. The method of claim 1, wherein the likelihood probabilities of each mapped phoneme sequences of the multi-scripted transcriptions of a word with the dictionary words phoneme sequences are obtained and multiplied so as to obtain the likelihood probability, and wherein, the dictionary word that provides maximum likelihood probability is considered as decoded word.

11. A system for transcribing one or more spoken word utterances of a source language using a multilingual mismatched crowd unfamiliar with the source language, the system comprising:

a processor;

a memory communicatively coupled to the processor and the memory contains instructions that are readable by the processor;

a database parted within the memory, wherein the database comprises an audio chunk table and a word transcription table, wherein the audio chunk table is configured to store one or more information of a plurality of workers, one or more spoken word segments of each of the plurality of workers, number of responses given by the each of the plurality of workers and transcription score of the each of the plurality of workers, and further wherein, the word transcription table is configured to store transcription responses of the spoken word segments presented to the plurality of workers, audio chunk id, each of the plurality of workers id, and the plurality of workers transcription text;

a plurality of typing interfaces are configured according to script preference of the each of the plurality of workers of the multilingual mismatched crowd;

a reputation module is configured to compute the worker reputation and filter out spammer from the plurality of workers;

a task allocation module is configured to compute word beliefs and to allocate the transcription tasks to the plurality of workers, wherein the reputation of the plurality of workers is estimated by simulating the worker specific insertion-deletion-substitution (IDS) channel on dictionary words; and

a transcription decoding module is configured to generate transcription in the source language from multi-scripted transcriptions of the plurality of workers.

12. The system of claim 11, wherein the transcription decoding module is configured to invoke a grapheme to phoneme model for mapping the multi-scripted transcriptions with a phoneme sequence of the source language.

13. The system of claim 11, wherein the transcription decoding module is configured to compute likelihood by aligning phoneme sequence of dictionary word with the phoneme sequence of the multi-scripted transcriptions of the plurality of workers.

14. The system of claim 11, wherein the transcription decoding module is configured to invoke insertion-deletion-substitution channel model for decoding the word transcription in the source language from the multi-scripted transcriptions.

15. The system of claim 11, wherein the task allocation module is to decide the word that needs more transcriptions from its belief probability.