System and method for robust access and entry to large structured data using voice form-filling

• US 8,924,212 B1
• Filed: 08/26/2005
• Issued: 12/30/2014
• Est. Priority Date: 08/26/2005
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: AT&T Intellectual Property II LP (AT&T, Inc.)
• Inventors: Parthasarathy, Sarangarajan, Allauzen, Cyril Georges Luc
• Primary Examiner(s): Desir, Pierre-Louis
• Assistant Examiner(s): KOVACEK, DAVID M
• Application Number:
  US11/213,458
• Time in Patent Office: 3,413 Days
• Field of Search: 704 9- 10, 704/231, 704236-257, 704E17001-E17008, 704E17013-E17014, 704E15014-E15024, 704/E15.038
• US Class Current: 704/251
• CPC Class Codes: G06F 40/174 Form filling; Merging
• CPC Class Codes: G10L 15/00 Speech recognition G10L17/00 takes precedence
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/06 Creation of reference templates; Training of speech recognition systems, e.g. adaptation to the characteristics of the speaker's voice G10L15/14 takes precedence
• CPC Class Codes: G10L 15/063 Training
• CPC Class Codes: G10L 15/187 Phonemic context, e.g. pronunciation rules, phonotactical constraints or phoneme n-grams
• CPC Class Codes: G10L 15/19 Grammatical context, e.g. disambiguation of the recognition hypotheses based on word sequence rules
• CPC Class Codes: G10L 15/197 Probabilistic grammars, e.g. word n-grams
• CPC Class Codes: G10L 2015/025 Phonemes, fenemes or fenones being the recognition units
• CPC Class Codes: G10L 2015/081 Search algorithms, e.g. Baum-Welch or Viterbi
• CPC Class Codes: G10L 2015/088 Word spotting

Abstract

A method, apparatus and machine-readable medium are provided. A phonotactic grammar is utilized to perform speech recognition on received speech and to generate a phoneme lattice. A document shortlist is generated based on using the phoneme lattice to query an index. A grammar is generated from the document shortlist. Data for each of at least one input field is identified based on the received speech and the generated grammar.

Claims

1. A method comprising:

performing, via a processor, a first pass of entries in a database to generate a shortlist of the entries, the first pass comprising;

utilizing a phonotactic grammar to perform speech recognition on received speech and to generate a phoneme lattice, wherein the utilizing ignores inter-field constraints of the phonotactic grammar;

removing silence and filler words from the phoneme lattice to yield a revised phoneme lattice;

normalizing costs in the revised phoneme lattice, to yield a normalized phoneme lattice with a best path having a cost of zero;

identifying N-best lowest cost indices of an index for the database based on the normalized phoneme lattice; and

generating the shortlist of the entries in the database based on the N-best lowest cost indices;

performing a second pass on the shortlist of the entries in the database to obtain a final result, the second pass comprising;

generating a grammar from the shortlist of the entries in the database to yield a generated grammar, having inter-field constraints; and

identifying data from the database for each of the input field based on the received speech, the generated grammar, and the inter-field constraints; and

populating an input field of a form using the final result.

2. The method of claim 1, wherein the index is previously generated based on a plurality of training phoneme lattices and factors of interest from valid entries in the database, the factors of interest comprising trigrams, and wherein the database is a name/address database.

3. The method of claim 2, wherein the factors of interest further comprise N-grams based on the valid entries in the database.

4. The method of claim 1, wherein the generated grammar comprises common variations of the input field.

5. The method of claim 1, wherein the index is divided into two sub-indices.

6. The method of claim 1, wherein the phonotactic grammar is an N-gram phonotactic grammar.

7. The method of claim 6, wherein the N-gram phonotactic grammar is unsmoothed, recognizing only N-grams which have been seen in data used to train the N-gram phonotactic grammar.

8. The method of claim 1, wherein the generated grammar is used to:

perform one of another speech recognition on the received speech, constrain recognition for each of the input field, and rescore a result of performing the speech recognition.

9. The method of claim 1, wherein generating the shortlist of the entries in the database further comprises:

utilizing only factors of interest from the index to generate a cost-normalized query based on the normalized lattice; and

selecting the N-best lowest indices from indices of the index containing the factors of interest and the cost-normalized query.

10. A system comprising:

a processor; and

a computer-readable storage medium having instructions stored which, when executed by the processor, cause the processor to perform operations comprising;

performing a first pass of entries in a database to generate a shortlist of the entries, the first pass comprising;

utilizing a phonotactic grammar to perform speech recognition on received speech and to generate a phoneme lattice, wherein the utilizing ignores inter-field constraints of the phonotactic grammar;

removing silence and filler words from the phoneme lattice to generate a revised phoneme lattice;

normalizing costs in the revised phoneme lattice, to yield a normalized phoneme lattice with a best path having a cost of zero;

identifying N-best lowest cost indices of an index for the database based on the normalized phoneme lattice; and

generating the shortlist of the entries in the database based on the N-best lowest cost indices;

perform a second pass on the shortlist of the entries in the database to obtain a final result, the second pass comprising;

generating a grammar from the shortlist of the entries in the database to yield a generated grammar, having inter-field constraints; and

identifying data from the database for each of the input field based on the received speech, the generated grammar, and the inter-field constraints; and

populating an input field of a form using the final result.

11. The system of claim 10, wherein the index is previously generated based on a plurality of training phoneme lattices and factors of interest from valid entries in the database, the factors of interest comprising trigrams, and wherein the database is a name/address database.

12. The system of claim 11, wherein the factors of interest comprise N-grams based on the valid entries in the database.

13. The system of claim 10, wherein the generated grammar comprises common variations of the input field.

14. The system of claim 10, wherein the index is divided into two sub-indices.

15. The system of claim 10, wherein the phonotactic grammar is an N-gram phonotactic grammar.

16. The system of claim 10, the computer-readable storage medium having additional instructions stored which result in the operations further comprising performing one of another speech recognition on the received speech, constrain recognition for the input field, and rescore a result of performing the speech recognition.

17. The speech processing device of claim 10, the computer-readable storage medium having additional instructions stored which result in the operations further comprising:

utilizing only factors of interest from the index to generate a cost-normalized query based on the normalized query; and

selecting the N-best lowest indices from indices of the index containing the factors of interest and the cost-normalized query.

18. A computer-readable storage device having instructions stored which, when executed by a computing device, cause the computing device to perform operations comprising:

instructions for performing, via a processor, a first pass of entries in a database to generate a shortlist of the entries, the first pass comprising;

utilizing a phonotactic grammar to perform speech recognition on received speech and to generate a phoneme lattice, wherein the utilizing ignores inter-field constraints of the phonotactic grammar;

removing silence and filler words from the phoneme lattice to yield a revised phoneme lattice;

normalizing costs in the revised phoneme lattice, to yield a normalized phoneme lattice with a best path having a cost of zero;

identifying N-best lowest cost indices of an index for the database based on the normalized phoneme lattice; and

generating the shortlist of the entries in the database based on the N-best lowest cost indices; and

instructions for performing a second pass on the shortlist of the entries in the database to obtain a final result, the second pass comprising;

generating a grammar from the shortlist of the entries in the database to yield a generated grammar, having inter-field constraints; and

identifying data from the database for each of the input field based on the received speech, the generated grammar, and the inter-field constraints; and

populating an input field of a form using the final result.

19. The computer-readable storage device of claim 18, wherein the index is previously generated based on a plurality of training phoneme lattices and factors of interest from valid entries in the database, the factors of interest comprising trigrams, and wherein the database is a name/address database.

20. The computer-readable storage device of claim 19, wherein the factors of interest comprise N-grams based on the valid entries in the database.

21. The computer-readable storage device of claim 18, wherein the generated grammar comprises common variations of the input field.

22. The computer-readable storage device of claim 18, wherein the instructions for utilizing a phonotactic grammar to perform speech recognition on the received speech and to generate a phoneme lattice further comprise instructions for generating the phoneme lattice by utilizing an N-gram phonotactic grammar.

23. The computer-readable storage device of claim 18, wherein generating the shortlist of the entries in the database further comprises:

utilizing only factors of interest from the index to generate a cost-normalized query based on the normalized lattice; and

selecting the N-best lowest indices from indices of the index containing the factors of interest and the cost-normalized query.