Computer program product for automatic recognition of a consistent message using multiple complimentary sources of information

• US 5,621,809 A
• Filed: 06/07/1995
• Issued: 04/15/1997
• Est. Priority Date: 06/09/1992
• Status: Expired due to Term

• Current Assignee: Cisco Technology, Inc. (Cisco Systems, Inc.)
• Original Assignee: International Business Machines Corporation
• Inventors: Bellegarda, Jerome R., Kanevsky, Dimitri
• Primary Examiner(s): Boudreau, Leo
• Assistant Examiner(s): Davis, Monica S.
• Application Number:
  US08/481,150
• Time in Patent Office: 678 Days
• Field of Search: 382/116, 382/159, 382/187, 382/228, 381/43
• US Class Current: 382/116
• CPC Class Codes: G06F 18/256 of results relating to different input data, e.g. multimodal recognition
• CPC Class Codes: G10L 15/10 using distance or distortion measures between unknown speech and reference templates
• CPC Class Codes: G10L 15/24 Speech recognition using non-acoustical features
• CPC Class Codes: G10L 2015/0635 updating or merging of old and new templates; Mean values; Weighting

Abstract

A general approach is provided for the combined use of several sources of information in the automatic recognition of a consistent message. For each message unit (e.g., word) the total likelihood score is assumed to be the weighted sum of the likelihood scores resulting from the separate evaluation of each information source. Emphasis is placed on the estimation of weighing factors used in forming this total likelihood. This method can be applied, for example, to the decoding of a consistent message using both handwriting and speech recognition. The present invention includes three procedures which provide the optimal weighing coefficients.

Claims

1. A computer program product for use with a computer system, comprising:

a computer usable medium having computer readable program code means embodied therein for causing a computer to transform a consistent message into a message recognizable by a computer, said computer program product having

first computer readable program code means for causing the computer to transform the consistent message generated by a human in at least two formats into electrical signal representations of the consistent message;

second computer readable program code means for causing the computer to produce from said electrical signal representations of the consistent message a set of parameters for each said format;

third computer readable program code means for causing the computer to generate a likelihood score of recognition from each said set of parameters;

fourth computer readable program code means for causing the computer to use said sets of parameters to train a weighting coefficient for each of the at least two formats of the consistent message, wherein said training of a weighting coefficient further comprises

partitioning means for partitioning information in each said format into one or more subunits, wherein each subunit corresponds to a piece of the message,

grouping means for grouping said subunits from each said format into a plurality of groups, wherein each said subunit in a group corresponds to the same piece of the message,

first determining means for determining a likelihood score of recognition for each of said groups of subunits,

second determining means for determining a global score for the information based on said likelihood score of recognition, and

third determining means for using said global score to determine said trained weighting coefficients;

fifth computer readable program code means for causing the computer to generate a weighted expression based on said trained weighting coefficients and said likelihood scores of recognition; and

sixth computer readable program code means for causing the computer to select a candidate message unit that maximizes said weighted expression to transform said electrical signal representations of the consistent message into a computer recognizable message.

2. A computer program product according to claim 1, wherein said trained weighting coefficients are used to produce a new set of parameters and said new set of parameters is applied to generate a new set of trained weighting coefficients.

3. A computer program product according to claim 2, wherein said new set of parameters is repeatedly produced until a stable set of trained weighting coefficients is isolated.

4. A computer program product according to claim 1, wherein the consistent message is transformed into said electrical signal representations using a transducer.

5. A computer program product according to claim 1, wherein said third determining means and said fourth determining means comprise means for generating and training said trained weighting coefficients using a deleted interpolation procedure.

6. A computer program product according to claim 5, wherein said third determining means comprises:

(a) means for initializing said trained weighting coefficients; and

(b) means for generating a weight factor for each of the at least two formats of the consistent message, whereby each of said weight factors represent quantitative information measuring the contribution of previously generated weighting coefficients; and

wherein said fourth determining means comprises means for generating said trained weighting coefficients based on a normalization of said weight factors for the at least two formats of the consistent message.

7. A computer program product according to claim 6, further comprising means for iteratively generating said weight factors and said trained weighting coefficients until said trained weighting coefficients stabilize or until a predetermined threshold is met.

8. A computer program product according to claim 1, wherein said third determining means and said fourth determining means comprise means for generating and training said trained weighting coefficients using an expectation-maximization procedure.

9. A computer program product according to claim 8, wherein said third determining means comprises:

(a) means for initializing a set of expectation probability scores and said trained weighting coefficients for each of the at least two formats of the consistent message;

(b) means for generating an expectation probability score based on previously generated expectation probability scores for each of the at least two formats of the consistent message; and

(c) means for generating said trained weighting coefficients as the average of said expectation probability scores; and

wherein said fourth determining means comprises means for iteratively generating said expectation probabilities and said training weight coefficients until a predetermined threshold is met.

10. A computer program product according to claim 1, wherein said third determining means and said fourth determining means comprise means for generating and training said trained weighting coefficients using a joint maximum likelihood procedure.

11. A computer program product according to claim 1, wherein said third determining means comprises:

(a) means for producing at least two sets of feature measurements from said at least two formats and organizing said set of measurements into at least two sets of feature vectors;

(b) means for providing a warping of said consistent message formed from said at least two formats;

(c) means for ordering said set of feature vectors based on some predefined criteria to form an ordered set of feature vectors;

(d) means for partitioning said ordered set of feature vectors to form a partitioned set of feature vectors;

(e) means for aligning said partitioned set of feature vectors in accordance with said warping step to form an aligned set of feature vectors;

(f) means for grouping said aligned set of feature vectors to form a grouped set of feature vectors, wherein each group of units from said grouped set of feature vectors gives rise to likelihood score; and

(g) means for producing a global score from said set of likelihood scores; and

wherein said fourth determining means comprises means for generating said trained weighting coefficients from said global score.

12. A computer program product according to claim 11, wherein said third determining means further comprises:

means for storing said global score;

means for reordering said set of feature vectors; and

wherein said fourth determining means further comprises means for generating said training weighting coefficients using said stored global score.

13. A computer program product according to claim 12, wherein said third determining means further comprises means for determining whether a predetermined set of criteria is met.

14. A program storage device readable by a machine, tangibly embodying a program of instructions executable by the machine to perform steps for transforming a consistent message into a message recognizable by a computer, said method steps comprising:

transforming the consistent message generated by a human in at least two formats into electrical signal representations of the consistent message;

producing from said electrical signal representations of the consistent message a set of parameters for each said format;

generating a likelihood score of recognition from each said set of parameters;

using said sets of parameters to train a weighting coefficient for each of the at least two formats of the consistent message, wherein said training comprises

partitioning information in each said format into one or more subunits, wherein each subunit corresponds to a piece of the message,

grouping said subunits from each said format into a plurality of groups, wherein each said subunit in a group corresponds to the same piece of the message,

determining a likelihood score of recognition for each of said groups of subunits,

determining a global score for the information based on said likelihood score, and

determining said trained weighting coefficients using said global score,

generating a weighted expression based on said trained weighting coefficient and said likelihood scores of recognition; and

selecting a candidate message unit that maximizes said weighted expressions to transform said electrical signal representations of the consistent message into a computer recognizable message.

15. A program storage device according to claim 14, wherein said trained weighting coefficients are used to produce a new set of parameters and said new set of parameters is applied to generate a new set of trained weighting coefficients.

16. A program storage device according to claim 15, wherein said new set of parameters is repeatedly produced until a stable set of trained weighting coefficients is isolated.

17. A program storage device according to claim 14, wherein the consistent message is transformed into said electrical signal representations using a transducer.

18. A program storage device according to claim 14, wherein said step of determining said trained weighting coefficients comprises a deleted interpolation procedure.

19. A program storage device according to claim 18, wherein said deleted interpolation procedure comprises the steps of:

(a) initializing said trained weighting coefficients;

(b) generating a weight factor for each of the at least two formats of the consistent message, whereby each of said weight factors represent quantitative information measuring the contribution of previously generated weighting coefficients; and

(c) generating said trained weighting coefficients based on a normalization of said weight factors for the at least two formats of the consistent message.

20. A program storage device according to claim 19, wherein steps of generating said weight factors and of generating said trained weighting coefficients are repeated until iterative values of said trained weighting coefficients stabilize or until a predetermined threshold is met.

21. A program storage device according to claim 14, wherein said step of determining said trained weighting coefficients comprises an expectation-maximization procedure.

22. A program storage device according to claim 21, wherein said expectation-maximization procedure comprises the steps of:

(a) initializing a set of expectation probability scores and said trained weighting coefficients for each of the at least two formats of the consistent message;

(b) generating an expectation probability score based on previously generated expectation probability scores for each of the at least two formats of the consistent message;

(c) generating said trained weighting coefficients as the average of said expectation probability scores; and

(d) repeating said step of generating said expectation probabilities and said step of generating said training weight coefficients until a predetermined threshold is met.

23. A program storage device according to claim 14, wherein said step of determining said trained weighting coefficients comprises a joint maximum likelihood procedure.

24. A program storage device according to claim 14, wherein said joint maximum likelihood procedure comprises the steps of:

(a) producing at least two sets of feature measurements from said at least two formats and organizing said set of measurements into at least two sets of feature vectors;

(b) providing a warping of said consistent message formed from said at least two formats;

(c) ordering said set of feature vectors based on some predefined criteria to form an ordered set of feature vectors;

(d) partitioning said ordered set of feature vectors to form a partitioned set of feature vectors;

(e) aligning said partitioned set of feature vectors in accordance with said warping step to form an aligned set of feature vectors;

(f) grouping said aligned set of feature vectors to form a grouped set of feature vectors, wherein each group of units from said grouped set of feature vectors gives rise to likelihood score;

(g) producing a global score from said set of likelihood scores; and

(h) generating said trained weighting coefficients from said global score.

25. A program storage device according to claim 24, wherein said joint maximum likelihood procedure further comprises the steps of:

storing said global score;

reordering said set of feature vectors; and

generating said training weighting coefficients using said stored global score.

26. A program storage device according to claim 25, wherein said joint maximum likelihood procedure further comprises the step of determining whether a predetermined set of criteria is met.