Maximum entropy and maximum likelihood criteria for feature selection from multivariate data

• US 6,609,094 B1
• Filed: 05/22/2000
• Issued: 08/19/2003
• Est. Priority Date: 05/22/2000
• Status: Expired due to Term

• Current Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Original Assignee: International Business Machines Corporation
• Inventors: Olsen, Peder, Micchelli, Charles A., Basu, Sankar
• Primary Examiner(s): Dorvil, Richemond
• Assistant Examiner(s): PATEL, KINARI M
• Application Number:
  US09/576,429
• Time in Patent Office: 1,184 Days
• Field of Search: 704/239, 704/240, 704/254
• US Class Current: 704/240
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit

Abstract

Improvements in speech recognition systems are achieved by considering projections of the high dimensional data on lower dimensional subspaces, subsequently by estimating the univariate probability densities via known univariate techniques, and then by reconstructing the density in the original higher dimensional space from the collection of univariate densities so obtained. The reconstructed density is by no means unique unless further restrictions on the estimated density are imposed. The variety of choices of candidate univariate densities as well as the choices of subspaces on which to project the data including their number further add to this non-uniqueness. Probability density functions are then considered that maximize certain optimality criterion as a solution to this problem. Specifically, those probability density function'"'"'s that either maximize the entropy functional, or alternatively, the likelihood associated with the data are considered.

Claims

1. A method of high dimensional density estimation and feature extraction from multivariate data in speech recognition where the number of directions is the same as the dimension of sampled data comprising the steps of:

projecting the multivariate data on chosen directions;

reconstructing a univariate probability density from the projected data for each choice of directions; and

using the univariate probability densities to reconstruct an unknown multivariate density.

2. The method of high dimensional density estimation and feature extraction from multivariate data in speech recognition recited in claim 1, wherein the univariate probability densities are reconstructed by a maximum entropy criterion.

3. The method of high dimensional density estimation and feature extraction from multivariate data in speech recognition recited in claim 1, wherein the univariate probability densities are reconstructed by a maximum likelihood criterion.

4. The method of high dimensional density estimation and feature extraction from multivariate data in speech recognition recited in claim 1, further comprising the step of choosing directions for projecting the multivariate data using an iterative numerical algorithm for estimating multivariate density and computing directions for projections via rotational transformations when the univariate densities are mixtures of Gaussians.

5. The method of high dimensional density estimation and feature extraction from multivariate data in speech recognition recited in claim 4, wherein the univariate probability densities are reconstructed by a maximum entropy criterion.

6. The method of high dimensional density estimation and feature extraction from multivariate data in speech recognition recited in claim 4, wherein the univariate probability densities are reconstructed by a maximum likelihood criterion.

7. The method of high dimensional density estimation and feature extraction from multivariate data in speech recognition recited in claim 1, further comprising the step of choosing directions for projecting the multivariate data using an iterative numerical algorithm for estimating multivariate density and computing directions for projections via rotational transformations when the univariate densities are Parzen density estimators.

8. The method of high dimensional density estimation and feature extraction from multivariate data in speech recognition recited in claim 7, wherein the univariate probability densities are reconstructed by a maximum entropy criterion.

9. The method of high dimensional density estimation and feature extraction from multivariate data in speech recognition recited in claim 7, wherein the univariate probability densities are reconstructed by a maximum likelihood criterion.