Methods and apparatus for training a pattern recognition system using maximal rank likelihood as an optimization function

US 6,850,888 B1
Filed: 10/06/2000
Issued: 02/01/2005
Est. Priority Date: 10/06/2000
Status: Expired due to Fees

First Claim

Patent Images

1. A method for selecting model parameters during a training phase of a pattern recognition system, comprising the steps of:

obtaining an observation sequence, o_t; and

evaluating a pseudo-rank likelihood of said observation sequence, o_t, to obtain model parameters during said training phase, wherein said pseudo-rank likelihood transforms discrete features of a rank likelihood to generate parameter estimations during said training phase.

View all claims

2 Assignments

Timeline View

Assignment View

0 Petitions

Accused Products

Abstract

A method and apparatus are disclosed for training a pattern recognition system, such as a speech recognition system, using an improved objective function. The concept of rank likelihood, previously applied only to the decding process, is applied in a novel manner to the parameter estimation of the training phase of a pattern recognition system. The disclosed objective function is based on a pseudo-rank likelihood that not only maximizes the likelihood of an observation for the correct class, but also minimizes the likelihoods of the observation for all other classes, such that the discrimination between classes is maximized. A training process is disclosed that utilizes the pseudo-rank likelihood objective function to identify model parameters that will result in a pattern recognizer with the lowest possible recognition error rate. The discrete nature of the rank-based rank likelihood objective function is transformed to allow the parameter estimations to be optimized during the training phase.

12 Citations

View as Search Results

31 Claims

1. A method for selecting model parameters during a training phase of a pattern recognition system, comprising the steps of:
- obtaining an observation sequence, o_t; and
  
  evaluating a pseudo-rank likelihood of said observation sequence, o_t, to obtain model parameters during said training phase, wherein said pseudo-rank likelihood transforms discrete features of a rank likelihood to generate parameter estimations during said training phase.
- View Dependent Claims (2, 3, 4, 5, 6, 7, 8)
- - 2. The method of claim 1, wherein said pseudo-rank likelihood is computed as follows:
    - $= \frac{1}{T} \sum_{t = 1}^{T} \log (L (\sum_{l^{i} = 1, l^{i}!= l^{c}}^{L} σ (\log \frac{p (o_{t} \langle l^{c})}{p (o_{t} \langle l^{i} i)})))$ where $L (r) = \frac{1}{r + 1}, σ (x) = \frac{1}{1 + e^{a x}},$ is a likelihood of o_tgiven the correct state l^c, and p(o_t|lⁱ) is a likelihood of o_tgiven any other state lⁱ, lⁱε
      
      {all states in the underlying system}.
  - 3. The method of claim 1, wherein said pseudo-rank likelihood transforms discrete features of a rank likelihood to allow parameter estimations to be optimized during said training phase.
  - 4. The method of claim 1, wherein said pseudo-rank likelihood is optimized using all training data to project feature vectors into a discriminative space with a reduced dimension.
  - 5. The method of claim 1, wherein said pseudo-rank likelihood is optimized using adaptation data to perform a feature space transformation for adaptation.
  - 6. The method of claim 1, wherein said pseudo-rank likelihood is optimized using adaptation data to perform a model space transformation for adaptation.
  - 7. The method of claim 1, wherein said pseudo-rank likelihood objective function is evaluated using a gradient search.
  - 8. The method of claim 1, wherein said pattern recognition system is a speech recognition system.

9. A method for selecting model parameters during a training phase of a pattern recognition system, comprising the steps of:
- obtaining an observation sequence, o_t; and
  
  evaluating a pseudo-rank likelihood objective function of said observation sequence, o_t, to obtain model parameters during said training phase, wherein said pseudo-rank likelihood objective function maximizes a likelihood of an observation for a correct class, and minimizes the likelihoods of said observation for all other classes such that the discrimination between said classes is maximized and wherein said pseudo-rank likelihood transforms discrete features of a rank likelihood to generate parameter estimations during said training phase.
- View Dependent Claims (10, 11)
- - 10. The method of claim 9, wherein said pseudo-rank likelihood is computed as follows:
    - $= \frac{1}{T} \sum_{t = 1}^{T} \log (L (\sum_{l^{i} = 1, l^{i}!= l^{c}}^{L} σ (\log \frac{p (o_{t} \langle l^{c})}{p (o_{t} \langle l^{i} i)})))$ where $L (r) = \frac{1}{r + 1}, σ (x) = \frac{1}{1 + e^{a x}},$ is a likelihood of o_tgiven the correct state l^c, and p(o_t|lⁱ) is a likelihood of o_tgiven any other state lⁱ, lⁱε
      
      {all states in the underlying system}.
  - 11. The method of claim 9, wherein said pseudo-rank likelihood objective function is evaluated using a gradient search.

12. A method for selecting parameters, comprising the steps of:
- obtaining an observation sequence, o_t; and
  
  evaluating a maximal rank likelihood objective function of said observation sequence, o_t, to obtain said parameters, wherein said maximal rank likelihood transforms discrete features of a rank likelihood to generate parameter estimations during said training phase.
- View Dependent Claims (13, 14)
- - 13. The method of claim 12, wherein said maximal rank likelihood is computed as follows:
    - $= \frac{1}{T} \sum_{t = 1}^{T} \log (L (\sum_{l^{i} = 1, l^{i} 1 = l^{c}}^{L} σ (\log \frac{p (o_{t} \rangle l^{c})}{p (o_{t} \rangle l^{i} i)})))$ where $L (r) = \frac{1}{r + 1}, σ (x) = \frac{1}{1 + e^{α x}},$ is a likelihood of o_tgiven the correct state l_c, and p(o_t|lⁱ) is a likelihood of o_tgiven any other state lⁱ, lⁱε
      
      {all states in the underlying system}.
  - 14. The method of claim 13, wherein said maximal rank likelihood is evaluated using a gradient search.

15. A method for selecting model parameters during a training phase of a pattern recognition system, comprising the steps of:
- obtaining an observation sequence, o_t; and
  
  evaluating a pseudo-rank likelihood objective function of said observation sequence, o_t, to obtain model parameters during said training phase, wherein said pseudo-rank likelihood objective function is evaluated using a gradient search.
- View Dependent Claims (16)
- - 16. The method of claim 15, wherein said pseudo-rank likelihood is computed as follows:
    - $= \frac{1}{T} \sum_{t = 1}^{T} \log (L (\sum_{l^{i} = 1, l^{i} 1 = l^{c}}^{L} σ (\log \frac{p (o_{t} \rangle l^{c})}{p (o_{t} \rangle l^{i} i)})))$ where $L (r) = \frac{1}{r + 1}, σ (x) = \frac{1}{1 + e^{α x}},$ is a likelihood of o_tgiven the correct state l^c, and p(o_t|lⁱ) is a likelihood of o_tgiven any other state lⁱ, lⁱε
      
      {all states in the underlying system}.

17. A method for selecting parameters for an acoustic model during a training phase of a speech recognition system, comprising the steps of:
- obtaining an observation sequence, o_t, of speech; and
  
  evaluating a pseudo-rank likelihood of said observation sequence, o_t, to obtain model parameters during said training phase of said speech recognition system, wherein said pseudo-rank likelihood transforms discrete features of a rank likelihood to generate parameter estimations during said training phase.
- View Dependent Claims (18, 19)
- - 18. The method of claim 17, wherein said speech is pulse-code modulated speech samples.
  - 19. The method of claim 17, wherein said pseudo-rank likelihood is computed as follows:
    - $= \frac{1}{T} \sum_{t = 1}^{T} \log (L (\sum_{l^{i} = 1, l^{i} 1 = l^{c}}^{L} σ (\log \frac{p (o_{t} \rangle l^{c})}{p (o_{t} \rangle l^{i} i)})))$ where $L (r) = \frac{1}{r + 1}, σ (x) = \frac{1}{1 + e^{α x}},$ is a likelihood of o_tgiven the correct state l^c, and p(o_t|lⁱ) is a likelihood of o_tgiven any other state lⁱ, lⁱε
      
      ) {all states in the underlying system}.

20. A system for selecting model parameters during a training phase of a pattern recognition system, comprising:
- a memory that stores computer-readable code; and
  
  a processor operatively coupled to said memory, said processor configured to implement said computer-readable code, said computer-readable code configured to;
  
  obtain an observation sequence, o_t; and
  
  evaluate a pseudo-rank likelihood of said observation sequence, o_t, to obtain model parameters during said training phase, wherein said pseudo-rank likelihood transforms discrete features of a rank likelihood to generate parameter estimations during said training phase.
- View Dependent Claims (21)
- - 21. The system of claim 20, wherein said pseudo-rank likelihood is computed as follows:
    - $= \frac{1}{T} \sum_{t = 1}^{T} \log (L (\sum_{l^{i} = 1, l^{i} 1 = l^{c}}^{L} σ (\log \frac{p (o_{t} \rangle l^{c})}{p (o_{t} \rangle l^{i} i)})))$ where $L (r) = \frac{1}{r + 1}, σ (x) = \frac{1}{1 + e^{α x}},$ is a likelihood of o^tgiven the correct state l^c, and p(o_t|lⁱ) is a likelihood of o_tgiven any other state lⁱ, lⁱε
      
      {fall states in the underlying system}.

22. A system for selecting model parameters during a training phase of a pattern recognition system, comprising:
- a memory that stores computer-readable code; and
  
  a processor operatively coupled to said memory, said processor configured to implement said computer-readable code, said computer-readable code configured to;
  
  obtain an observation sequence, o_t; and
  
  evaluate a pseudo-rank likelihood objective function of said observation sequence, o_t, to obtain model parameters during said training phase, wherein said pseudo-rank likelihood objective function maximizes a likelihood of an observation for a correct class, and minimizes the likelihoods of said observation for all other classes such that the discrimination between said classes is maximized, wherein said pseudo-rank likelihood transforms discrete features of a rank likelihood to generate parameter estimations during said training phase.
- View Dependent Claims (23)
- - 23. The system of claim 22 wherein said pseudo-rank likelihood is computed as follows:
    - $= \frac{1}{T} \sum_{t = 1}^{T} \log (L (\sum_{l^{i} = 1, l^{i} 1 = l^{c}}^{L} σ (\log \frac{p (o_{t} \rangle l^{c})}{p (o_{t} \rangle l^{i} i)})))$ where $L (r) = \frac{1}{r + 1}, σ (x) = \frac{1}{1 + e^{α x}},$ is a likelihood of o_tgiven the correct state l^c, and p(o_t|lⁱ) is a likelihood of o_tgiven any other state lⁱ, lⁱε
      
      {all states in the underlying system}.

24. A system for selecting model parameters during a training phase of a pattern recognition system, comprising:
- a memory that stores computer-readable code; and
  
  a processor operatively coupled to said memory, said processor configured to implement said computer-readable code, said computer-readable code configured to;
  
  obtain an observation sequence, o_t; and
  
  evaluate a pseudo-rank likelihood objective function of said observation sequence, o_t, to obtain model parameters during said training phase, wherein said pseudo-rank likelihood objective function is evaluated using a gradient search.
- View Dependent Claims (25)
- - 25. The system of claim 24, wherein said pseudo-rank likelihood is computed as follows:
    - $= \frac{1}{T} \sum_{t = 1}^{T} \log (L (\sum_{l^{i} = 1, l^{i} 1 = l^{c}}^{L} σ (\log \frac{p (o_{t} \rangle l^{c})}{p (o_{t} \rangle l^{i} i)})))$ where $L (r) = \frac{1}{r + 1}, σ (x) = \frac{1}{1 + e^{α x}},$ is a likelihood of o_tgiven the correct state l^c, and p(o_t|lⁱ) is a likelihood of o_tgiven any other state lⁱ, lⁱε
      
      {all states in the underlying system}.

26. A system for selecting parameters for an acoustic model during a training phase of a speech recognition system, comprising:
- a memory that stores computer-readable code; and
  
  a processor operatively coupled to said memory, said processor configured to implement said computer-readable code, said computer-readable code configured to;
  
  obtain an observation sequence, o_t, of speech; and
  
  evaluate a pseudo-rank likelihood of said observation sequence, o_t, to obtain model parameters during said training phase of said speech recognition system, wherein said pseudo-rank likelihood transforms discrete features of a rank likelihood to generate parameter estimations during said training phase.
- View Dependent Claims (27, 28)
- - 27. The system of claim 26, wherein said speech is pulse-code modulated speech samples.
  - 28. The system of claim 26, wherein said pseudo-rank likelihood is computed as follows:
    - $= \frac{1}{T} \sum_{t = 1}^{T} \log (L (\sum_{l^{i} = 1, l^{i} 1 = l^{c}}^{L} σ (\log \frac{p (o_{t} \rangle l^{c})}{p (o_{t} \rangle l^{i} i)})))$ where $L (r) = \frac{1}{r + 1}, σ (x) = \frac{1}{1 + e^{α x}},$ is a likelihood of o_tgiven the correct state l^c, and p(o_t|lⁱ) is a likelihood of o_tgiven any other state lⁱ, lⁱε
      
      {all states in the underlying system}.

29. An article of manufacture for selecting model parameters during a training phase of a pattern recognition system, comprising:
- a computer readable medium having computer readable code means embodied thereon, said computer readable program code means comprising;
  
  a step to obtain an observation sequence, o_t; and
  
  a step to evaluate a pseudo-rank likelihood of said observation sequence, o_t, to obtain model parameters during said training phase, wherein said pseudo-rank likelihood transforms discrete features of a rank likelihood to generate parameter estimations during said training phase.

30. An article of manufacture for selecting model parameters during a training phase of a pattern recognition system, comprising:
- a computer readable medium having computer readable code means embodied thereon, said computer readable program code means comprising;
  
  a step to obtain an observation sequence, o_t; and
  
  a step to evaluate a pseudo-rank likelihood objective function of said observation sequence, o_t, to obtain model parameters during said training phase, wherein said pseudo-rank likelihood objective function maximizes a likelihood of an observation for a correct class, and minimizes the likelihoods of said observation for all other classes such that the discrimination between said classes is maximized and wherein said pseudo-rank likelihood transforms discrete features of a rank likelihood to generate parameter estimations during said training phase.

31. An article of manufacture for selecting parameters for an acoustic model during a training phase of a speech recognition system, comprising:
- a computer readable medium having computer readable code means embodied thereon, said computer readable program code means comprising;
  
  a step to obtain an observation sequence, o_t, of speech; and
  
  a step to evaluate a pseudo-rank likelihood of said observation sequence, o_t, to obtain model parameters during said training phase of said speech recognition system, wherein said pseudo-rank likelihood transforms discrete features of a rank likelihood to generate parameter estimations during said training phase.

Specification

Resources

Litigation Campaign Assessment

Current Assignee
Nuance Communications, Inc. (Microsoft Corporation)
Original Assignee
International Business Machines Corporation
Inventors
Li, Yongxin, Gao, Yuqing, Picheny, Michael Alan
Primary Examiner(s)
ABEBE, DANIEL DEMELASH

Application Number

US09/680,706
Time in Patent Office

1,579 Days
Field of Search

704/200, 704/231, 704/243, 704/245, 704/250, 704/256, 704/244
US Class Current

704/256
CPC Class Codes

G10L 15/144 Training of HMMs

Methods and apparatus for training a pattern recognition system using maximal rank likelihood as an optimization function

First Claim

2 Assignments

0 Petitions

Accused Products

Abstract

12 Citations

31 Claims

Specification

Solutions

Use Cases

Quick Links

Methods and apparatus for training a pattern recognition system using maximal rank likelihood as an optimization function

First Claim

2 Assignments

Subscription Required

Subscription Required

0 Petitions

Subscription Required

Accused Products

Subscription Required

Abstract

12 Citations

31 Claims

Specification

Subscription Required

Solutions

Use Cases

Quick Links