Exponential priors for maximum entropy models

• US 7,219,035 B2
• Filed: 07/21/2005
• Issued: 05/15/2007
• Est. Priority Date: 01/28/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Goodman, Joshua T.
• Primary Examiner(s): Hoff; Marc S.
• Assistant Examiner(s): Huynh; Phuong
• Application Number:
  US11/186,318
• Publication Number:
  US 20050256680A1
• Time in Patent Office: 663 Days
• Field of Search: 702/181, 702/180, 702/182, 702/184, 709/204, 709/206, 709/224, 704/257
• US Class Current: 702/181
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]

Abstract

The subject invention provides for systems and methods that facilitate optimizing one or mores sets of training data by utilizing an Exponential distribution as the prior on one or more parameters in connection with a maximum entropy (maxent) model to mitigate overfitting. Maxent is also known as logistic regression. More specifically, the systems and methods can facilitate optimizing probabilities that are assigned to the training data for later use in machine learning processes, for example. In practice, training data can be assigned their respective weights and then a probability distribution can be assigned to those weights.

Claims

1. A computer implemented method that facilitates maximizing probability values comprising:

employing a maximum entropy model using at least one of a plurality of Exponential priors to maximize probability values;

employing an update function for the maximum entropy model, the update function comprising an observed_count—

discount term;

bounding a parameter value;

employing the maximum entropy model to classify data; and

performing at least one of storing all or part of the classified data on a computer readable medium or presenting all or part of the classified data to a user.

2. The method of claim 1, bounding the parameter value at 0.

3. The method of claim 1, the plurality of Exponential priors corresponding to a plurality of different features, respectively.

4. The method of claim 1, wherein the Exponential prior employed depends on counts of the features.

5. The method of claim of 1, wherein the Exponential prior employed depends in part upon a usefulness of a feature.

6. The method of claim 4, the counts are based in part upon a Good-Turing estimate.

7. The method of claim 2, the update function comprising:

8. The method of claim 7, n is equal to 1.

9. The method of claim 7, n is equal to ƒ

^# which is a maximum sum of features.

10. The method of claim 2, the update function comprises solving for:

11. A data packet transmitted between two or more computer processes embodied on one or more computer readable mediums facilitating providing suggestions to an online user, the data packet comprising:

information associated with employing a maximum entropy model using at least one of a plurality of Exponential priors to maximize probability values;

employing an update function for the maximum entropy model, the update function comprising an observed_count—

discount term;

bounding a stored parameter value;

employing the maximum entropy model to classify data; and

performing at least one of storing all or part of the classified data on a computer readable medium or presenting all or part of the classified data to a user.