Exponential priors for maximum entropy models

• US 7,483,813 B2
• Filed: 10/19/2006
• Issued: 01/27/2009
• Est. Priority Date: 01/28/2004
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Goodman, Joshua T.
• Primary Examiner(s): Feliciano; Eliseo Ramos
• Assistant Examiner(s): Huynh; Phuong
• Application Number:
  US11/550,908
• Publication Number:
  US 20070043556A1
• Time in Patent Office: 831 Days
• Field of Search: 702181-184, 702/179, 702/188, 702/189, 703/2, 704/2, 704/55, 709/204, 709/206, 709/224, 713/154
• US Class Current: 702/181
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]

Abstract

The subject invention provides for systems and methods that facilitate optimizing one or mores sets of training data by utilizing an Exponential distribution as the prior on one or more parameters in connection with a maximum entropy (maxent) model to mitigate overfitting. Maxent is also known as logistic regression. More specifically, the systems and methods can facilitate optimizing probabilities that are assigned to the training data for later use in machine learning processes, for example. In practice, training data can be assigned their respective weights and then a probability distribution can be assigned to those weights.

Claims

1. A machine implemented system that facilitates maximizing probabilities comprising:

a data input component that provides one or more types of data for analysis;

an analysis component that analyzes at least a subset of the one or more types of data to compute maximized probabilities by employing an improved iterative scaling function, a plurality of Exponential priors that respectively correspond to a plurality of different features, and at least one of a LaPlacian prior, a non-Gaussian distribution, or an alternative iterative scaling function;

And

a training component that computes one or more weights for respective test data based at least in part on the maximized probabilities.

2. The system of claim 1, the alternative iterative scaling function comprises generalized iterative scaling.

3. The system of claim 1, the alternative iterative scaling function comprises sequential generalized iterative scaling.

4. The system of claim 1, the plurality of Exponential priors employed depends on counts of the corresponding features.

5. The system of claim of 1, the plurality of Exponential priors depends in part upon a usefulness of a feature.

6. The system of claim 4, the counts are based in part upon a Good-Turing estimate.

7. The system of claim 1, the analysis component comprising:

a maximization component that provides instructions for computing a maximum value;

a model component operatively coupled to the maximization component that receives data from at least the maximization component and at least an Exponential prior component; and

a probability processing component that employs information collected by the model component to compute one or more values.

8. A method effectuated on a machine that facilitates maximizing probability values, comprising:

receiving types of data for analysis; and

analyzing a subset of the types of data to ascertain maximized probabilities by utilizing an improved iterative scaling function and employing a plurality of Exponential priors that are respectively associated with different features; and

training a machine learning system at least in part by applying weights to respective test data based on the maximized probabilities.

9. The method of claim 8, the analyzing further includes employing at least a LaPlacian prior or a non-Gaussian distribution.

10. The method of claim 8, the analyzing further includes utilizing other iterative scaling functions.

11. The method of claim 10, the other iterative scaling functions include sequential generalized iterative scaling.

12. The method of claim 10, the other iterative scaling functions include generalized iterative scaling.

13. The method of claim 8, the plurality of Exponential priors are employed based on counts of the associated features.

14. The method of claim 8, the plurality of Exponential priors are employed based on a usefulness feature.

15. The method of claim 14, the usefulness feature is an IP address.

16. The method of claim 8, the analysis further includes providing instructions that determine a maximum value.

17. The method of claim 8, further includes determining an Exponential distribution as a prior.

18. A system that facilitates maximizing probabilities, comprising:

a memory for storing data; and

a processor coupled to the memory configured to act as;

a data input component that provides one or more types of data for analysis;

an analysis component that analyzes at least a subset of the one or more types of data to compute maximized probabilities at least in part by employing an improved iterative scaling function, at least two Exponential priors corresponding to respective disparate features, and at least one of a LaPlacian prior or a non-Gaussian distribution; and

a training component that trains a machine learning system based at least in part on the maximized probabilities.