Exponential priors for maximum entropy models

• US 20050256685A1
• Filed: 07/21/2005
• Published: 11/17/2005
• Est. Priority Date: 01/28/2004
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Goodman, Joshua T.
• Granted Patent: US 7,340,376 B2
• Time in Patent Office: Days
• US Class Current: 703/2
• CPC Class Codes: G06F 18/21 Design or setup of recognition systems or techniques; Extraction of features in feature space; Blind source separation
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/10 using kernel methods, e.g. support vector machines [SVM]

Abstract

The subject invention provides for systems and methods that facilitate optimizing one or mores sets of training data by utilizing an Exponential distribution as the prior on one or more parameters in connection with a maximum entropy (maxent) model to mitigate overfitting. Maxent is also known as logistic regression. More specifically, the systems and methods can facilitate optimizing probabilities that are assigned to the training data for later use in machine learning processes, for example. In practice, training data can be assigned their respective weights and then a probability distribution can be assigned to those weights.

Claims

1. A computer implemented method that maximizes probability values to facilitate training a machine learning system comprising:

receiving a data set;

determining an Exponential distribution as a prior;

defining one or more parameters; and

training a model based at least in part upon a subset of the data set, the Exponential prior and the one or more parameters.

2. The method of claim 1, determining an Exponential prior comprises:

providing a relatively large data set;

training a model using the large data set and the Gaussian prior;

graphing a distribution of parameter values that have at least 30 training instances; and

determining the Exponential prior by examining the distribution of parameters.

3. The method of claim 1, the Exponential prior being determined based at least in part upon a particular feature of interest.

4. The method of claim 2, the feature is an IP address.