System and method for constructing and personalizing a universal information classifier

• US 7,293,013 B1
• Filed: 10/19/2004
• Issued: 11/06/2007
• Est. Priority Date: 02/12/2001
• Status: Expired due to Fees

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Horvitz, Eric J., Baribault, Gregory P., Jacobs, Andrew W.
• Primary Examiner(s): LE, DEBBIE M
• Application Number:
  US10/968,211
• Time in Patent Office: 1,113 Days
• Field of Search: None
• US Class Current: 707/694
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A system and method for personalizing an information classifier is provided. The system includes a default or universal classifier that is pre-trained with features that have relatively universal discriminatory relationships for multiple people and that is operable to produce a measure that a message is classified as having one of several characteristics. The system further includes a second classifier that is constructed and personalized through training via a more general search through the space of potentially discriminatory features. The second classifier, after personalization, is intended to classify information at a level exceeding that of the first classifier based on the specific preferences, habits, and desires of the user who personalizes the second classifier. The system further includes a weighting component that facilitates a combining component producing an integrated measure based on input from both the first classifier and the second classifier.

Claims

1. A system for personalizing an information classifier comprising a computer processor for executing the following software components, the system comprising:

a first classifier, pre-trained with training data, that produces a first measure associated with a message classification;

a second classifier, trained with adapting data, that produces a second measure associated with the message classification;

a combining component that combines the first measure and the second measure to produce a third measure associated with the message classification; and

a user interface that displays and accepts information concerning personalization of the second classifier, the information concerning personalization of the second classifier comprises information related to at least one of the amount of adapting data required before a confidence level is associated with the personalized classifier and the coverage of adapting data required before a confidence level is associated with the personalized classifier.

2. The system of claim 1, further comprising:

a weighting component that assigns a first weight to the first measure and a second weight to the second measure; and

the combining component further combines the first measure and the second measure to produce the third measure associated with the message classification, based, at least in part, on the first measure, the second measure, the first weight and the second weight.

3. The system of claim 1, further comprising:

an aging component that modifies the relevance of one or more messages based, at least in part, on time-based information associated with a message; and

an adapting component that modifies the second classifier.

4. The system of claim 1, the information concerning personalizing the second classifier further comprises information related to a time period associated with a subset of adapting data, for which the relevance of adapting data is altered.

5. The system of claim 4, the information concerning retraining the second classifier further comprising information related to a point in time back to which modifications to the second classifier are to be rescinded.

6. The system of claim 1, the first classifier and the second classifier are implemented in one component.

7. The system of claim 1, the first classifier and the second classifier are implemented in separate components.

8. The system of claim 1, the training data employed in training the first classifier includes at least one of a header structure, an indicia of junk mail, a percentage of non-alphanumeric characters, capitalization patterns, relationships in an organization chart, length of messages, times of messages, dates of messages, tense usage, presence of questions and number of questions.

9. The system of claim 1, the adapting data includes at least one of an explicit data set and an observation data set.

10. The system of claim 9, the explicit data set comprises one or more pre-determined messages to be classified by a user of the system, the user classifications of the pre-determined messages are employed in adapting the second classifier.

11. The system of claim 9, the observation data set comprises at least one message received by a user of the system.

12. The system of claim 11, the observation data set further comprising action data associated with the at least one message received by a user of the system.

13. The system of claim 1, the first classifier employs at least one of a support vector methodology, a naï

ve Bayesian processing methodology, a sophisticated Bayesian processing methodology, a similarity analysis employing dot product and/or cosine function processing and decision tree processing to produce the first measure.

14. The system of claim 1, the second classifier employs at least one of a support vector methodology, a naï

ve Bayesian processing methodology, a sophisticated Bayesian processing methodology, a similarity analysis employing dot product and/or cosine function processing and decision tree processing to produce the second measure.

15. The system of claim 1, the first measure is associated with at least one of a probability that the message has a known classification type, a priority of the message and an urgency score of the message.

16. The system of claim 1, the second measure is associated with at least one of a probability that the message has a known classification type, a priority of the message and an urgency score of the message.

17. The system of claim 1, the third measure is associated with at least one of a probability that the message has a known classification type, a priority of the message and an urgency score of the message.

18. The system of claim 2, the third measure is computed using the formula

F=m1(1−

w)+m2(w), m1 is the first measure, m2 is the second measure, w is the weight assigned to the second measure and (1−

w) is the weight assigned to the first measure.

19. The system of claim 18, the third measure is normalized to a range associated with a range of the first measure and the second measure.

20. The system of claim 1, the third measure is employed to determine how an email message should be routed.

21. The system of claim 1, the third measure is employed to determine at least one of, when an email message should be routed, when an email message should be sent to a cell phone, when an email message should be archived, when an email message should be encrypted and when an email message should be deleted.

22. The system of claim 2, the weighting component determines at least one of the first weight and the second weight based, at least in part, on the amount of adapting data that has been employed in personalizing the second classifier.

23. The system of claim 2, the weighting component determines at least one of the first weight and the second weight based, at least in part, on the coverage of the adapting data that has been employed in personalizing the second classifier.

24. The system of claim 2, the combining component produces the third measure by applying the first weight to the first measure to produce a first weighted measure and applying the second weight to the second measure to produce a second weighted measure and by combining the first weighted measure and the second weighted measure.

25. The system of claim 3, the aging component modifies the relevance of one or more messages by manipulating at least one of, a weight associated with a message and a weight associated with one or more pieces of message data.

26. The system of claim 1, further comprising a first data store that stores at least one of, one or more messages employed in training the first classifying component and one or more pieces of message data employed in training the first classifying component.

27. The system of claim 26, further comprising a second data store that stores at least one of, one or more messages employed in personalizing the second classifier and one or more pieces of message data employed in personalizing the second classifier.

28. The system of claim 26, the aging component modifies the relevance of one or more messages by removing at least one of, the one or more messages and the one or more pieces of message data from the first data store.

29. The system of claim 27, the aging component modifies the relevance of one or more messages by removing at least one of, the one or more messages from the second data store and one or more pieces of message data from the second data store.

30. The system of claim 3, the adapting component modifies the second classifier by adjusting at least one of, one or more data stores, one or more data structures, one or more algorithms and one or more rules associated with the second classifier.

31. The system of claim 30, the adapting component modifies the second classifier based, at least in part, one a relationship between the first measure and the second measure.

32. A computer readable medium containing computer executable components of a system for personalizing an information classifier, comprising:

a first classifying component, pre-trained with training data, to produce a first measure associated with a message classification;

a second classifying component, trained with adapting data, to produce a second measure associated with the message classification;

a weighting component to assign a first weight to the first measure and a second weight to the second measure;

a combining component to combine the first measure and the second measure to produce a third measure associated with the message classification, the combining component basing the combination, at least in part, on the first measure, the second measure, the first weight and the second weight;

an aging component to modify the relevance of one or more messages based and/or one or more pieces of message data, at least in part, on time-based information associated with a message;

an adapting component to modify the second classifying component; and

a user interface to display and accept information concerning personalization of the second classifying component, the information concerning personalization of the second classifying component comprises information related to at least one of the amount of adapting data required before a confidence level is associated with the personalized classifying component and the coverage of adapting data required before a confidence level is associated with the personalized classifying component.

33. A method for personalizing an information classifying process, comprising:

receiving a message to be classified;

producing a first measure that the message is classified as having one of N characteristics, N being an integer;

producing a second measure that the message is classified as having one of N characteristics, N being an integer;

combining the first measure with the second measure to produce a third measure that the message is classified as having one of N characteristics, N being an integer, the combining depends, at least in part, on a first weight associated with the first measure and a second weight associated with the second measure;

updating at least one of a data store, a data structure, an algorithm, a process, a thread and a rule employed in generating the second measure, based, at least in part, on a relationship between the first measure and the second measure; and

displaying and accepting information concerning personalization of the second measure, the information concerning personalization of the second measure comprises information related to at least one of the amount of adapting data required before a confidence level is associated with the personalized classifier and the coverage of adapting data required before a confidence level is associated with the personalized classifier.

34. The method of claim 33, comprising displaying information associated with personalizing the information classifying process.

35. The method of claim 34, where displaying information associated with personalizing the information classifying process includes displaying at least one of a measure associated with the degree of personalization associated with generating the second measure and one or more time periods over which the personalizing has occurred.

36. The method of claim 33, further comprising accepting information associated with personalizing the information classifying process.

37. The method of claim 36, where the information associated with personalizing the information classifying process includes at least one of a size associated with an adapting data set employed in personalizing the classifying process, one or more time periods for which the weights assigned to personalization applied to the process employed in generating the second measure is to be manipulated and a point in time to which the process for generating the second measure should be reset.

38. The method of claim 33, further comprising:

storing at least one of one or more classified messages and one or more message data points; and

manipulating the relevance of at least one of the one or more classified messages and the one or more message data points as related to determining at least one of the first and second probabilities based, at least in part, on temporal data associated with the one or more stored classified messages and one or more message data points.

39. The method of claim 38, manipulating the relevance of at least one of the one or more classified messages and one or more message data points includes deleting at least one of the one or more classified messages and the one or more message data points.

40. The method of claim 38, manipulating the relevance of at least one of the one or more classified messages and one or more message data points involves changing one or more weights associated with at least one of the one or more classified messages and the one or more message data points.

41. The method of claim 33, the N characteristics comprise at least one of the probability that a message is of a known type, a message priority, an urgency score and a computed expected urgency.

42. A method for personalizing an information classifying process, comprising:

producing a measure associated with a message classification;

updating at least one of a data store, a data structure, an algorithm, a process, a thread and a rule employed in generating the measure, based, at least in part, on a relationship between the measure and a user input provided during supervised learning; and

displaying and accepting information concerning personalization of the measure, the information concerning personalization of the measure comprises information related to at least one of the amount of adapting data required before a confidence level is associated with the personalized classifier and the coverage of adapting data required before a confidence level is associated with the personalized classifier.

43. A system for personalizing an information classifier comprising a computer processor for executing the following software components, the system is recorded on a computer-readable medium and capable of execution by a computer, comprising:

means for producing a first measure associated with a message classification, the first measure being associated with at least one of a probability that the message has a known classification type, the priority of the message and the urgency score of the message;

means for producing a second measure associated with the message classification, the second measure being associated with at least one of the probability that the message has a known classification type, the priority of the message and the urgency score of the message;

means for combining the first measure and the second measure to produce a third measure associated with the message classification, the third measure being produced using the formula F=m1(1−

w)+m2(w), m1 is the first measure, m2 is the second measure, w is the weight assigned to the second measure and (1−

w) is the weight assigned to the first measure; and

means for displaying and accepting information concerning personalization of the second measure, the information concerning personalization of the second measure comprises information related to at least one of the amount of adapting data required before a confidence level is associated with the personalized classifier and the coverage of adapting data required before a confidence level is associated with the personalized classifier.