System and method for constructing and personalizing a universal information classifier

• US 6,901,398 B1
• Filed: 06/28/2001
• Issued: 05/31/2005
• Est. Priority Date: 02/12/2001
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Horvitz, Eric J., Baribault, Gregory P., Jacobs, Andrew W.
• Primary Examiner(s): Robinson, Greta
• Assistant Examiner(s): Le, Debbie M.
• Application Number:
  US09/893,941
• Time in Patent Office: 1,433 Days
• Field of Search: 707 1- 6, 707/10, 709/206
• US Class Current: 709/207
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: Y10S 707/99932 Access augmentation or optimizing
• CPC Class Codes: Y10S 707/99933 Query processing, i.e. searching
• CPC Class Codes: Y10S 707/99935 Query augmenting and refining, e.g. inexact access
• CPC Class Codes: Y10S 707/99936 Pattern matching access
• CPC Class Codes: Y10S 707/99945 Object-oriented database structure processing

Abstract

A system and method for personalizing an information classifier is provided. The system includes a default or universal classifier that is pre-trained with features that have relatively universal discriminatory relationships for multiple people and that is operable to produce a measure that a message is classified as having one of several characteristics. The system further includes a second classifier that is constructed and personalized through via a more general search through the space of potentially discriminatory features. The second classifier, after personalization, is intended to classify information at a level exceeding that of the first classifier based on the specific preferences, habits, and desires of the user who personalizes the second classifier. The system further includes a weighting component that facilitates a combining component producing an integrated measure based on input from both the first classifier and the second classifier.

Claims

1. A system for personalizing an information classifier, comprising:

a first classifier, pre-trained with training data, that produces a first measure associated with a message classification;

a second classifier, trained with adapting data, that produces a second measure associated with the message classification;

a combining component that combines the first measure and the second measure to produce a third measure associated with message classification; and

a user interface that accepts information concerning personalizing the second classifier, the information comprises information related to at least one of an amount of the adapting data required before a confidence level is associated with the personalized classifier and the coverage of adapting data required before a confidence level is associated with the personalized classifier.

2. The system of claim 1, further comprising:

a weighting component that assigns a first weight to the first measure and a second weight to the second measure; and

the combining component further combines the first measure and the second measure to produce the third measure associated with the message classification, based, at least in part, on the first measure, the second measure, the first weight and the second weight.

3. The system of claim 1, further comprising:

an aging component that modifies a relevance of one or more messages based, at least in part, on time-based information associated with a message; and

an adapting component that modifies the second classifier.

4. The system of claim 1, the user interface displays information concerning the personalization of the second classifier.

5. The system of claim 4, the information concerning the personalization of the second classifier is displayed as a graph.

6. The system of claim 1, the information concerning personalizing the second classifier further comprises information related to a time period associated with a subset of adapting data, for which the relevance of adapting data is altered.

7. The system of claim 6, the information concerning retaining the second classifier further comprising information related to a point in time back to which modifications to the second classifier are to be rescinded.

8. The system of claim 1, the first classifier and the second classifier are implemented in one component.

9. The system of claim 1, the first classifier and the second classifier are implemented in separate components.

10. The system of claim 1, the training data employed in training the first classifier includes at least one of a header structure, an indicia of junk mail, a percentage of non-alphanumeric characters, capitalization patterns, relationships in an organization chart, length of messages, times of messages, dates of messages, tense usage, presence of questions and number of questions.

11. The system of claim 1, the adapting data includes at least one of an explicit data set and an observation data set.

12. The system of claim 11, the explicit data set comprises one or more pre-determined messages to be classified by a user of the system, the user classification of the pre-determined messages are employed in adapting the second classifier.

13. The system of claim 11, the observation data set comprises at least one message received by a user of the system.

14. The system of claim 13, the observation data set further comprising action data associated with the at least one message received by a user of the system.

15. The system of claim 1, the first classifier employs at least one of a support vector methodology, a naï

ve Bayesian processing methodology, a sophisticated Bayesian processing methodology, a similarity analysis employing dot product and/or cosine function processing and decision tree processing to produce the first measure.

16. The system of claim 1, the second classifier employs at least one of a support vector methodology, a naï

ve Bayesian processing methodology, a sophisticated Bayesian processing methodology, a similarity analysis employing dot product and/or cosine function processing and decision tree processing to produce the second measure.

17. The system of claim 1, the first measure is associated with at least one of a probability that the message has a known classification type, a priority of the message and an urgency score of the message.

18. The system of claim 1, where the second measure is associated with at least one of a probability that a message has a known classification type, a priority of a message and an urgency score of a message.

19. The system of claim 1, the third measure is associated with at least one of a probability that a message has a known classification type, a priority of a message and an urgency score of a message.

20. The system of claim 2, the third measure is computed using the formula

F=m1(1−

w)+m2(w), m1 is the first measure, m2 is the second measure, w is the weight assigned to the second measure and (1−

w) is the weight assigned to the first measure.

21. The system of claim 20, the third measure is normalized to a range associated with a range of the first measure and the second measure.

22. The system of claim 1, the third measure is employed to determine how an email message should be routed.

23. The system of claim 1, the third measure is employed to determine at least one of, when an email message should be routed, when an email message should be sent to a cell phone, when an email message should be archived, when an email message should be encrypted and when an email message should be deleted.

24. The system of claim 2, the weighting component determines at least one of the first weight and the second weight based, at least in part, on the amount of adapting data that has been employed in personalizing the second classifier.

25. The system of claim 2, the weighting component determines at least one of the first weight and the second weight based, at least in part, on the coverage of the adapting data that has been employed in personalizing the second classifier.

26. The system of claim 2, the combining component produces the third measure by applying the first weight to the first measurement to produce a first weighted measure and applying the second weight to the second measure to produce a second weighted measure and by combining the first weighted measure and the second weighted measure.

27. The system of claim 3, the aging component modifies the relevance of one or more messages by manipulating at least one of, a weight associated with a message and a weight associated with one or more pieces of message data.

28. The system of claim 1, further comprising a first data store that stores at least one of, one or more messages employed in training the first classifying component and one or more pieces of message data employed in training the first classifying component.

29. The system of claim 28, further comprising a second data store that stores at least one of, one or more messages employed in personalizing the second classifier and one or more pieces of message data employed in personalizing the second classifier.

30. The system of claim 28, the aging component modifies the relevance of one or more messages by removing at least one of, the one or more messages and the one or more pieces of message data from the first data store.

31. The system of claim 29, the aging component modifies the relevance of one or more messages by removing at least one of, the one or more messages from the second data store and one or more pieces of message data from the second data store.

32. The system of claim 3, the adapting component modifies the second classifier by adjusting at least one of, one or more data stores, one or more data structures, one or more algorithms and one or more rules associated with the second classifier.

33. The system of claim 32, the adapting component modifies the second classifier based, at least in part, one a relationship between the first measure and the second measure.

34. A data packet adapted to be transmitted between two or more computer processes comprising:

information related to personalizing an information classifier, the information comprising at least one of weighting data, aging and adapting data, the information corresponds to at least one of an amount of adapting data that yields a confidence level associated with a classifier that is personalized and a coverage of adapting data that results in a confidence level associated with a classifier that is personalized.

35. A computer readable medium containing computer executable components of a system for personalizing an information classifier, comprising:

a first classifying component, pre-trained with training data, operable to produce a first measure associated with a message classification;

a second classifying component, trained with adapting data, operable to produce a second measure associated with the message classification;

a weighting component adapted to assign a first weight to the first measure and a second weight to the second measure;

a combining component adapted to combine the first measure and the second measure to produce a third measure associated with the message classification, the combining component basing the combination, at least in part, on the first measure, the second measure, the first weight and the second weight;

an aging component adapted to modify the relevance of one or more messages based and/or one or more pieces of message data, at least in part, on time-based information associated with a message;

an adapting component operable to modify the second classifier; and

a user interface that accepts information concerning personalizing the second classifier, the information comprises information related to at least one of an amount of the adapting data required before a confidence level is associated with the personalized classifier and the coverage of adapting data required before a confidence level is associated with the personalized classifier.

36. A method for personalizing an information classifying process, comprising:

receiving a message to be classified;

producing a first measure that the message is classified as having one of N characteristics, N being an integer;

producing a second measure that the message is classified as having one of N characteristics, N being an integer;

combining the first measure with the second measure to produce a third measure that the message is classified as having one of N characteristics, N being an integer, where the combining depends, at least in part, on a first weight associated with the first measure and a second weight associated with the second measure;

updating at least one of a data store, a data structure, an algorithm, a process, a thread and a rule employed in generating the second measure, based, at least in part, on a relationship between the first measure and the second measure; and

accepting information associated with personalizing the information classifying process, the information comprises information related to at least one of an amount of adapting data required before a confidence level is associated with the personalization and a coverage of adapting data required before a confidence level is associated with the personalization.

37. The method of claim 36, comprising displaying information associated with personalizing the information classifying process.

38. The method of claim 37, where displaying information associated with personalizing the information classifying process includes displaying at least one of a measure associated with the degree of personalization associated with generating the second measure and one or more time periods over which the personalizing has occurred.

39. The method of claim 36, where the information associated with personalizing the information process includes at least one of a size associated with an adapting data set employed in personalizing the classifying process, one or more time periods for which the weights assigned to personalization applied to the process employed in generating the second measure is to be manipulated and a point in time to which the process for generating the second measure should be reset.

40. The method of claim 36, further comprising:

storing at least one of one or more classified messages and one or more message data points; and

manipulating the relevance of at least one of the one or more classified messages and the one or more message data points as related to determining at least one of the first and second probabilities based, at least in part, on temporal data associated with the one or more stored classified messages and one or more message data points.

41. The method of claim 40, manipulating the relevance of at least one of the one or more classified messages and one or more message data points includes deleting at least one of the one or more classified messages and the one or more message data points.

42. The method of claim 40, manipulating the relevance of at least one of the one or more classified messages and one or more message data points involves changing one or more weights associated with at least one of the one or more classified messages and the one or more message data points.

43. The method of claim 36, the N characteristic comprise at least one of the probability that a message is of a known type, a message priority, an urgency score and a computed expected urgency.

44. A method for personalizing information classifying process, comprising:

producing a measure associated with a message classification; and

updating at least one of a data store, a data structure, an algorithm, a process, a thread and a rule employed in generating the measure, based, at least in part, on a relationship between the measure and a user input provided during supervised learning, updating utilizes information concerning personalizing a classifier related to at least one of an amount of the adapting data required before a confidence level is associated with the personalized classifier and the coverage of adapting data required before a confidence level is associated with the personalized classifier.

45. A system for personalizing an information classifier, comprising:

means for producing a first measure associated with a message classification, the first measure being associated with at least one of a probability that the message has a known classification type, a priority of the message and an urgency score of the message;

means for producing a second measure associated with the message classification, the second measure being associated with at least one of the probability that the message has a known classification type, the priority of the message and the urgency score of the message;

means for combining the first measure and the second measure to produce a third measure associated with the message classification, the third measure being produced using the formula F=m1(1−

w)+m2(w), m1 is the first measure, m2 is the second measure, w is the weight assigned to the second measure and (1−

w) is the weight assigned to the first measure; and

means for accepting information concerning personalization, the information comprises information related to at least one of an amount of the adapting data required before a confidence level is associated with the personalization and the coverage of adapting data required before a confidence level is associated with the personalization.