SYSTEMS AND METHODS FOR ESTIMATING USER JUDGMENT BASED ON PARTIAL FEEDBACK AND APPLYING IT TO MESSAGE CATEGORIZATION

• US 20160156579A1
• Filed: 12/01/2014
• Published: 06/02/2016
• Est. Priority Date: 12/01/2014
• Status: Abandoned Application

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kaufmann, Tobias
• Application Number:
  US14/557,307
• Publication Number:
  US 20160156579A1
• Time in Patent Office: Days
• CPC Class Codes: H04L 51/212 using filtering or selective blocking
• CPC Class Codes: H04L 51/42 Mailbox-related aspects, e.g. synchronisation of mailboxes

Abstract

Messages in a first and second plurality of messages are respectively classified using a first and second classifier into message categories in a set of message categories, with messages in the first and second plurality of messages being associated with message reputation carriers in a plurality of message reputation carriers. The classified messages are delivered to recipients and message category correction events are collected. Correction weights are determined for correction types associated with the set of message categories using the initial message categorizations and the category correction events. At least a subset of the calculated correction weights is used to determine a probability or likelihood that a particular message reputation carrier in the plurality of carriers is associated with a first message category in the set of message categories. The particular carrier is whitelisted to the first message category when the calculated probability or likelihood satisfies a whitelisting criterion.

Claims

1. A method of whitelisting a first message reputation carrier to a first message category in a set of message categories, the method comprising:

at a computer system having one or more processors, and memory storing one or more programs for execution by the one or more processors;

classifying each message in a first plurality of messages using a first classifier, thereby independently identifying an initial message category in the set of message categories for each respective message in the first plurality of messages, wherein the first plurality of messages includes, for each respective message reputation carrier in a plurality of message reputation carriers, at least one message associated with the respective message reputation carrier and wherein the plurality of message reputation carriers includes the first message reputation carrier;

classifying each message in a second plurality of messages using a second classifier, thereby independently identifying an initial message category in the set of message categories for each respective message in the second plurality of messages, wherein the second plurality of messages includes, for each respective message reputation carrier in the plurality of message reputation carriers, at least one message associated with the respective message reputation carrier;

delivering the first and second plurality of messages to a plurality of recipients with a designation of the message category of each respective message in the first and second plurality of messages, as respectively determined by the first and second classifier;

collecting a plurality of recipient initiated message category correction events for messages in the first and second plurality of messages;

determining a correction weight for each respective correction type associated with the set of message categories using at least (i) the initial message category for each respective message in the first plurality of messages assigned by the first classifier, (ii) the initial message category for each respective message in the second plurality of messages assigned by the second classifier and (iii) the plurality of recipient initiated message category correction events;

using the correction weight for each correction type associated with a message category in the set of message categories to determine a probability or likelihood that the first message reputation carrier is associated with the first message category in the set of message categories; and

whitelisting the first message reputation carrier to the first message category when the calculated probability or likelihood satisfies a whitelisting criterion.

2. The method of claim 1, wherein the set of message categories comprises promotions, social, updates, and forums.

3. The method of claim 1, wherein

the first classifier and the second classifier are the same classifier,

the first plurality of messages is classified by the classifier at a time before a subset of message reputation carriers in the plurality of message reputation carriers are whitelisted to message categories in the set of message categories, and

the second plurality of messages is classified by the classifier at a time after the subset of message reputation carriers in the plurality of message reputation carriers are whitelisted to message categories.

4. The method of claim 1, wherein the determining a correction weight for each respective correction type associated with the set of message categories comprises minimizing the loss function:

5. The method of claim 4, wherein, the corrections weights are forced to be equal.

6. The method of claim 4, wherein, the loss function is regularized.

7. The method of claim 1, wherein the determining a correction weight for each respective correction type associated with the set of message categories comprises minimizing a loss function of the form:

8. The method of claim 7, wherein the loss function is evaluated by a gradient descent approach.

9. The method of claim 1, wherein the determining a correction weight for each respective correction type associated with the set of message categories comprises:

minimizing, as a loss function, empirical data comprising (i) the initial message category for each respective message in the first plurality of messages assigned by the first classifier (ii) the initial message category for each respective message in the second plurality of messages assigned by the second classifier, and (iii) the recipient initiated message category correction events for messages in the first and second plurality of messages against a set of model parameters, by a gradient descent approach for the correction weight for each respective correction type associated with the set of message categories.

10. The method of claim 9, wherein the gradient descent approach is a Newton method, a Broyden-Fletcher-Goldfarb-Shanno (BFGS) algorithm, or a limited memory BFGS algorithm.

11. The method of claim 1, wherein the determination of the probability or likelihood, P(C=k), that the first message reputation carrier is associated with the first message category in the set of message categories, using a correction weight for a correction type associated with the set of message categories, is determined by:

12. The method of claim 1, the method further comprising:

upon whitelisting the first message reputation carrier to the first message category,

classifying each message in a third plurality of messages associated with the first message reputation carrier into the whitelisted message category for the first message reputation carrier, and

delivering the classified third plurality of messages to recipients specified by the third plurality of messages.

13. The method of claim 1, wherein

the whitelisting criterion is a threshold probability or likelihood, and

the calculated probability or likelihood satisfies the whitelisting criterion when the calculated probability or likelihood is equal to or greater than the threshold probability or likelihood.

14. The method of claim 1, wherein

the first classifier and the second classifier are a single neural network with the same trained weights,

the first plurality of messages are classified by the neural network at a time before at least a subset of message reputation carriers in the plurality of message reputation carriers are whitelisted to message categories in the set of message categories, and

the second plurality of messages are classified by the neural network at a time after the subset of message reputation carriers in the plurality of message reputation carriers are whitelisted to message categories.

15. The method of claim 1, wherein the first classifier and the second classifier are the same classifier and the method further comprises retraining the classifier using the probability or likelihood that the first message reputation carriers is associated with the first message category in the set of message categories.

16. The method of claim 1, wherein the first category is one of promotions, social, updates, and forums.

17. The method of claim 1, wherein each recipient in the plurality of recipients is associated with a different client in a plurality of clients and wherein the delivering the first and second plurality of messages comprises delivering each respective message in the first and second plurality of messages to the client in the plurality of clients that is associated with the respective message.

18. The method of claim 1, wherein the first classifier is a single first classifier and the second classifier is a single second classifier.

19. The method of claim 18, wherein the first classifier and the second classifier are the same.

20. The method of claim 18, wherein the first classifier is different than the second classifier.

21. The method of claim 18, wherein the first classifier and the second classifier are the same classifier and wherein the first classifier and the second classifier are used in disjoint or different time periods.

22. The method of claim 1, the method further comprising:

classifying each message in a third plurality of messages using a third classifier, thereby independently identifying an initial message category in the set of message categories for each respective message in the third plurality of messages, wherein the third plurality of messages includes, for each respective message reputation carrier in the plurality of message reputation carriers, at least one message associated with the respective message reputation carrier, and wherein

the delivering further comprises delivering the third plurality of messages to the plurality of recipients with a designation of the message category of each respective message in the third plurality of messages, as respectively determined by the third classifier,

the collecting the plurality of recipient initiated message category correction events comprises collecting the plurality of recipient initiated message category correction events for messages in the first, second, and third plurality of messages, and

the determining the correction weight for each respective correction type associated with the set of message categories further uses the initial message category for each respective message in the third plurality of messages assigned by the third classifier.

23. The method of claim 21, the method further comprising:

classifying each message in a fourth plurality of messages using a fourth classifier thereby independently identifying an initial message category in the set of message categories for each respective message in the fourth plurality of messages, wherein the fourth plurality of messages includes, for each respective message reputation carrier in a the plurality of message reputation carriers, at least one message associated with the respective message reputation carrier in the plurality of message reputation carriers, and wherein

the delivering further comprises delivering the fourth plurality of messages to the plurality of recipients with a designation of the message category of each respective message in the fourth plurality of messages, as respectively determined by the fourth classifier,

the collecting the plurality of recipient initiated message category correction events comprises collecting the plurality of recipient initiated message category correction events for messages in the first, second, third and fourth plurality of messages, and

the determining the correction weight for each respective correction type associated with the set of message categories further uses the initial message category for each respective message in the fourth plurality of messages assigned by the fourth classifier.

24. The method of claim 1, wherein

the first classifier and the second classifier are the same classifier,

the classifying each message in the first plurality of messages by the first classifier occurs during a first time interval [t₁, . . . t₂], and

the classifying each message in the second plurality of messages by the second classifier occurs during a second time interval [t₃, . . . t₄] disjoint from the first interval, wherein t₂<

t₃.

25. The method of claim 24, wherein the first classifier evolves during the first time interval and the second classifier evolves in the second time interval.

26. The method of claim 1, wherein each respective message reputation carrier in the plurality of message reputation carriers is an identity of a message sender.

27. The method of claim 26, wherein the message sender is an identity of an actual person, an identity of a business, an identity of a plurality of businesses, an identity of a government organization, an identity of a group of government organizations, an identity of a plurality of people, a URL, an IP address, or a MAC address.

28. The method of claim 1, where each respective message reputation carrier in the plurality of message reputations is based on message content.

29. A computing system, comprising:

one or more processors;

memory storing one or more programs to be executed by the one or more processors;

the one or more programs comprising instructions for;

classifying each message in a first plurality of messages using a first classifier thereby independently identifying an initial message category in a set of message categories for each respective message in the first plurality of messages, wherein the first plurality of messages includes, for each respective message reputation carrier in a plurality of message reputation carriers, at least one message associated with the respective message reputation carrier;

classifying each message in a second plurality of messages using a second classifier, thereby independently identifying an initial message category in the set of message categories for each respective message in the second plurality of messages, wherein the second plurality of messages includes, for each respective message reputation carrier in the plurality of message reputation carriers, at least one message associated with the respective message reputation carrier;

delivering the first and second plurality of messages to a plurality of recipients with a designation of the message category of each respective message in the first and second plurality of messages, as respectively determined by the first and second classifier;

collecting a plurality of recipient initiated message category correction events for messages in the first and second plurality of messages;

determining a correction weight for each respective correction type associated with the set of message categories using at least (i) the initial message category for each respective message in the first plurality of messages assigned by the first classifier, (ii) the initial message category for each respective message in the second plurality of messages assigned by the second classifier, and (iii) the plurality of recipient initiated message category correction events;

using the correction weight for each correction type associated with the set of message categories to determine a probability or likelihood that a first message reputation carrier in the plurality of message reputation carriers is associated with a first message category in the set of message categories; and

whitelisting the first message reputation carrier to the first message category when the calculated probability or likelihood satisfies a whitelisting criterion.

30. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

classifying each message in a first plurality of messages using a first classifier, thereby independently identifying an initial message category in a set of message categories for each respective message in the first plurality of messages, wherein the first plurality of messages includes, for each respective message reputation carrier in a plurality of message reputation carriers, at least one message associated with the respective message reputation carrier;

classifying each message in a second plurality of messages using a second classifier, thereby independently identifying an initial message category in the set of message categories for each respective message in the second plurality of messages, wherein the second plurality of messages includes, for each respective message reputation carrier in the plurality of message reputation carriers, at least one message associated with the respective message reputation carrier;

delivering the first and second plurality of messages to a plurality of recipients with a designation of the message category of each respective message in the first and second plurality of messages, as respectively determined by the first and second classifier;

collecting a plurality of recipient initiated message category correction events for messages in the first and second plurality of messages;

determining a correction weight for each respective correction type associated with the set of message categories using at least (i) the initial message category for each respective message in the first plurality of messages assigned by the first classifier, (ii) the initial message category for each respective message in the second plurality of messages assigned by the second classifier, and (iii) the plurality of recipient initiated message category correction events;

using the correction weight for each correction type associated with the set of message categories to determine a probability or likelihood that a first message reputation carrier in the plurality of message reputation carriers is associated with a first message category in the set of message categories; and

whitelisting the first message reputation carrier to the first message category when the calculated probability or likelihood satisfies a whitelisting criterion.