Systems and Methods for Generating a Message Topic Training Dataset from User Interactions in Message Clients

• US 20160149849A1
• Filed: 11/26/2014
• Published: 05/26/2016
• Est. Priority Date: 11/26/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Kurach, Karol, Lukcs, Lszl, Miklos, Balint
• Granted Patent: US 10,454,871 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: H04L 51/234 for tracking messages
• CPC Class Codes: H04L 51/42 Mailbox-related aspects, e.g. synchronisation of mailboxes
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements

Abstract

Systems and methods for classifying messages are provided. Each message in a plurality of messages is classified, thereby independently identifying a message category in a set of message categories for each respective message in the plurality of messages. The plurality of messages is delivered to a plurality of recipients with a designation of the message category of each respective message in the first plurality of messages. A plurality of recipient initiated message interaction events for messages in the first plurality of messages over a predetermined period of time is collected from the plurality of recipients. A message categorization dataset is then constructed from (i) the first plurality of messages, (ii) the designation of the message category of each respective message in the subset of the first plurality of messages, and (iii) the plurality of recipient initiated message interaction events. This message categorization dataset is used to train or evaluate a message classifier.

Claims

1. A method of classifying messages, the method comprising:

at a computer system having one or more processors, and memory storing one or more programs for execution by the one or more processors;

classifying each message in a first plurality of messages, thereby independently identifying a message category in a set of message categories for each respective message in the first plurality of messages;

delivering the first plurality of messages to a plurality of recipients with a designation of the message category of each respective message in the first plurality of messages;

collecting a plurality of recipient initiated message interaction events for messages in the first plurality of messages from the plurality of recipients;

constructing a message categorization dataset from (i) the first plurality of messages, (ii) the designation of the message category of each respective message in the plurality of messages, and (iii) the plurality of recipient initiated message interaction events; and

using the message categorization dataset to train a test message classifier or measure the performance of the test message classifier.

2. The method of claim 1, wherein the classifying is performed by a baseline message classifier that was constructed at a time prior to the classifying.

3. The method of claim 1, wherein the set of message categories comprises promotions, social, updates, forums, travel, finance and receipts.

4. The method of claim 1, wherein each recipient in the plurality of recipients is associated with a client in a plurality of clients and wherein the delivering the first plurality of messages comprises delivering each respective message in the first plurality of messages to the client in the plurality of clients that is associated with the recipient of the respective message.

5. The method of claim 1, wherein

the classifying is performed by a baseline message classifier that was constructed at a time prior to the classifying,

the plurality of recipient initiated message interaction events are message category correction events in which a recipient has changed the category of a message from the message category assigned by the baseline message classifier to a different message category in the set of message categories, and

the constructing comprises identifying a plurality of senders (i) whose messages in the first plurality of messages are consistently categorized by the baseline message classifier into the same respective message categories in the set of message categories and (ii) whose messages are associated with less than a predetermined amount of message interaction events.

6. The method of claim 5, wherein messages in the first plurality of messages are consistently categorized by the baseline message classifier into the same respective message categories in the set of message categories when at least a predetermined threshold percentage of the messages from the plurality of senders are categorized into two or less message categories in the set of message categories by the baseline message classifier.

7. The method of claim 6, wherein each sender in the plurality of senders has more than a threshold number of messages in the first plurality of messages.

8. The method of claim 5, wherein the message categorization dataset comprises a subset of the first plurality of messages.

9. The method of claim 1, wherein

the message categorization dataset comprises a subset of the first plurality of messages,

the classifying is performed by a baseline message classifier that was constructed at a time prior to the classifying,

the plurality of recipient initiated message interaction events are events in which the recipient changes the category of a message from the message category assigned by the baseline message classifier to a different message category in the set of message categories, and

the constructing comprises selecting, for the subset of the first plurality of messages, those messages in the first plurality of messages that have undergone a recipient initiated message interaction event.

10. The method of claim 1, wherein

the message categorization dataset comprises a subset of the first plurality of messages, and

the constructing comprises selecting, for the subset of the first plurality of messages, those messages in the first plurality of messages associated with a recipient initiated message interaction event.

11. The method of claim 10, wherein the recipient initiated message interaction event comprises a recipient of a respective message in the first plurality of messages opening the respective message for at least a predetermined amount of time.

12. The method of claim 10, wherein the recipient initiated message interaction event comprises a recipient of a respective message in the first plurality of messages assigning the respective message a priority designation.

13. The method of claim 10, wherein the recipient initiated message interaction event comprises a recipient of a respective message in the first plurality of messages replying to the respective message.

14. The method of claim 10, wherein the recipient initiated message interaction event comprises a recipient of a respective message in the first plurality of messages forwarding the respective message.

15. The method of claim 1, wherein the message categorization dataset comprises a subset of the first plurality of messages, and wherein the constructing comprises excluding from the subset of the first plurality of messages those messages in the first plurality of messages associated with a recipient initiated message interaction event.

16. The method of claim 15, wherein the classifying is performed by a baseline message classifier that was constructed at a time prior to the classifying, and wherein the recipient initiated message interaction event comprises a recipient changing a message category of a respective message in the first plurality of messages that was assigned to the respective message by the baseline message classifier.

17. The method of claim 15 wherein the recipient initiated message interaction event comprises a recipient deleting a respective message in the first plurality of messages.

18. The method of claim 1, the method further comprising:

classifying each message in a second plurality of messages using the test message classifier, thereby independently identifying a message category in the set of message categories for each respective message in the second plurality of messages; and

delivering the second plurality of messages to the plurality of recipients with a designation of the message category of each respective message in the second plurality of messages, as respectively determined by the test message classifier.

19. The method of claim 1, wherein each message in the first plurality of messages comprises a calendar entry, an email, short message service (SMS) text message, a multimedia messaging service (MMS) message, a file, a document, a video, an image, or an electronic conversation.

20. The method of claim 1, wherein the constructing a message categorization dataset comprises independently assigning a weight to a respective message in the plurality of messages based upon one or more recipient initiated message interaction events that are associated with the respective message.

21. The method of claim 20, wherein the weight is a categorical weight, and wherein

the assigning assigns a first value to the weight when the one or more recipient initiated message interaction events are positive interactions, and

the assigning assigns a second value to the weight when the one or more recipient initiated message interaction events are negative interactions.

22. The method of claim 20, wherein the weight is a continuous weight, and wherein the assigning assigns a value to the weight as a function of the one or more recipient initiated message interaction events associated with the respective message.

23. A computing system, comprising:

one or more processors;

memory storing one or more programs to be executed by the one or more processors;

the one or more programs comprising instructions for;

classifying each message in a first plurality of messages, thereby independently identifying a message category in a set of message categories for each respective message in the first plurality of messages;

delivering the first plurality of messages to a plurality of recipients with a designation of the message category of each respective message in the first plurality of messages;

collecting a plurality of recipient initiated message interaction events for messages in the first plurality of messages from the plurality of recipients;

constructing a message categorization dataset from (i) a subset of the first plurality of messages, (ii) the designation of the message category of each respective message in the subset of the first plurality of messages, and (iii) the plurality of recipient initiated message interaction events; and

using the message categorization dataset to train a test message classifier or measure the performance of the test message classifier.

24. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

classifying each message in a first plurality of messages, thereby independently identifying a message category in a set of message categories for each respective message in the first plurality of messages;

delivering the first plurality of messages to a plurality of recipients with a designation of the message category of each respective message in the first plurality of messages;

collecting a plurality of recipient initiated message interaction events for messages in the first plurality of messages from the plurality of recipients;

constructing a message categorization dataset from (i) a subset of the first plurality of messages, (ii) the designation of the message category of each respective message in the subset of the first plurality of messages, and (iii) the plurality of recipient initiated message interaction events; and

using the message categorization dataset to train a test message classifier or measure the performance of the test message classifier.