Systems and methods for generating a message topic training dataset from user interactions in message clients

• US 10,454,871 B2
• Filed: 11/26/2014
• Issued: 10/22/2019
• Est. Priority Date: 11/26/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Lukacs, Laszlo, Kurach, Karol, Miklos, Balint
• Primary Examiner(s): Nano, Sargon N
• Assistant Examiner(s): Robinson, Christopher B
• Application Number:
  US14/555,312
• Publication Number:
  US 20160149849A1
• Time in Patent Office: 1,791 Days
• Field of Search: None
• CPC Class Codes: H04L 51/234 for tracking messages
• CPC Class Codes: H04L 51/42 Mailbox-related aspects, e.g. synchronisation of mailboxes
• CPC Class Codes: H04W 4/12 Messaging; Mailboxes; Announcements

Abstract

Systems and methods for classifying messages are provided. Each message in a plurality of messages is classified, thereby independently identifying a message category in a set of message categories for each respective message in the plurality of messages. The plurality of messages is delivered to a plurality of recipients with a designation of the message category of each respective message in the first plurality of messages. A plurality of recipient initiated message interaction events for messages in the first plurality of messages over a predetermined period of time is collected from the plurality of recipients. A message categorization dataset is then constructed from (i) the first plurality of messages, (ii) the designation of the message category of each respective message in the subset of the first plurality of messages, and (iii) the plurality of recipient initiated message interaction events. This message categorization dataset is used to train or evaluate a message classifier.

Claims

1. A method of classifying messages, the method comprising:

at a computer system having one or more processors, and memory storing one or more programs configured for execution by the one or more processors;

classifying, according to a first message classifier, each message in a first plurality of messages, thereby independently identifying a respective original message category in a set of message categories for each message in the first plurality of messages;

delivering the first plurality of messages to a plurality of recipients with a designation of the respective original message category of each message in the first plurality of messages;

collecting a plurality of recipient initiated message interaction events for messages in the first plurality of messages from the plurality of recipients, including a plurality of priority designation assignments and a plurality of message category change events in which a recipient has changed the category of a message from the original message category assigned by the first message classifier to an updated message category in the set of message categories;

identifying a subset of the first plurality of messages as having trusted message category assignments based on the plurality of recipient initiated message interaction events;

constructing a message categorization dataset from the identified subset, comprising, for each message from the identified subset, information identifying the message, information corresponding to the currently assigned message category for the message, and information corresponding to a level of confidence in the message category for the message;

training a second message classifier using the message categorization dataset;

after training the second message classifier, classifying each message in a second plurality of messages using the second message classifier; and

delivering the second plurality of messages to the plurality of recipients with a designation of the respective message category of each message in the second plurality of messages, as determined by the second message classifier;

wherein;

the message categorization dataset comprises a subset of the first plurality of messages;

the classifying is performed by the first message classifier that was constructed at a time prior to the classifying;

the plurality of recipient initiated message interaction events include events in which the recipient changes the category of a message from the original message category assigned by the first message classifier to an updated message category in the set of message categories; and

the constructing includes selecting, for the subset of the first plurality of messages, those messages in the first plurality of messages that have undergone a recipient initiated message interaction event.

2. The method of claim 1, wherein the first message classifier was constructed at a time prior to the classifying.

3. The method of claim 1, wherein the set of message categories comprises promotions, social, updates, forums, travel, finance and receipts.

4. The method of claim 1, wherein each recipient in the plurality of recipients is associated with a client in a plurality of clients and wherein delivering the first plurality of messages comprises delivering each message in the first plurality of messages to the respective client in the plurality of clients that is associated with the respective recipient of the message.

5. The method of claim 1, wherein

the constructing further includes identifying a plurality of senders (i) whose messages in the first plurality of messages are consistently categorized by the first message classifier into the same respective original message categories in the set of message categories and (ii) whose messages are associated with less than a predetermined number of message category change events.

6. The method of claim 5, wherein messages in the first plurality of messages are consistently categorized by the first message classifier into the same respective original message categories in the set of message categories when at least a predetermined threshold percentage of the messages from the plurality of senders are categorized into two or less message categories in the set of message categories by the first message classifier.

7. The method of claim 6, wherein each sender in the plurality of senders has more than a threshold number of messages in the first plurality of messages.

8. The method of claim 1, wherein the recipient initiated message interaction event comprises a recipient of a respective message in the first plurality of messages replying to the respective message;

and

the constructing comprises selecting the respective message in the first plurality of messages where the recipient replied to the respective message.

9. The method of claim 1, wherein the recipient initiated message interaction event comprises a recipient of a respective message in the first plurality of messages forwarding the respective message;

and

the constructing comprises selecting the respective message in the first plurality of messages where the recipient forwarded the respective message.

10. The method of claim 1, wherein each message in the first plurality of messages comprises a calendar entry, an email, short message service (SMS) text message, a multimedia messaging service (MMS) message, a file, a document, a video, an image, or an electronic conversation.

11. The method of claim 1, wherein the constructing a message categorization dataset comprises assigning a weight to a respective message in the plurality of messages based upon one or more recipient initiated message interaction events that are associated with the respective message, and wherein the information corresponding to the level of confidence in the original message category for the respective message comprises the weight assigned to the respective message.

12. The method of claim 11, wherein:

the assigning assigns a first value to the weight for a respective message when the one or more recipient initiated message interaction events for the respective message are positive interactions with respect to the original message category of the respective message; and

the assigning assigns a second value to the weight for the respective message when the one or more recipient initiated message interaction events for the respective message are negative interactions with respect to the original message category of the respective message.

13. The method of claim 11, wherein the weight is a weight on a continuous scale, and wherein the assigning assigns a value to the weight as a function of the one or more recipient initiated message interaction events associated with the respective message.

16. The method of claim 1, wherein the collected recipient initiated message interaction events include positive events that explicitly or tacitly confirm the original message category for respective messages in the first plurality of messages;

and

the method further comprises;

upweighting the level of confidence in the original message category for a respective message in response to positive events, if any, for the respective message; and

downweighting the level of confidence in the original message category for the respective message in response to message category change events, if any, for the respective message.

14. A computing system, comprising:

one or more processors;

memory storing one or more programs to be executed by the one or more processors, wherein the one or more programs comprise instructions for;

classifying, according to a first message classifier, each message in a first plurality of messages, thereby independently identifying a respective original message category in a set of message categories for each respective message in the first plurality of messages;

delivering the first plurality of messages to a plurality of recipients with a designation of the respective original message category of each message in the first plurality of messages;

collecting a plurality of recipient initiated message interaction events for messages in the first plurality of messages from the plurality of recipients, including a plurality of priority designation assignments and a plurality of message category change events in which a recipient has changed the category of a message from the original message category assigned by the first message classifier to an updated message category in the set of message categories;

identifying a subset of the first plurality of messages as having trusted message category assignments based on the plurality of recipient initiated message interaction events;

constructing a message categorization dataset from the identified subset, comprising, for each message from the identified subset, information identifying the message, information corresponding to the currently assigned message category for the message, and information corresponding to a level of confidence in the message category for the message;

training a second message classifier using the message categorization dataset;

after training the second message classifier, classifying each message in a second plurality of messages using the second message classifier; and

delivering the second plurality of messages to the plurality of recipients with a designation of the respective message category of each message in the second plurality of messages, as determined by the second message classifier;

wherein;

the message categorization dataset comprises a subset of the first plurality of messages;

the classifying is performed by the first message classifier that was constructed at a time prior to the classifying;

the plurality of recipient initiated message interaction events include events in which the recipient changes the category of a message from the original message category assigned by the first message classifier to an updated message category in the set of message categories; and

the constructing includes selecting, for the subset of the first plurality of messages, those messages in the first plurality of messages that have undergone a recipient initiated message interaction event.

15. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

classifying, according to a first message classifier, each message in a first plurality of messages, thereby independently identifying a respective original message category in a set of message categories for each respective message in the first plurality of messages;

delivering the first plurality of messages to a plurality of recipients with a designation of the respective original message category of each message in the first plurality of messages;

collecting a plurality of recipient initiated message interaction events for messages in the first plurality of messages from the plurality of recipients, including a plurality of priority designation assignments and a plurality of message category change events in which a recipient has changed the category of a message from the original message category assigned by the first message classifier to an updated message category in the set of message categories;

identifying a subset of the first plurality of messages as having trusted message category assignments based on the plurality of recipient initiated message interaction events;

constructing a message categorization dataset from the identified subset, comprising, for each message from the identified subset, information identifying the message, information corresponding to the currently assigned message category for the message, and information corresponding to a level of confidence in the message category for the message;

training a second message classifier using the message categorization dataset;

after training the second message classifier, classifying each message in a second plurality of messages using the second message classifier; and

delivering the second plurality of messages to the plurality of recipients with a designation of the respective message category of each message in the second plurality of messages, as determined by the second message classifier;

wherein;

the message categorization dataset comprises a subset of the first plurality of messages;

the classifying is performed by the first message classifier that was constructed at a time prior to the classifying;

the plurality of recipient initiated message interaction events include events in which the recipient changes the category of a message from the original message category assigned by the first message classifier to an updated message category in the set of message categories; and

the constructing includes selecting, for the subset of the first plurality of messages, those messages in the first plurality of messages that have undergone a recipient initiated message interaction event.