Systems and Methods for Generating a Message Topic Training Dataset from User Interactions in Message Clients
First Claim
1. A method of classifying messages, the method comprising:
- at a computer system having one or more processors, and memory storing one or more programs for execution by the one or more processors;
classifying each message in a first plurality of messages, thereby independently identifying a message category in a set of message categories for each respective message in the first plurality of messages;
delivering the first plurality of messages to a plurality of recipients with a designation of the message category of each respective message in the first plurality of messages;
collecting a plurality of recipient initiated message interaction events for messages in the first plurality of messages from the plurality of recipients;
constructing a message categorization dataset from (i) the first plurality of messages, (ii) the designation of the message category of each respective message in the plurality of messages, and (iii) the plurality of recipient initiated message interaction events; and
using the message categorization dataset to train a test message classifier or measure the performance of the test message classifier.
2 Assignments
0 Petitions
Accused Products
Abstract
Systems and methods for classifying messages are provided. Each message in a plurality of messages is classified, thereby independently identifying a message category in a set of message categories for each respective message in the plurality of messages. The plurality of messages is delivered to a plurality of recipients with a designation of the message category of each respective message in the first plurality of messages. A plurality of recipient initiated message interaction events for messages in the first plurality of messages over a predetermined period of time is collected from the plurality of recipients. A message categorization dataset is then constructed from (i) the first plurality of messages, (ii) the designation of the message category of each respective message in the subset of the first plurality of messages, and (iii) the plurality of recipient initiated message interaction events. This message categorization dataset is used to train or evaluate a message classifier.
19 Citations
24 Claims
-
1. A method of classifying messages, the method comprising:
at a computer system having one or more processors, and memory storing one or more programs for execution by the one or more processors; classifying each message in a first plurality of messages, thereby independently identifying a message category in a set of message categories for each respective message in the first plurality of messages; delivering the first plurality of messages to a plurality of recipients with a designation of the message category of each respective message in the first plurality of messages; collecting a plurality of recipient initiated message interaction events for messages in the first plurality of messages from the plurality of recipients; constructing a message categorization dataset from (i) the first plurality of messages, (ii) the designation of the message category of each respective message in the plurality of messages, and (iii) the plurality of recipient initiated message interaction events; and using the message categorization dataset to train a test message classifier or measure the performance of the test message classifier. - View Dependent Claims (2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22)
-
23. A computing system, comprising:
-
one or more processors; memory storing one or more programs to be executed by the one or more processors; the one or more programs comprising instructions for; classifying each message in a first plurality of messages, thereby independently identifying a message category in a set of message categories for each respective message in the first plurality of messages; delivering the first plurality of messages to a plurality of recipients with a designation of the message category of each respective message in the first plurality of messages; collecting a plurality of recipient initiated message interaction events for messages in the first plurality of messages from the plurality of recipients; constructing a message categorization dataset from (i) a subset of the first plurality of messages, (ii) the designation of the message category of each respective message in the subset of the first plurality of messages, and (iii) the plurality of recipient initiated message interaction events; and using the message categorization dataset to train a test message classifier or measure the performance of the test message classifier.
-
-
24. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:
-
classifying each message in a first plurality of messages, thereby independently identifying a message category in a set of message categories for each respective message in the first plurality of messages; delivering the first plurality of messages to a plurality of recipients with a designation of the message category of each respective message in the first plurality of messages; collecting a plurality of recipient initiated message interaction events for messages in the first plurality of messages from the plurality of recipients; constructing a message categorization dataset from (i) a subset of the first plurality of messages, (ii) the designation of the message category of each respective message in the subset of the first plurality of messages, and (iii) the plurality of recipient initiated message interaction events; and using the message categorization dataset to train a test message classifier or measure the performance of the test message classifier.
-
Specification