Apparatus, method and article to effect electronic message reply rate matching in a network environment

• US 10,540,607 B1
• Filed: 12/08/2014
• Issued: 01/21/2020
• Est. Priority Date: 12/10/2013
• Status: Active Grant

• Current Assignee: Plentyoffish Media ULC (Match Group, Inc.)
• Original Assignee: Plentyoffish Media ULC (Match Group, Inc.)
• Inventors: Oldridge, Steve, Levi, Thomas, Li, Sa
• Primary Examiner(s): Zhen, Li B.
• Assistant Examiner(s): Lee, Tsu-Chang
• Application Number:
  US14/563,504
• Time in Patent Office: 1,870 Days
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: H04L 51/212 using filtering or selective blocking

Abstract

Relationship building Websites collect considerable self-reported and autonomously collected attribute data on users. Attribute data may be useful for identifying users having compatible or potentially compatible interests, likes, goals, and/or aspirations that the formation of a relationship between the users is possible. At least a portion of the data collected by relationship building Websites may include inbound and outbound messaging statistics and behaviors. When used in conjunction with profile attributes, these messaging statistics and behaviors may be used as training data to generate one or more response predictive models that provide an indication of the profile attributes and messaging behaviors to which a particular user is most likely to respond. Since messaging traffic is a key indicator of relationship building Website health and vitality, it is advantageous to provide users with matches or potential matches with whom they are more likely to exchange messages.

Claims

1. A method of operation in a system to enhance messaging between users, comprising:

using a communication network that includes servers, processors, display devices, input devices, and system memory devices to send messages between users and autonomously collect actual historical messaging data;

forming an initial data set, the initial data set including;

user profile information for a plurality of users, user behavioral information for at least some of the plurality of users, and message data indicative of messaging activity between at least some of the plurality of users, wherein forming an initial data set comprises selecting users who have received at least a first defined minimum number of messages, and who responded to at least a second defined minimum number of the received messages, and including the user profile information, the user behavioral information, and the messaging data for the selected users in the initial data set;

organizing the initial data set into a training data set and a test data set;

generating a response predictive model using a machine learning system from the training data set that includes the user profile information, the user behavioral information and message data;

evaluating the response predictive model generated from the training data set against the corresponding test data set, wherein at least some of the test data set is based on actual historical messaging data; and

determining an accuracy of the response prediction model using the actual historical messaging data.

2. The method of claim 1 wherein organizing the initial data set into a training data set and a test data set includes separating the initial data set into the training data set and the test data set.

3. The method of claim 1 wherein selecting users further includes selecting only users who have a logically associated attribute value indicative of a female gender.

4. The method of claim 3 wherein organizing the initial data set into a training data set and a test data set includes randomly selecting for inclusion in either the training data set or the test data set the user profile information, the user behavioral information, and the messaging data from all of the selected users on a message-by-message basis.

5. The method of claim 4 wherein the test data set and the training data set are not mutually exclusive with respect to the users represented in each.

6. The method of claim 3 wherein organizing the initial data set into a training data set and a test data set includes randomly separating the selected users for representation in one or the other of the training data set or the test data set.

7. The method of claim 6 wherein the test data set and the training data set are mutually exclusive with respect to the users represented in each.

8. The method of claim 6 wherein organizing the initial data set into a training data set and a test data set further includes, for each of a first set of randomly selected users, including all messages for each of the users in the first set of users in a training data set, and for a second set of users, different than the first set of users, including all messages for each of the users in the second set in a test data set.

9. The method of claim 8 wherein the first set of users and the second set of users are mutually exclusive with respect to the users represented in each of the sets of users.

10. The method of claim 1 wherein generating a response predictive model based from the training data set includes adjusting model weights based on back propagation.

11. The method of claim 10 wherein generating a response predictive model from the training set includes determining whether a performance in the training has plateaued, and terminating the response predictive model generation in response to determining that the performance has plateaued.

12. The method of claim 11 wherein generating a response predictive model from the training set includes determining whether a defined maximum number of training epochs has been reached, and terminating the response predictive model generation in response to reaching the defined maximum number of training epochs.

13. The method of claim 11 wherein the response predictive model is predictive of whether a message sent by one user will receive a response.

14. The method of claim 1, further comprising:

collecting at least one of;

the user profile information, the user behavioral information, or the messaging data for actual messaging between the plurality of users.

15. The method of claim 14 wherein collecting user profile information includes collecting at least one of:

a user age, a user gender, a user country, a user ethnicity, or a user profession.

16. The method of claim 14 wherein collecting user behavioral information includes collecting at least one of:

user specific messaging rates, user specific page views, or user specific login data.

17. The method of claim 1, further comprising:

converting at least one of;

the user profile information, the user behavioral information, or the messaging data to at least one of a binary or a numerical representation.

18. The method of claim 1, further comprising:

identifying at least one response predictive model for use.

19. The method of claim 18 wherein identifying at least one response predictive model for use includes comparing the response predictive model to other response predictive models.

20. The method of claim 1, further comprising:

applying the identified response predictive model.

21. The method of claim 20, further comprising:

incorporating the identified response predictive model into a determining of potential candidates.

22. The method of claim 21 wherein incorporating the identified response predictive model into a determining of potential candidates includes adjusting weights in a matching algorithm.

23. The method of claim 21, further comprising:

on creation of a new user profile, informing other users who have a high predicted response rate of responding to an initial message from a new user logically associated with the new user profile.

24. A system to enhance messaging between users, the system comprising:

a communication network that includes servers, processors, display devices, input devices, and system memory devices to send messages between users and autonomously collect actual historical messaging data;

at least one processor; and

at least one non-transitory processor-readable medium communicatively coupled to the least one processor, wherein the at least one processor;

forms an initial data set, the initial data set comprising user profile information for a plurality of users, user behavioral information for at least some of the plurality of users, and message data indicative of messaging activity between at least some of the plurality of users;

organizes the initial data set into a training data set and a test data set, wherein in order to organize the initial data set into a training data set and a test data set, the at least one processor randomly separates the selected users for representation in one or the other of the training data set or the test data set;

generates a response predictive model using a machine learning system from the training data set which includes the user profile information, the user behavioral information and message data;

evaluates the response predictive model generated from the training data set against the corresponding test data set, wherein at least some of the test data set is based on actual historical messaging data; and

determines an accuracy of the response prediction model using the actual historical messaging data.

25. The system of claim 24 wherein in order to form an initial data set, the at least one processor selects users who have received at least a first defined minimum number of messages, and who responded to at least a second defined minimum number of the received messages, and including in the initial data set the user profile information, the user behavioral information and the messaging data for the selected users.

26. The system of claim 25 wherein the at least one processor further select only users who have a logically associated attribute value indicative of a female gender.

27. The system of claim 24 wherein the test data set and the training data set are mutually exclusive with respect to the users represented in each.

28. The system of claim 24 wherein the at least one processor adjusts model weights based on back propagation in order to generate the response predictive model.

29. The system of claim 24 wherein the at least one processor further collects at least one of:

the user profile information, the user behavioral information, or the messaging data for actual messaging between the plurality of users.

30. The system of claim 29 wherein to collect user profile information, the at least one processor collects an attribute value indicative of at least one of:

a user'"'"'s age, a user'"'"'s gender, a user'"'"'s country, a user'"'"'s ethnicity, or a user'"'"'s profession.

31. The system of claim 29 wherein to collect user behavioral information, the at least one processor collects data indicative of at least one of:

user specific messaging rates, user specific page views, user specific login data.

32. The system of claim 24 wherein the at least one processor further converts at least one of:

the user profile information, the user behavioral information, or the messaging data to at least one of a binary or a numerical representation.

33. The system of claim 24 wherein the at least one processor further:

compares the response predictive model to other response predictive models; and

identifies at least one response predictive model for use based on the comparison.

34. The system of claim 24 wherein the at least one processor further:

uses the identified response predictive model; and

incorporates the identified response predictive model into a determination of potential candidates.

35. The system of claim 24 wherein the at least one processor further:

informs a new user of other users who have a high predicted response rate of responding to an initial message from the new user based at least in part on the responsive predictive model.

36. The system of claim 24 wherein the at least one processor further determines whether a global minimum has been found, and terminates generation of the response predictive model generation in response to determining that the global minimum has been found.

37. A system to enhance messaging between users, the system comprising:

a communication network that includes servers, processors, display devices, input devices, and system memory devices to send messages between users and autonomously collect actual historical messaging data;

at least one processor; and

at least one non-transitory processor-readable medium communicatively coupled to the least one processor, wherein the at least one processor;

forms an initial data set, the initial data set comprising user profile information for a plurality of users, user behavioral information for at least some of the plurality of users, and message data indicative of messaging activity between at least some of the plurality of users;

organizes the initial data set into a training data set and a test data set, wherein in order to organize the initial data set into a training data set and a test data set, the at least one processor randomly selects for inclusion in either the training data set or the test data set the user profile information, the user behavioral information, and the messaging data from all of the selected users on a message by message basis;

generates a response predictive model using a machine learning system from the training data set which includes the user profile information, the user behavioral information and message data;

evaluates the response predictive model generated from the training data set against the corresponding test data set, wherein at least some of the test data set is based on actual historical messaging data; and

determines the accuracy of the response prediction model using the actual historical messaging data.

38. The system of claim 37 wherein the test data set and the training data set are not mutually exclusive with respect to the users represented in each.

39. A system to enhance messaging between users, the system comprising:

a communication network that includes servers, processors, display devices, input devices, and system memory devices to send messages between users and autonomously collect actual historical messaging data;

at least one processor; and

at least one non-transitory processor-readable medium communicatively coupled to the least one processor, wherein the at least one processor;

forms an initial data set, the initial data set comprising user profile information for a plurality of users, user behavioral information for at least some of the plurality of users, and message data indicative of messaging activity between at least some of the plurality of users;

organizes the initial data set into a training data set and a test data set, wherein to organize the initial data set into a training data set and a test data set, the at least one processor further, for each of a first set of randomly selected users, includes all messages for each of the users in the first set of users in a training data set, and for a second set of users, different than the first set of users, includes all messages for each of the users in the second set in a test data set;

generates a response predictive model using a machine learning system from the training data set which includes the user profile information, the user behavioral information and message data;

evaluates the response predictive model generated from the training data set against the corresponding test data set, wherein at least some of the test data set is based on actual historical messaging data; and

determines the accuracy of the response prediction model using the actual historical messaging data.

40. The system of claim 39 wherein the first set of users and the second set of users are mutually exclusive with respect to the users represented in each of the sets of users.