System and method for detecting unwanted content

• US 9,455,941 B1
• Filed: 11/17/2014
• Issued: 09/27/2016
• Est. Priority Date: 10/09/2012
• Status: Active Grant

• Current Assignee: WhatsApp LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: WhatsApp LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: O'Connor, Bryan D.
• Primary Examiner(s): Nano, Sargon
• Application Number:
  US14/543,589
• Time in Patent Office: 680 Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 51/212 using filtering or selective blocking
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

A system and method for detecting unwanted electronic content, such as spam. As a user operates an application to send messages, several metrics are tracked to allow the system to analyze her activity. Illustrative metrics may include, but are not limited to, block count (e.g., how many other users have blocked her), block rate (e.g., the rate at which other users block her), peer symmetry (e.g., percentage of her messages that are to other users that have her in their address book), message uniqueness (e.g., how unique her messages are from each other), peer uniqueness (e.g., how unique the addressees of her messages are), and message rate (e.g., the rate at which she sends messages). Periodically, metrics may be compared to corresponding thresholds. Depending on whether a threshold is crossed, and which threshold is crossed, she may be banned from using the application or placed on a watch list.

Claims

1. A system comprising:

one or more application servers hosting an application for exchanging messages between users of the application;

one or more processors;

message rate logic executable by the one or more processors to track a message rate at which a first user sends messages;

block rate logic executable by the one or more processors to track a block rate at which the first user is blocked by other users;

message uniqueness logic executable by the one or more processors to track the message uniqueness of the messages sent by the first user;

peer uniqueness logic executable by the one or more processors to track the message addressee uniqueness of the messages sent by the first user;

peer symmetry logic executable by the one or more processors to track a percentage of the messages sent by the first user that are addressed to other users that are (a) within a first address book of the first user and that (b) have other address books that include the first user;

unwanted content logic executable by the one or more processors to determine whether two or more of the message rate, the block rate, the message uniqueness, or the percentage of the messages tracked by the peer symmetry logic indicates a likelihood that the first user is sending unwanted content; and

watch list logic that, when executed by the one or more processors, causes the one or more processors to add the first user to a watch list of suspicious behavior when the wanted content logic determines that the first user is likely sending unwanted content.

2. The system of claim 1, further comprising a storage device configured to store:

a total block count representing how many users other than the first user have blocked the first user;

address books of all users of the application, including the first address book and the other address books;

a first subscription list identifying users to whom the first user is subscribed; and

a second subscription list identifying users that have subscribed to the first user.

3. The system of claim 1, wherein the message uniqueness logic includes instructions that, when executed by the one or more processors, cause the system to:

store snapshots of multiple messages sent by the first user; and

upon submission of a new message by the first user, compare a portion of the new message to the snapshots.

4. The system of claim 1, further comprising:

block count logic executable by the one or more processors to track a number of other users that have blocked the first user.

5. The system of claim 1, further comprising:

subscription logic executable by the one or more processors to monitor subscriptions among users of the application.

6. The system of claim 5, wherein the subscriptions include:

permanent subscriptions of one user to another user, whereby the one user is advised of the other user'"'"'s activity with the application; and

temporary subscriptions between pairs of users while those users are both online with the application and communicating with each other;

wherein creation of a permanent subscription requires explicit approval of the one user and the other user; and

creation of a temporary subscription is automatic.

7. The system of claim 1, further comprising:

group manipulation logic executable by the one or more processors to monitor the first user'"'"'s manipulation of groups of users of the application.

8. The system of claim 7, wherein the group manipulation logic comprises instructions that, when executed by the one or more processors, cause the system to:

track a group creation rate at which the first user creates new groups;

track a total number of groups created by the first user within a time period; and

track, for each group created by the first user, a length of time the user remains a member of the group.

9. The system of claim 8, wherein the group manipulation logic further comprises instructions that, when executed by the one or more processors, cause the system to:

track, for each group created by the first user, a per-group symmetry metric indicating a percentage of other members of the group that have address books that include the first user.

10. The system of claim 8, wherein the group manipulation logic further comprises instructions that, when executed by the one or more processors, cause the system to:

track, among all groups created by the first user, a total group symmetry metric indicating a percentage of other members of the group that have address books that include the first user.

11. The system of claim 1, further comprising:

pattern detection logic executable by the one or more processors to identify, within messaging of the first user, a pattern of activity indicative of distribution of unwanted content.

12. The system of claim 11, wherein a first pattern of activity indicative of distribution of unwanted content comprises use of a predictable sequence of addresses during the messaging.

13. The system of claim 11, wherein:

addresses of messages sent by the first user comprise telephone numbers;

a first pattern of activity indicative of distribution of unwanted content comprises use of a predictable sequence of telephone numbers during the messaging;

a second pattern of activity indicative of distribution of unwanted content comprises using only telephone numbers having a first common telephone code; and

a third pattern of activity indicative of distribution of unwanted content comprises avoiding telephone numbers having a second common telephone code.

14. A method comprising:

receiving from a first user an electronic message having one or more addressees;

updating a message rate metric identifying a rate at which the first user is sending electronic messages;

updating a message uniqueness metric identifying uniqueness of messages sent by the first user;

updating a peer uniqueness metric identifying uniqueness of addressees of messages sent by the first user;

for each addressee in the one or more addressees, updating a peer symmetry metric identifying a percentage of the messages sent by the first user that are addressed to other users that (a) are within a first address book of the first user and (b) have other address books that include the first user;

comparing each of the updated metrics to one or more threshold values;

determining that two or more of the updated metrics indicates a likelihood that the first user is sending unwanted content when a respective updated metric exceeds at least one of the one or more threshold values; and

adding the first user to a watch list of suspicious behavior when it is determined that the first user is likely sending unwanted content.

15. The method of claim 14, further comprising:

if the updated message rate metric exceeds a first message rate threshold, preventing the user from sending additional electronic messages; and

if the updated message rate metrics exceeds a second message rate threshold, but not the first message rate threshold, placing the first user in a watch list.

16. The method of claim 14, further comprising:

based on said comparisons, banning the first user from operating an application used generate the electronic message.

17. The method of claim 14, further comprising:

based on said comparisons, placing the first user on a watch list;

wherein, as a consequence of being on the watch list, additional metrics are tracked for the first user.

18. The method of claim 14, wherein the additional metrics include one or more of:

a block count metric identifying a number of users that have blocked the first user;

a subscription metric identifying subscriptions between the first user and other users; and

one or more group metrics reflecting the first user'"'"'s manipulation of groups of users.

19. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method comprising:

receiving from a first user an electronic message having one or more addressees;

updating a message rate metric identifying a rate at which the first user is sending electronic messages;

updating a message uniqueness metric identifying uniqueness of messages sent by the first user;

updating a peer uniqueness metric identifying uniqueness of addressees of messages sent by the first user;

for each addressee in the one or more addressees, updating a peer symmetry metric to reflect whether the first user is included in an address book of the addressee;

comparing each of the updated metrics to one or more threshold values;

determining that two or more of the updated metrics indicates a likelihood that the first user is sending unwanted content when a respective updated metric exceeds at least one of the one or more threshold values; and

adding the first user to a watch list of suspicious behavior when it is determined that the first user is likely sending unwanted content.