System and method for detecting unwanted content

• US 9,258,261 B1
• Filed: 11/17/2014
• Issued: 02/09/2016
• Est. Priority Date: 10/09/2012
• Status: Active Grant

• Current Assignee: WhatsApp LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Original Assignee: WhatsApp LLC (Meta Platforms, Inc. (f/k/a Facebook, Inc.))
• Inventors: O'Connor, Bryan D.
• Primary Examiner(s): NANO, SARGON N
• Application Number:
  US14/543,612
• Time in Patent Office: 449 Days
• US Class Current: 1/1
• CPC Class Codes: G06Q 10/107 Computer-aided management of electronic mailing [e-mailing]
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 43/16 Threshold monitoring
• CPC Class Codes: H04L 51/212 using filtering or selective blocking
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/535 Tracking the activity of the user network monitoring arrangements H04L43/00; recording of computer activity G06F11/34

Abstract

A system and method for detecting unwanted electronic content, such as spam. As a user operates an application to send messages, several metrics are tracked to allow the system to analyze her activity. Illustrative metrics may include, but are not limited to, block count (e.g., how many other users have blocked her), block rate (e.g., the rate at which other users block her), peer symmetry (e.g., percentage of her messages that are to other users that have her in their address book), message uniqueness (e.g., how unique her messages are from each other), peer uniqueness (e.g., how unique the addressees of her messages are), and message rate (e.g., the rate at which she sends messages). Periodically, metrics may be compared to corresponding thresholds. Depending on whether a threshold is crossed, and which threshold is crossed, she may be banned from using the application or placed on a watch list.

Claims

1. A method comprising:

receiving, from a first user of a communication application, multiple electronic messages addressed to other users of the communication application;

calculating a message rate at which the first user sent the multiple electronic messages;

determining a total block count of the first user, wherein the total block count identifies a total number of users of the communication application that have blocked the first user;

determining a number of temporary subscriptions of the first user during a current session of use of the communication application;

determining, based on the message rate, the total block count, and the number of temporary subscriptions, whether the electronic messages are likely to include unwanted content; and

based on the determining of whether the electronic messages are likely to include unwanted content, automatically creating a temporary subscription between the first user and another user when the first user and the other user exchange messages during the current session, and terminating the temporary subscription automatically when the current session ends.

2. The method of claim 1, further comprising:

comparing each of the message rate, the total block count, and the number of temporary subscriptions to corresponding thresholds.

3. The method of claim 2, further comprising, if any of the corresponding thresholds are exceeded:

calculating a message uniqueness metric indicating uniqueness of electronic messages sent by the first user.

4. The method of claim 3, wherein calculating the message uniqueness metric comprises, during a finite time period in which a plurality of additional electronic messages is received from the first user:

comparing each additional electronic message to previous electronic messages received from the first user to determine if the additional electronic message is unique; and

calculating a ratio of the number of unique additional electronic messages to the number of additional electronic messages.

5. The method of claim 2, further comprising, if any of the corresponding thresholds are exceeded:

calculating a peer uniqueness metric indicating uniqueness of addressees of electronic messages sent by the first user.

6. The method of claim 5, wherein calculating the peer uniqueness metric comprises, during a finite time period in which a plurality of additional electronic messages is received from the first user:

comparing addressees of each additional electronic message to addressees of previous electronic messages of the first user to determine if the addressees of the additional electronic message are unique; and

calculating a ratio of the number of unique addressees of the additional electronic messages to the number of additional electronic messages.

7. The method of claim 2, further comprising, if any of the corresponding thresholds are exceeded:

examining the temporary subscriptions of the first user for peer symmetry.

8. The method of claim 7, wherein examining the temporary subscriptions of the first user for peer symmetry comprises, for each temporary subscription between the first user and a second user:

determining whether the second user is included in a first collection of contacts of the first user; and

determining whether the first user is included in a second collection of contacts of the second user.

9. The method of claim 1, further comprising:

if the message rate exceeds a first message rate threshold, preventing the user from sending additional electronic messages; and

if the message rate exceeds a second message rate threshold, but not the first message rate threshold, adding the first user to a watch list.

10. The method of claim 1, further comprising:

preventing one or more additional messages sent by the first user from being delivered to addressees of the additional messages if the total block count exceeds a first block count threshold; and

adding the first user to a watch list if the total block count exceeds a second block count threshold but not the first block count threshold.

11. The method of claim 10, further comprising:

calculating a block rate identifying a rate at which the first user is blocked by other users of the communication application; and

adding the first user to the watch list if the block rate exceeds a block rate threshold.

12. A non-transitory computer-readable medium storing instructions that, when executed by a processor, cause the processor to perform a method comprising:

receiving, from a first user of a communication application, multiple electronic messages addressed to other users of the communication application;

calculating a message rate at which the first user sent the multiple electronic messages;

determining a total block count of the first user, wherein the total block count identifies a total number of users of the communication application that have blocked the first user;

determining a number of temporary subscriptions of the first user during a current session of use of the communication application;

determining, based on the message rate, the total block count, and the number of temporary subscriptions, whether the electronic messages are likely to include unwanted content; and

based on the determining of whether the electronic messages are likely to include unwanted content, automatically creating a temporary subscription between the first user and another user when the first user and the other user exchange messages, and terminating the temporary subscription automatically when the current session ends.

13. An apparatus comprising:

one or more processors; and

a non-transitory computer-readable medium storing instructions that, when executed by the one or more processors, cause the apparatus to;

receive, from a first user of a communication application, multiple electronic messages addressed to other users of the communication application;

calculate a message rate at which the first user sent the multiple electronic messages;

determine a total block count of the first user, wherein the total block count identifies a total number of users of the communication application that have blocked the first user;

determine a number of temporary subscriptions of the first user during a current session of use of the communication application;

determining, based on the message rate, the total block count, and the number of temporary subscriptions, whether the electronic messages are likely to include unwanted content; and

based on the determining of whether the electronic messages are likely to include unwanted content, automatically creating a temporary subscription between the first user and another user when the first user and the other user exchange messages during the current session, and terminating the temporary subscription automatically when the current session ends.

14. The apparatus of claim 13, the non-transitory computer-readable medium further storing instructions that, when executed by the one or more processors, cause the apparatus to:

compare each of the message rate, the total block count, and the number of temporary subscriptions to corresponding thresholds.

15. The apparatus of claim 14, the non-transitory computer-readable medium further storing instructions that, when executed by the one or more processors, cause the apparatus to, if any of the corresponding thresholds are exceeded:

calculate a message uniqueness metric indicating uniqueness of electronic messages sent by the first user.

16. The apparatus of claim 15, wherein calculating the message uniqueness metric comprises, during a finite time period in which a plurality of additional electronic messages is received from the first user:

comparing each additional electronic message to previous electronic messages received from the first user to determine if the additional electronic message is unique; and

calculating a ratio of the number of unique additional electronic messages to the number of additional electronic messages.

17. The apparatus of claim 14, the non-transitory computer-readable medium further storing instructions that, when executed by the one or more processors, cause the apparatus to, if any of the corresponding thresholds are exceeded:

calculate a peer uniqueness metric indicating uniqueness of addressees of electronic messages sent by the first user.

18. The apparatus of claim 17, wherein calculating the peer uniqueness metric comprises, during a finite time period in which a plurality of additional electronic messages is received from the first user:

comparing addressees of each additional electronic message to addressees of previous electronic messages of the first user to determine if the addressees of the additional electronic message are unique; and

calculating a ratio of the number of unique addressees of the additional electronic messages to the number of additional electronic messages.

19. The apparatus of claim 14, the non-transitory computer-readable medium further storing instructions that, when executed by the one or more processors, cause the apparatus to, if any of the corresponding thresholds are exceeded:

examine the temporary subscriptions of the first user for peer symmetry.

20. The apparatus of claim 19, wherein examining the temporary subscriptions of the first user for peer symmetry comprises, for each temporary subscription between the first user and a second user:

determining whether the second user is included in a first collection of contacts of the first user; and

determining whether the first user is included in a second collection of contacts of the second user.

21. The apparatus of claim 13, the non-transitory computer-readable medium further storing instructions that, when executed by the one or more processors, cause the apparatus to:

if the message rate exceeds a first message rate threshold, prevent the user from sending additional electronic messages; and

if the message rate exceeds a second message rate threshold, but not the first message rate threshold, add the first user to a watch list.

22. The apparatus of claim 13, the non-transitory computer-readable medium further storing instructions that, when executed by the one or more processors, cause the apparatus to:

prevent one or more additional messages sent by the first user from being delivered to addressees of the additional messages if the total block count exceeds a first block count threshold; and

add the first user to a watch list if the total block count exceeds a second block count threshold but not the first block count threshold.

23. The apparatus of claim 22, the non-transitory computer-readable medium further storing instructions that, when executed by the one or more processors, cause the apparatus to:

calculate a block rate identifying a rate at which the first user is blocked by other users of the communication application; and

add the first user to the watch list if the block rate exceeds a block rate threshold.