DETECTING MESSAGES WITH OFFENSIVE CONTENT

• US 20160294755A1
• Filed: 06/20/2016
• Published: 10/06/2016
• Est. Priority Date: 06/14/2014
• Status: Active Grant

• Current Assignee: Trisha N. Prabhu
• Original Assignee: Trisha N. Prabhu
• Inventors: Prabhu, Trisha N.
• Granted Patent: US 10,250,538 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/063 Content adaptation, e.g. replacement of unsuitable content
• CPC Class Codes: H04L 51/212 using filtering or selective blocking
• CPC Class Codes: H04L 51/52 for supporting social networking services

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage medium, to facilitate interception of messages that include offensive content. In one aspect, a method includes actions of receiving input on a user device that includes message content, determining, on the user device, whether the message content includes offensive content, and in response to determining, on the user device, that the message content includes offensive content, generating an alert message for display on the user device that provides an indication that the message includes offensive content.

Claims

1. A method comprising:

receiving input on a user device that includes message content;

determining, on the user device, whether the message content includes offensive content; and

in response to determining, on the user device, that the message content includes offensive content, generating an alert message for display on the user device that provides an indication that the message includes offensive content.

2. The method of claim 1, wherein determining, on the user device, whether the message includes offensive content comprises:

obtaining at least a portion of the message content; and

determining whether the obtained message content includes an offensive trigger word.

3. The method of claim 1, wherein determining, on the user device, whether the message includes offensive content comprises:

obtaining at least a portion of the message content;

determining whether the obtained message content includes an offensive trigger word; and

in response to determining whether the obtained message includes an offensive trigger word, analyzing the context in which the offensive trigger word is used, wherein analyzing the context in which the offensive trigger word is used comprises;

determining whether the offensive trigger word falls within a predetermined proximity of (i) a pronoun that refers to a person other than the user of the user device or (ii) an entity name that refers to a person other than the user of the user device.

4. The method of claim 1, wherein determining, on the user device, whether the message includes offensive content comprises:

obtaining at least a portion of the message content;

processing the obtained message content by a neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items;

receiving a value from the neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items; and

determining, based on the value received from the neural network, whether the message content includes offensive content.

5. The method of claim 1, wherein the alert message includes one or more options for interacting with the message that includes offensive content.

6. The method of claim 5, wherein the one or more options for interacting with the message that includes offensive content includes (i) discarding the message that includes offensive content, (ii) sending the message that includes offensive content, and (iii) editing the message that includes offensive content.

7. The method of claim 6, further comprising:

receiving a selection, on the user device, of an option that corresponds to editing the message that includes offensive content;

receiving input, on the user device, that replaces an offensive portion of the message with a non-offensive portion; and

generating, on the user device, a positive keyword log entry that stores at least (i) the offensive portion of the message and (ii) the non-offensive portion of the message.

8. A user device comprising:

one or more processors and one or more storage devices storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising;

receiving input on the user device that includes message content;

determining, on the user device, whether the message content includes offensive content; and

in response to determining, on the user device, that the message content includes offensive content, generating an alert message for display on the user device that provides an indication that the message includes offensive content.

9. The user device of claim 8, wherein determining, on the user device, whether the message includes offensive content comprises:

obtaining at least a portion of the message content; and

determining whether the obtained message content includes an offensive trigger word.

10. The user device of claim 8, wherein determining, on the user device, whether the message includes offensive content comprises:

obtaining at least a portion of the message content;

determining whether the obtained message includes an offensive trigger word; and

in response to determining whether the obtained message includes an offensive trigger word, analyzing the context in which the offensive trigger word is used, wherein analyzing the context in which the offensive trigger word is used comprises;

determining whether the offensive trigger word falls within a predetermined proximity of (i) a pronoun that refers to a person other than the user of the user device or (ii) an entity name that refers to a person other than the user of the user device.

11. The user device of claim 8, wherein determining, on the user device, whether the message includes offensive content comprises:

obtaining at least a portion of the message content;

processing the obtained message content by a neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items;

receiving a value from the neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items; and

determining, based on the value received from the neural network, whether the message content includes offensive content.

12. The user device of claim 8, wherein the alert message includes one or more options for interacting with the message that includes offensive content.

13. The user device of claim 12, wherein the one or more options for interacting with the message that includes offensive content includes (i) discarding the message that includes offensive content, (ii) sending the message that includes offensive content, and (iii) editing the message that includes offensive content.

14. The user device of claim 13, the operations further comprising:

receiving a selection, on the user device, of an option that corresponds to editing the message that includes offensive content;

receiving input, on the user device, that replaces an offensive portion of the message with a non-offensive portion; and

generating, on the user device, a positive keyword log entry that stores at least (i) the offensive portion of the message and (ii) the non-offensive portion of the message.

15. A non-transitory computer-readable medium storing software comprising instructions executable by one or more processors which, upon such execution, cause the one or more processors to perform operations comprising:

receiving input on a user device that includes message content;

determining, on the user device, whether the message content includes offensive content; and

in response to determining, on the user device, that the message content includes offensive content, generating an alert message for display on the user device that provides an indication that the message includes offensive content.

16. The computer-readable medium of claim 15, wherein determining, on the user device, whether the message includes offensive content comprises:

obtaining at least a portion of the message content; and

determining whether the obtained message content includes an offensive trigger word.

17. The computer-readable medium of claim 15, wherein determining, on the user device, whether the message includes offensive content comprises:

obtaining at least a portion of the message content;

determining whether the obtained message includes an offensive trigger word; and

in response to determining whether the obtained message includes an offensive trigger word, analyzing the context in which the offensive trigger word is used, wherein analyzing the context in which the offensive trigger word is used comprises;

determining whether the offensive trigger word falls within a predetermined proximity of (i) a pronoun that refers to a person other than the user of the user device or (ii) an entity name that refers to a person other than the user of the user device.

18. The computer-readable medium of claim 15, wherein determining, on the user device, whether the message includes offensive content comprises:

obtaining at least a portion of the message content;

processing the obtained message content by a neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items;

receiving a value from the neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items; and

determining, based on the value received from the neural network, whether the message content includes offensive content.

19. The computer-readable medium of claim 15,

wherein the alert message includes one or more options for interacting with the message that includes offensive content,

wherein the one or more options for interacting with the message that includes offensive content includes (i) discarding the message that includes offensive content, (ii) sending the message that includes offensive content, and (iii) editing the message that includes offensive content.

20. The computer-readable medium of claim 19, the operations further comprising:

receiving a selection, on the user device, of an option that corresponds to editing the message that includes offensive content;

receiving input, on the user device, that replaces an offensive portion of the message with a non-offensive portion; and

generating, on the user device, a positive keyword log entry that stores at least (i) the offensive portion of the message and (ii) the non-offensive portion of the message.