Detecting messages with offensive content

• US 10,250,538 B2
• Filed: 06/20/2016
• Issued: 04/02/2019
• Est. Priority Date: 06/14/2014
• Status: Active Grant

• Current Assignee: Trisha N. Prabhu
• Original Assignee: Trisha N. Prabhu
• Inventors: Prabhu, Trisha N.
• Primary Examiner(s): Khan, Aftab N.
• Application Number:
  US15/187,674
• Publication Number:
  US 20160294755A1
• Time in Patent Office: 1,016 Days
• CPC Class Codes: G06F 3/04817 using icons graphical or visual programming using iconic symbols G06F8/34
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: H04L 51/046 Interoperability with other network applications or services
• CPC Class Codes: H04L 51/063 Content adaptation, e.g. replacement of unsuitable content
• CPC Class Codes: H04L 51/212 using filtering or selective blocking
• CPC Class Codes: H04L 51/52 for supporting social networking services

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage medium, to facilitate interception of messages that include offensive content. In one aspect, a method includes actions of receiving input on a user device that includes message content, determining, on the user device, whether the message content includes offensive content, and in response to determining, on the user device, that the message content includes offensive content, generating an alert message for display on the user device that provides an indication that the message includes offensive content.

Claims

1. A method comprising:

receiving input that includes message content using a virtual keyboard that is generated for output on a display of a user device, the virtual keyboard being generated by an application program installed on the user device that includes multiple software modules;

determining, by the user device and using one or more of the software modules of the application program, whether the message content includes offensive content, wherein determining, by the user device and using one or more of the software modules of the application program comprises;

determining whether the received message content includes an offensive trigger word;

in response to determining that the received message content includes an offensive trigger word, analyzing the context in which the offensive trigger word is used in the message content, wherein analyzing the context in which the offensive trigger word is used comprises;

determining whether the offensive trigger word falls within a predetermined proximity of (i) a pronoun that refers to a person other than the user of the user device or (ii) an entity name that refers to a person other than the user of the user device; and

in response to determining, by the user device and using one or more of the software modules of the application program, that the received message content includes an offensive trigger word that falls within a predetermined proximity of (i) a pronoun that refers to a person other than the user of the user device or (ii) an entity name that refers to a person other than the user of the user device, generating an alert message for display on the user device that provides an indication that the received message content includes offensive content.

3. The method of claim 1, wherein the alert message includes one or more options for interacting with the message that includes offensive content.

4. The method of claim 3, wherein the one or more options for interacting with the message that includes offensive content includes (i) discarding the message that includes offensive content, (ii) sending the message that includes offensive content, and (iii) editing the message that includes offensive content.

5. The method of claim 4, further comprising:

receiving a selection, on the user device, of an option that corresponds to editing the message that includes offensive content;

receiving input, on the user device, that replaces an offensive portion of the message with a non-offensive portion; and

generating, on the user device, a positive keyword log entry that stores at least (i) the offensive portion of the message and (ii) the non-offensive portion of the message.

2. A method comprising:

receiving input that includes message content using a virtual keyboard that is generated for output on a display of a user device, the virtual keyboard being generated by an application program installed on the user device that includes multiple software modules;

determining, by the user device and using one or more of the software modules of the application program, whether the message content includes offensive content, wherein determining, by the user device and using the one or more of the software modules of the application program, whether the message includes offensive content comprises;

providing at least a portion of the received message content to a neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items;

receiving a value that was generated by the neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items, wherein the value is generated by the neural network based on the neural network'"'"'s processing of the provided portion of the received message content; and

determining, based on the value generated by the neural network, whether the message content includes offensive content; and

in response to determining, by the user device and using one or more of the software modules of the application program and based on the value generated by the neural network, that the received message content includes offensive content, generating an alert message for display on the user device that provides an indication that the received message content includes offensive content.

6. A user device comprising:

one or more processors and one or more storage devices storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising;

receiving input on a user device that includes message content;

determining, by the user device, whether the received message content includes offensive content, wherein determining whether the received message content includes offensive content comprises;

determining whether the received message content includes an offensive trigger word;

in response to determining that the received message includes an offensive trigger word, analyzing the context in which the offensive trigger word is used, wherein analyzing the context in which the offensive trigger word is used comprises;

determining whether the offensive trigger word falls within a predetermined proximity of (i) a pronoun that refers to a person other than the user of the user device or (ii) an entity name that refers to a person other than the user of the user device; and

in response to determining, by the user device, that the received message content includes an offensive trigger word that falls within a predetermined proximity of (i) a pronoun that refers to a person other than the user of the user device or (ii) an entity name that refers to a person other than the user of the user device, generating an alert message for display on the user device that provides an indication that the received message content includes offensive content.

7. The user device of claim 6, wherein the alert message includes one or more options for interacting with the message that includes offensive content.

8. The user device of claim 7, wherein the one or more options for interacting with the message that includes offensive content includes (i) discarding the message that includes offensive content, (ii) sending the message that includes offensive content, and (iii) editing the message that includes offensive content.

9. The user device of claim 8, the operations further comprising:

receiving a selection, on the user device, of an option that corresponds to editing the message that includes offensive content;

receiving input, on the user device, that replaces an offensive portion of the message with a non-offensive portion; and

generating, on the user device, a positive keyword log entry that stores at least (i) the offensive portion of the message and (ii) the non-offensive portion of the message.

10. A non-transitory computer-readable medium storing software comprising instructions executable by one or more processors which, upon such execution, cause the one or more processors to perform operations comprising:

receiving input on a user device that includes message content;

determining, by the user device, whether the received message content includes offensive content, wherein determining whether the received message content includes offensive content comprises;

determining whether the received message content includes an offensive trigger word;

in response to determining that the received message includes an offensive trigger word, analyzing the context in which the offensive trigger word is used, wherein analyzing the context in which the offensive trigger word is used comprises;

determining whether the offensive trigger word falls within a predetermined proximity of (i) a pronoun that refers to a person other than the user of the user device or (ii) an entity name that refers to a person other than the user of the user device; and

in response to determining, by the user device, that the received message content includes an offensive trigger word that falls within a predetermined proximity of (i) a pronoun that refers to a person other than the user of the user device or (ii) an entity name that refers to a person other than the user of the user device, generating an alert message for display on the user device that provides an indication that the received message content includes offensive content.

11. The computer-readable medium of claim 10,

wherein the alert message includes one or more options for interacting with the message that includes offensive content,

wherein the one or more options for interacting with the message that includes offensive content includes (i) discarding the message that includes offensive content, (ii) sending the message that includes offensive content, and (iii) editing the message that includes offensive content.

12. The computer-readable medium of claim 11, the operations further comprising:

receiving a selection, on the user device, of an option that corresponds to editing the message that includes offensive content;

receiving input, on the user device, that replaces an offensive portion of the message with a non-offensive portion; and

generating, on the user device, a positive keyword log entry that stores at least (i) the offensive portion of the message and (ii) the non-offensive portion of the message.

13. A user device comprising:

one or more processors and one or more storage devices storing instructions that are operable, when executed by the one or more processors, to cause the one or more processors to perform operations comprising;

receiving input on a user device that includes message content;

determining, by the user device, whether the received message content includes offensive content, wherein determining whether the received message content includes offensive content comprises;

providing at least a portion of the received message content to a neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items;

receiving a value that was generated by the neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items, wherein the value is generated by the neural network based on the neural network'"'"'s processing of the provided portion of the received message content; and

determining, based on the value generated by the neural network, whether the message content includes offensive content; and

in response to determining, by the user device and based on the value generated by the neural network, that the received message content includes offensive content, generating an alert message for display on the user device that provides an indication that the received message content includes offensive content.

14. A non-transitory computer-readable medium storing software comprising instructions executable by one or more processors which, upon such execution, cause the one or more processors to perform operations comprising:

receiving input on a user device that includes message content;

determining, by the user device, whether the received message content includes offensive content, wherein determining whether the received message content includes offensive content comprises;

providing at least a portion of the received message content to a neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items;

receiving a value that was generated by the neural network that has been trained using multiple content items that have been labeled as offensive content items and multiple content items that have been labeled as non-offensive content items, wherein the value is generated by the neural network based on the neural network'"'"'s processing of the provided portion of the received message content; and

determining, based on the value generated by the neural network, whether the message content includes offensive content; and

in response to determining, by the user device and based on the value generated by the neural network, that the received message content includes offensive content, generating an alert message for display on the user device that provides an indication that the received message content includes offensive content.