Comprehensive safety schema for ensuring appropriateness of language in online chat

• US 10,303,762 B2
• Filed: 04/30/2013
• Issued: 05/28/2019
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Original Assignee: Disney Enterprises Incorporated (The Walt Disney Company)
• Inventors: Markman, Vita, O'Dell, Sean, Beechum, Drew, Pak, Paul
• Primary Examiner(s): Spooner, Lamont M
• Assistant Examiner(s): Ogunbiyi, Oluwadamilola M
• Application Number:
  US13/873,455
• Publication Number:
  US 20140278367A1
• Time in Patent Office: 2,219 Days
• Field of Search: 704 9
• CPC Class Codes: G06F 40/253 Grammatical analysis; Style critique
• CPC Class Codes: G06F 40/279 Recognition of textual entities
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

A method is disclosed for evaluating a chat message sent between users of an online environment. The method may include associating each word in the chat message with metadata. The metadata identifies a word type and usage for each word in the chat message. This method may also include identifying one or more safety rules associated with the metadata. Each safety rule identifies an ordered sequence of one or more sets of words. This method may also include applying the safety rule to the chat message to determine whether a sequence of words in the chat message present in the ordered sequence of sets of words. Upon determining a word, from each set of words in the ordered sequence of sets of words, matches a respective one of the words in the chat message, the chat message is blocked from being sent to a message recipient.

Claims

1. A computer-implemented method to evaluate a chat message requested to be sent between users of an online game environment having an associated program, the computer-implemented method comprising:

receiving, from a sender in the online game environment, a request to send the chat message in real-time to a specified recipient in the online game environment, wherein the program is configured to;

approve the chat message to be sent to the specified recipient, upon determining that the chat message matches any of a set of preapproved phrases; and

otherwise, deny the chat message from being sent to the specified recipient, upon determining that the chat message contains any prohibited words or any prohibited combinations of words; and

upon determining that the chat message does not match any of the set of preapproved phrases, does not contain any prohibited words, and does not contain any prohibited combinations of words, further analyzing the chat message for approval in real-time, comprising;

denying the chat message from being sent to the specified recipient, upon determining, in real-time by one or more computer processors when executing the program, that the chat message contains a prohibited metaphorical meaning or contains phonetically prohibited language, despite absence of prohibited words and prohibited combinations of words in the chat message;

wherein the program is further configured to approve the chat message to be sent to the specified recipient, upon determining that the chat message does not contain any prohibited metaphorical meaning and does not contain any phonetically prohibited language, whereafter the approved chat message is sent to the specified recipient.

2. The computer-implemented method of claim 1, wherein determining whether the chat message contains phonetically prohibited language comprises:

generating bigrams for each word in the chat message;

replacing the bigrams with phonetic keys to generate a sequence of phonetic keys;

determining whether the sequence of phonetic keys matches a sequence in a literal safety rule; and

upon determining the phonetic keys were identified in the literal safety rule, blocking the chat message from being sent to the specified recipient.

3. The computer-implemented method of claim 1, further comprising:

monitoring each word in the chat message, for a change in a frequency of usage in chat message sent between users of the online game environment;

determining whether the change in usage demonstrates an increase in usage or a new usage; and

responsive to the change in usage exceeding a threshold, generating a report for the word.

4. The computer-implemented method of claim 1, wherein determining whether the chat message contains any prohibited metaphorical meaning includes identifying one or more safety rules, wherein each safety rule identifies an ordered sequence of one or more sets of word types, wherein the request is received by an application configured to:

upon determining at least a first word from one of the sets of words does not match a respective word in the chat message, approving the message to be sent to the specified recipient.

5. The computer-implemented method of claim 1, wherein determining whether the chat message contains any prohibited metaphorical meaning includes identifying one or more safety rules, at least one safety rule of which identifies combinations of words in the chat message which have a secondary semantic meaning.

6. The computer-implemented method of claim 1, wherein determining whether the chat message contains any prohibited metaphorical meaning includes identifying one or more safety rules, at least one safety rule of which includes sub rules to stop applying the at least one safety rule to the chat message and permitting a number of words to be within a sequence of words in the chat message.

7. The computer-implemented method of claim 1, wherein the chat message comprises a first chat message, wherein a plurality of messages is received from the sender to be sent in real-time to the recipient, the plurality of messages including the first chat message, a second chat message, a third chat message, a fourth chat message, a fifth chat message, and a sixth chat message, wherein the first, second, third, fourth, fifth, and sixth chat messages comprise distinct chat messages;

wherein determining that the first chat message contains a prohibited metaphorical meaning or contains phonetically prohibited language comprises determining that the first chat message contains a prohibited metaphorical meaning, wherein the computer-implemented method further comprises;

upon determining that the second chat message matches any of the set of preapproved phrases, approving the second chat message to be sent to the specified recipient, whereafter the second chat message is sent to the specified recipient;

upon determining that the third chat message does not match any of the set of preapproved phrases and contains any prohibited words, denying the third chat message from being sent to the specified recipient; and

upon determining that the fourth chat message does not match any of the set of preapproved phrases and contains any prohibited combinations of words, denying the fourth chat message from being sent to the specified recipient.

8. The computer-implemented method of claim 7, wherein the computer-implemented method further comprises:

upon determining that the fifth chat message does not match any of the set of preapproved phrases, does not contain any prohibited words, does not contain any prohibited combinations of words, and contains phonetically prohibited language, denying the fifth chat message from being sent to the specified recipient; and

upon determining that the sixth chat message does not match any of the set of preapproved phrases, does not contain any prohibited words, does not contain any prohibited combinations of words, does not contain any prohibited metaphorical meaning, and does not contain any phonetically prohibited language, approving the sixth chat message to be sent to the specified recipient, whereafter the sixth chat message is sent to the specified recipient.

9. The computer-implemented method of claim 8, wherein the program includes a plurality of components including a static chat classification engine, a dynamic chat classification engine, and a safety schema component;

wherein the first chat message is determined not to match any of the set of preapproved phrases by the static chat classification engine;

wherein the fifth chat message is determined not to contain any prohibited words or prohibited combinations of words by the dynamic chat classification engine;

wherein the first chat message is determined to contain a prohibited metaphorical meaning by the safety schema component;

wherein the fifth chat message is determined to contain phonetically prohibited language by the safety schema component.

10. The computer-implemented method of claim 9, wherein the safety schema component includes a plurality of subcomponents including a semantic component, a phonetic component, and a trending component;

wherein the semantic component is configured to detect whether the first chat message contains any prohibited metaphorical meaning;

wherein the phonetic component is configured to detect whether the fifth chat message contains any phonetically prohibited language;

wherein the trending component is configured to monitor usage trends of words and phrases in the online game environment in order to identify new words or combinations of words to approve for sending or deny from being sent.

11. The computer-implemented method of claim 10, wherein determining whether the first chat message contains any prohibited metaphorical meaning comprises:

identifying word types and usages including a word type and usage for each word in the first chat message;

identifying one or more safety rules based on the word type and usage, wherein each safety rule identifies an ordered sequence of one or more sets of word types;

applying the one or more safety rules to the first chat message in order to determine whether a sequence of words in the word types of the first chat message matches a first one of the ordered sequence of one or more sets of word types in the one or more safety rules; and

determining that a word type, from the first set of word types in the ordered sequence of one or more sets of word types, matches a respective one of the word types of the first chat message.

12. The computer-implemented method of claim 11, wherein determining whether the fifth chat message contains phonetically prohibited language comprises:

generating bigrams for each word in the fifth chat message;

replacing the bigrams with phonetic keys to generate a sequence of phonetic keys;

determining whether the sequence of phonetic keys matches a sequence in a literal safety rule; and

upon determining the phonetic keys were identified in the literal safety rule, blocking the fifth chat message from being sent to the specified recipient;

wherein monitoring usage trends of words and phrases in the online game environment comprises;

monitoring each word in the first chat message, for a change in a frequency of usage in first chat message sent between users of the online game environment;

determining whether the change in usage demonstrates an increase in usage or a new usage; and

responsive to the change in usage exceeding a threshold, generating a report for the word.

13. The computer-implemented method of claim 12, further comprising identifying a part of speech and a safe category for each word, wherein the safe category is selected from a plurality of predefined safe categories including sexual innuendo, violence, bullying, personally identifying information, substance abuse, and cheating, wherein the safe category comprises, in respective instances, each category of the plurality of predefined safe categories;

wherein the safety rule identifies combinations of words in the first chat message which have a secondary semantic meaning that is prohibited;

wherein the safety rule includes sub rules to stop applying the safety rule to the first chat message and permitting a number of words to be within the sequence of words in the first chat message.

14. A non-transitory computer-readable storage medium storing a program executable to perform an operation to evaluate a chat message requested to be sent between users of an online game environment, the operation comprising:

receiving, from a sender in the online game environment, a request to send the chat message in real-time to a specified recipient in the online game environment, wherein the program is configured to;

approve the chat message to be sent to the specified recipient, upon determining that the chat message matches any of a set of preapproved phrases; and

otherwise, deny the chat message from being sent to the specified recipient, upon determining that the chat message contains any prohibited words or any prohibited combinations of words; and

upon determining that the chat message does not match any of the set of preapproved phrases, does not contain any prohibited words, and does not contain any prohibited combinations of words, further analyzing the chat message for approval in real-time, comprising;

denying the chat message from being sent to the specified recipient, upon determining, in real-time by one or more computer processors when executing the program, that the chat message contains a prohibited metaphorical meaning or contains phonetically prohibited language, despite absence of prohibited words and prohibited combinations of words in the chat message;

wherein the program is further configured to approve the chat message to be sent to the specified recipient, upon determining that the chat message does not contain any prohibited metaphorical meaning and does not contain any phonetically prohibited language, whereafter the approved chat message is sent to the specified recipient.

15. The non-transitory computer-readable storage medium of claim 14, wherein determining whether the chat message contains phonetically prohibited language comprises:

generating bigrams for each word in the chat message;

replacing the bigrams with phonetic keys to generate a sequence of phonetic keys;

determining whether the sequence of phonetic keys matches a sequence in a literal safety rule; and

upon determining the phonetic keys were identified in the literal safety rule, blocking the chat message from being sent to the specified recipient.

16. The non-transitory computer-readable storage medium of claim 14, wherein the operation further comprises:

monitoring each word in the chat message, for a change in a frequency of usage in chat message sent between users of the online game environment;

determining whether the change in usage demonstrates an increase in usage or a new usage; and

responsive to the change in usage exceeding a threshold, generating a report for the word.

17. The non-transitory computer-readable storage medium of claim 14, wherein determining whether the chat message contains any prohibited metaphorical meaning includes identifying one or more safety rules, wherein each safety rule identifies an ordered sequence of one or more sets of word types, wherein the request is received by an application configured to:

upon determining at least a first word from one of the sets of words does not match a respective word in the chat message, approving the message to be sent to the specified recipient.

18. The non-transitory computer-readable storage medium of claim 14, wherein determining whether the chat message contains any prohibited metaphorical meaning includes identifying one or more safety rules, at least one safety rule of which identifies combinations of words in the chat message which have a secondary semantic meaning.

19. The non-transitory computer-readable storage medium of claim 14, wherein determining whether the chat message contains any prohibited metaphorical meaning includes identifying one or more safety rules, at least one safety rule of which includes sub rules to stop applying the at least one safety rule to the chat message and permitting a number of words to be within a sequence of words in the chat message.

20. A system to evaluate a chat message requested to be sent between users of an online game environment, the system comprising:

one or more computer processors; and

a memory containing a program which, when executed by the one or more computer processors, performs an operation comprising;

receiving, from a sender in the online game environment, a request to send the chat message in real-time to a specified recipient in the online game environment, wherein the program is configured to;

approve the chat message to be sent to the specified recipient, upon determining that the chat message matches any of a set of preapproved phrases; and

otherwise, deny the chat message from being sent to the specified recipient, upon determining that the chat message contains any prohibited words or any prohibited combinations of words; and

upon determining that the chat message does not match any of the set of preapproved phrases, does not contain any prohibited words, and does not contain any prohibited combinations of words, further analyzing the chat message for approval in real-time, comprising;

denying the chat message from being sent to the specified recipient, upon determining, in real-time, that the chat message contains a prohibited metaphorical meaning or contains phonetically prohibited language, despite absence of prohibited words and prohibited combinations of words in the chat message;

wherein the program is further configured to approve the chat message to be sent to the specified recipient, upon determining that the chat message does not contain any prohibited metaphorical meaning and does not contain any phonetically prohibited language, whereafter the approved chat message is sent to the specified recipient.

21. The system of claim 20, wherein determining whether the chat message contains phonetically prohibited language comprises:

generating bigrams for each word in the chat message;

replacing the bigrams with phonetic keys to generate a sequence of phonetic keys;

determining whether the sequence of phonetic keys matches a sequence in a literal safety rule; and

upon determining the phonetic keys were identified in the literal safety rule, blocking the chat message from being sent to the specified recipient.

22. The system of claim 20, wherein the operation further comprises:

monitoring each word in the chat message, for a change in a frequency of usage in chat message sent between users of the online game environment;

determining whether the change in usage demonstrates an increase in usage or a new usage; and

responsive to the change in usage exceeding a threshold, generating a report for the word.

23. The system of claim 20, wherein determining whether the chat message contains any prohibited metaphorical meaning includes identifying one or more safety rules, wherein each safety rule identifies an ordered sequence of one or more sets of word types, wherein the request is received by an application configured to:

upon determining at least a first word from one of the sets of words does not match a respective word in the chat message, approving the message to be sent to the specified recipient.

24. The system of claim 20, wherein determining whether the chat message contains any prohibited metaphorical meaning includes identifying one or more safety rules, at least one safety rule of which:

identifies combinations of words in the chat message which have a secondary semantic meaning, or includes sub rules to stop applying the safety rule to the chat message and permitting a number of words to be within a sequence of words in the chat message.