Facial gesture recognition and video analysis tool

• US 10,021,344 B2
• Filed: 12/21/2016
• Issued: 07/10/2018
• Est. Priority Date: 07/02/2015
• Status: Active Grant

• Current Assignee: Krush Technologies, LLC
• Original Assignee: Krush Technologies, LLC
• Inventors: Farrell, Matthew J., Nauseef, John P., Wire, Christopher S., Faust, Brian T., Murray, Patrick M., Free, Linsey Ann, Riggins, Gary T., Nesbitt, John C., Clinard, Dustin L.
• Primary Examiner(s): Anwah, Olisa
• Application Number:
  US15/387,172
• Publication Number:
  US 20170104958A1
• Time in Patent Office: 566 Days
• Field of Search: 348 1401, 348 1402, 348 1403, 348 1408
• CPC Class Codes: A63F 13/213 comprising photodetecting means, e.g. cameras, photodiodes or infrared cells A63F13/219, A63F13/655 take precedence
• CPC Class Codes: A63F 13/216 using geographical information, e.g. location of the game device or player using GPS
• CPC Class Codes: A63F 13/352 involving special game server arrangements, e.g. regional servers connected to a national server or a plurality of servers managing partitions of the game world
• CPC Class Codes: A63F 13/79 involving player-related data, e.g. identities, accounts, preferences or play histories
• CPC Class Codes: A63F 13/795 for finding other players; for building a team; for providing a buddy list
• CPC Class Codes: A63F 13/87 Communicating with other players during game play, e.g. by e-mail or chat
• CPC Class Codes: A63F 2300/8094 Unusual game types, e.g. virtual cooking
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 40/168 Feature extraction; Face representation
• CPC Class Codes: G06V 40/172 Classification, e.g. identification
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/70 Multimodal biometrics, e.g. combining information from different biometric modalities
• CPC Class Codes: G10L 15/02 Feature extraction for speech recognition; Selection of recognition unit
• CPC Class Codes: G10L 15/22 Procedures used during a speech recognition process, e.g. man-machine dialogue
• CPC Class Codes: G10L 25/90 Pitch determination of speech signals
• CPC Class Codes: H04L 65/1069 Session establishment or de-establishment
• CPC Class Codes: H04L 67/303 Terminal profiles
• CPC Class Codes: H04N 7/141 between two video terminals, e.g. videophone telephonic communication systems combined with television receiver for reception of entertainment or information matter H04M11/085
• CPC Class Codes: H04N 7/147 Communication arrangements, e.g. identifying the communication as a video-communication, intermediate storage of the signals selecting H04Q
• CPC Class Codes: H04N 7/15 : Conference systems

Abstract

Embodiments disclosed herein may be directed to a video communication server. In some embodiments, the video communication server includes: at least one memory including instructions; and at least one processing device configured for executing the instructions, wherein the instructions cause the at least one processing device to perform the operations of: determining a time duration of a video communication connection between a first user of a first user device and a second user of a second user device; analyzing video content transmitted between the first user device and the second user device; determining at least one gesture of at least one of the first user and the second user based on analyzing the video content; and generating a compatibility score of the first user and the second user based at least in part on the determined time duration and the at least one determined gesture.

Claims

1. A video communications and personal compatibility communications system, the system comprising:

a) at least one memory comprising instructions;

b) a video communications device operable to receive and transmit video communications between first and second user devices;

c) at least one profile management unit;

d) at least one profile database;

e) at least one compatibility unit; and

f) at least one processing device in communication with and operable to manage the video communications device, the at least one profile management unit, the at least one profile database, and the at least one compatibility unit, the at least one processing device further operable to execute the instructions stored in the at least one memory, wherein executing the instructions causes the at least one processing device to;

initiate a video communication session between first and second users on the first and second user devices respectively, the at least one profile database operable to receive profile data for each of the first and second users from the profile management unit;

analyze video transmitted between the first and second user devices in coordination with the video communications device, which further causes the at least one processing device to determine a gesture associated with at least one of the first user or the second user in the transmitted video;

determine during the video communication session whether the transmitted video comprises objectionable content;

implement a safety protocol based on determining the transmitted video comprises the objectionable content;

determine a compatibility indication based on the transmitted video between the first and second user devices;

receive the compatibility indication from each of the first and second user devices if the safety protocol does not otherwise terminate the video communication session;

determine, using the compatibility unit, a compatibility between the first and second users based at least in part on the received compatibility indications; and

communicate an indication of the determined compatibility to the first and second user devices, wherein executing the instructions that cause the at least one processing device to determine the transmitted video comprises the objectionable content further causes the at least one processing device to;

identify an object of interest comprised in the transmitted video;

compare the object of interest to flagged objects stored in a flagged objects database, wherein the flagged objects comprise objects other than facial features of the first and second users; and

determine at least a partial match between the object of interest and a flagged object in the flagged objects database,

wherein the compatibility indication is further determined based on the gesture.

2. The system of claim 1, wherein executing the instructions that cause the at least one processing device to determine the transmitted video comprises the objectionable content further causes the at least one processing device to:

compare the object of interest to permitted objects stored in a permitted objects database, wherein the permitted objects comprise the facial features of the first and second users; and

determine no match between the object of interest and a permitted object in the permitted object database.

3. The system of claim 1, wherein executing the instructions that cause the at least one processing device to implement the safety protocol further causes the at least one processing device to at least one of:

modify at least a portion of the transmitted video prior to the transmitted video being displayed on at least one of the first user device and the second user device;

suspend the video communication session for a predetermined amount of time; and

terminate the video communication session.

4. The system of claim 1, wherein executing the instructions further causes the at least one processing device to:

generate an indication that a rule of the video communication session has been violated in response to determining that the transmitted video comprises the objectionable content; and

transmit the indication that the rule of the video communication session has been violated by at least one of the first and second user devices.

5. The system of claim 1, wherein the compatibility indications are generated based at least in part on user input received from the first and second users at the first and second user devices, respectively.

6. The system of claim 1, wherein the objectionable content is comprised in an audio portion of the transmitted video.

7. A non-transitory computer readable medium comprising code, wherein the code, when executed by at least one processing device of a video communication server, causes the at least one processing device to perform the operations of:

establishing, using a communication unit comprised in the at least one processing device, a video communication connection between a first user of a first user device and a second user of a second user device, wherein the video communication connection enables the first and second users to participate in a video conference;

receiving, at a video analysis unit comprised in the at least one processing device and during the video conference, video content transmitted between the first user device and the second user device;

determining, using a recognition unit comprised in the at least one processing device and during the video conference, the video content comprises objectionable content based on analyzing the video content; and

implementing, using a graphical processing unit (GPU) comprised in the at least one processing device, a safety protocol during the video conference based on determining the video content comprises the objectionable content, wherein implementing the safety protocol comprises;

identifying an object of interest comprised in the video content;

comparing the object of interest to permitted objects stored in a permitted objects database, wherein the permitted objects comprise previously stored facial features of the first and second users; and

determining no match between the object of interest and a permitted object in the permitted objects database.

8. The non-transitory computer readable medium of claim 7, wherein the code that, when executed by the at least one processing device of the video communication server, causes the at least one processing device to perform the operations of determining the video content comprises the objectionable content further causes the at least one processing device to crop the object of interest.

9. The non-transitory computer readable medium of claim 7, wherein the code that, when executed by the at least one processing device of the video communication server, causes the at least one processing device to perform the operations of determining the video content comprises the objectionable content further causes the at least one processing device to perform the operations of:

determining a compatibility of the first user and the second user based on a social cue of the first user or the second user in the video content and based on a duration of the video conference.

10. The non-transitory computer readable medium of claim 7, wherein the code that, when executed by the at least one processing device of the video communication server, causes the at least one processing device to perform the operations of implementing the safety protocol further causes the at least one processing device to determine whether the first user device or the second user device is an offending user device that caused implementation of the safety protocol, and adjust a setting of a camera of the offending user device.

11. The non-transitory computer readable medium of claim 7, wherein the code that, when executed by the at least one processing device of the video communication server, causes the at least one processing device to perform the operations of implementing the safety protocol further causes the at least one processing device to perform the operations of:

modifying, using the GPU, at least a portion of the video content prior to the video content being displayed on at least one of the first user device and the second user device.

12. The non-transitory computer readable medium of claim 7, further comprising code that, when executed by the at least one processing device of the video communication server, causes the at least one processing device to perform the operations of:

generating, using the GPU, an indication that a rule of the video conference has been broken in response to determining that the video content comprises the objectionable content; and

transmitting, using the communication unit, the indication to at least one of the first and second user devices.

13. The non-transitory computer readable medium of claim 7, wherein the objectionable content is comprised in at least one of an image frame of the video content and an audio portion of the video content.

14. A method comprising:

establishing, using a communication unit comprised in at least one processing device of a video communications server, a video communication connection between a first user of a first user device and a second user of a second user device, wherein the video communication connection enables the first and second users to participate in a video conference;

receiving, at a video analysis unit comprised in the at least one processing device and during the video conference, video content transmitted between the first user device and the second user device;

determining, using a recognition unit comprised in the at least one processing device and during the video conference, the video content comprises objectionable content based on analyzing the video content; and

implementing, using a graphical processing unit (GPU) comprised in the at least one processing device, a safety protocol during the video conference based on determining the video content comprises the objectionable content, wherein implementing the safety protocol comprises;

modifying, using the GPU, a size of the objectionable content comprised in the video content prior to the video content being displayed on at least one of the first user device and the second user device; and

automatically terminating the video conference.

15. The method of claim 14, wherein determining the video content comprises the objectionable content comprises:

identifying, using the recognition unit, an object of interest comprised in the video content;

comparing, using the recognition unit, the object of interest to objects stored in a database, the objects in the database comprising a facial feature of the first user for recall during analysis of the video content; and

determining, using the recognition unit, at least a partial match between the object of interest and an object in the database.

16. The method of claim 14, wherein determining the video content comprises the objectionable content comprises:

identifying, using the recognition unit, an object of interest comprised in the video content;

comparing, using the recognition unit, the object of interest to objects stored in a database; and

determining, using the recognition unit, no match between the object of interest and an object in the database.

17. The method of claim 14, wherein implementing the safety protocol comprises:

identifying an object of interest comprised in the video content;

comparing the object of interest to flagged objects stored in a flagged objects database, wherein the flagged objects comprise objects other than facial features of the first and second users; and

determining at least a partial match between the object of interest and a flagged object in the flagged objects database.

18. The method of claim 14, further comprising:

generating, using the GPU, an indication that a rule of the video conference has been broken in response to determining that the video content comprises the objectionable content, the video conference being part of a video dating application; and

transmitting, using the communication unit, the indication to at least one of the first and second user devices.