Emotional reaction sharing

• US 10,573,048 B2
• Filed: 07/25/2016
• Issued: 02/25/2020
• Est. Priority Date: 07/25/2016
• Status: Active Grant

• Current Assignee: Yahoo Assets LLC
• Original Assignee: Oath Inc. (Verizon Communications Inc.)
• Inventors: Ni, Bin, Choi, Gregory Davis, Mathes, Adam Bryan
• Primary Examiner(s): Doan, Phuc N
• Application Number:
  US15/218,268
• Publication Number:
  US 20180027307A1
• Time in Patent Office: 1,310 Days
• CPC Class Codes: G06T 13/205 driven by audio data
• CPC Class Codes: G06T 13/40 of characters, e.g. humans, animals or virtual beings
• CPC Class Codes: G06T 2200/08 involving all processing steps from image acquisition to 3D model generation
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/30201 Face
• CPC Class Codes: G06V 40/165 using facial parts and geometric relationships
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104

Abstract

One or more computing devices, systems, and/or methods for emotional reaction sharing are provided. For example, a client device captures video of a user viewing content, such as a live stream video. Landmark points, corresponding to facial features of the user, are identified and provided to a user reaction distribution service that evaluates the landmark points to identify a facial expression of the user, such as a crying facial expression. The facial expression, such as landmark points that can be applied to a three-dimensional model of an avatar to recreate the facial expression, are provided to client devices of users viewing the content, such as a second client device. The second client device applies the landmark points of the facial expression to a bone structure mapping and a muscle movement mapping to create an expressive avatar having the facial expression for display to a second user.

Claims

1. A computing device comprising:

a processor; and

memory comprising processor-executable instructions that when executed by the processor cause performance of operations, the operations comprising;

receiving, at a user reaction distribution service, a first set of landmark points, a second set of landmark points, and a mood of a first user from a client device, wherein;

the first set of landmark points represents a set of facial features of the first user at a first point in time and the second set of landmark points represents the set of facial features of the first user at a second point in time while the first user is viewing content through the client device, and

the mood is identified at the client device from audio of the first user while the first user is viewing the content through the client device;

evaluating, at the user reaction distribution service, the first set of landmark points and the second set of landmark points, using a facial expression recognition algorithm that maps changes in location of landmark points to facial movements indicative of facial expressions, to identify a facial expression of the first user while the first user is viewing the content;

verifying the facial expression of the first user based upon the mood;

identifying, at the user reaction distribution service, a set of facial expressions of other users viewing the content during a time interval between the first point in time and the second point in time based upon landmark points received from client devices of the other users, wherein;

the client device of the first user and the client devices of the other users define a group of client devices, and

the facial expression of the first user and the set of facial expressions of other users define a group of facial expressions;

ranking, at the user reaction distribution service, the group of facial expressions to determine a most frequently occurring facial expression, amongst the group of facial expressions, during the time interval; and

sending, from the user reaction distribution service, the most frequently occurring facial expression to a plurality of client devices amongst the group of client devices in real-time during viewing of the content by the first user and by the other users.

2. The computing device of claim 1, wherein the operations comprise:

training the facial expression recognition algorithm to identify a type of facial expression based upon a training set of landmark points associated with the type of facial expression, the training set of landmark points corresponding to facial movement indicative of the type of facial expression.

3. The computing device of claim 1, wherein:

the first user and the other users define a group of users, and

the operations comprise;

identifying, at the user reaction distribution service, a second set of facial expressions of the group of users viewing the content;

ranking, at the user reaction distribution service, the second set of facial expressions based upon frequency;

identifying, at the user reaction distribution service, a change to the most frequently occurring facial expression based upon the ranking the second set of facial expressions; and

sending, from the user reaction distribution service, an update to the plurality of client devices in real-time during viewing of the content by the group of users based upon the change.

4. The computing device of claim 3, wherein:

the second set of facial expressions comprises facial expressions of the group of users during a second time interval, and

the time interval corresponds to a first interval of time during which a first portion of the content is viewed and the second time interval corresponds to a second interval of time during which a second portion of the content is viewed.

5. The computing device of claim 1, wherein the sending comprises sending a set of landmark points corresponding to the most frequently occurring facial expression to the plurality of client devices.

6. The computing device of claim 1, wherein the sending comprises sending at least one of an image, animation, text, or symbol representative of the most frequently occurring facial expression to the plurality of client devices.

7. A method of emotional reaction sharing, the method comprising:

receiving, at a user reaction distribution service, a first set of landmark points, a second set of landmark points, and a mood of a first user from a client device, wherein;

the first set of landmark points represents a set of facial features of the first user at a first point in time and the second set of landmark points represents the set of facial features of the first user at a second point in time while the first user is viewing content through the client device, and

the mood is identified at the client device from audio of the first user while the first user is viewing the content through the client device;

evaluating, at the user reaction distribution service, the first set of landmark points and the second set of landmark points, using a facial expression recognition algorithm that maps changes in location of landmark points to facial movements indicative of facial expressions, to identify a facial expression of the first user while the first user is viewing the content;

verifying the facial expression of the first user based upon the mood;

identifying, at the user reaction distribution service, a set of facial expressions of other users viewing the content during a time interval between the first point in time and the second point in time based upon landmark points received from client devices of the other users, wherein;

the client device of the first user and the client devices of the other users define a group of client devices, and

the facial expression of the first user and the set of facial expressions of other users define a group of facial expressions;

ranking, at the user reaction distribution service, the group of facial expressions to determine a most frequently occurring facial expression, amongst the group of facial expressions, during the time interval; and

sending, from the user reaction distribution service, the most frequently occurring facial expression to a plurality of client devices amongst the group of client devices in real-time during viewing of the content by the first user and by the other users.

8. The method of claim 7, comprising:

training the facial expression recognition algorithm to identify a type of facial expression based upon a training set of landmark points associated with the type of facial expression, the training set of landmark points corresponding to facial movement indicative of the type of facial expression.

9. The method of claim 7, wherein:

the first user and the other users define a group of users, and

the method comprises;

identifying, at the user reaction distribution service, a second set of facial expressions of the group of users viewing the content;

ranking, at the user reaction distribution service, the second set of facial expressions based upon frequency;

identifying, at the user reaction distribution service, a change to the most frequently occurring facial expression based upon the ranking the second set of facial expressions; and

sending, from the user reaction distribution service, an update to the plurality of client devices in real-time during viewing of the content by the group of users based upon the change.

10. The method of claim 9, wherein:

the second set of facial expressions comprises facial expressions of the group of users during a second time interval, and

the time interval corresponds to a first interval of time during which a first portion of the content is viewed and the second time interval corresponds to a second interval of time during which a second portion of the content is viewed.

11. A non-transitory machine readable medium having stored thereon processor-executable instructions that when executed cause performance of operations, the operations comprising:

receiving, at a user reaction distribution service, a first set of landmark points, a second set of landmark points, and a mood of a first user from a client device, wherein;

the first set of landmark points represents a set of facial features of the first user at a first point in time and the second set of landmark points represents the set of facial features of the first user at a second point in time while the first user is viewing content through the client device, and

the mood is identified at the client device from audio of the first user while the first user is viewing the content through the client device;

evaluating, at the user reaction distribution service, the first set of landmark points and the second set of landmark points, using a facial expression recognition algorithm that maps changes in location of landmark points to facial movements indicative of facial expressions, to identify a facial expression of the first user while the first user is viewing the content;

verifying the facial expression of the first user based upon the mood;

identifying, at the user reaction distribution service, a set of facial expressions of other users viewing the content during a time interval between the first point in time and the second point in time based upon landmark points received from client devices of the other users, wherein;

the client device of the first user and the client devices of the other users define a group of client devices, the first user and the other users define a group of users, and the facial expression of the first user and the set of facial expressions of other users define a group of facial expressions;

ranking, at the user reaction distribution service, the group of facial expressions to determine a most frequently occurring facial expression, amongst the group of facial expressions, during the time interval; and

sending, from the user reaction distribution service, the most frequently occurring facial expression to a plurality of client devices amongst the group of client devices in real-time during viewing of the content by the first user and by the other users.

12. The non-transitory machine readable medium of claim 11, comprising:

training the facial expression recognition algorithm to identify a type of facial expression based upon a training set of landmark points associated with the type of facial expression, the training set of landmark points corresponding to facial movement indicative of the type of facial expression.

13. The non-transitory machine readable medium of claim 11, wherein the sending the most frequently occurring facial expression comprises sending a set of landmark points corresponding to the most frequently occurring facial expression to the plurality of client devices.

14. The non-transitory machine readable medium of claim 11, the operations comprising:

identifying, at the user reaction distribution service, a second set of facial expressions of the group of users viewing the content;

ranking, at the user reaction distribution service, the second set of facial expressions based upon frequency;

identifying, at the user reaction distribution service, a change to the most frequently occurring facial expression based upon the ranking the second set of facial expressions; and

sending, from the user reaction distribution service, an update to the plurality of client devices in real-time during viewing of the content by the group of users based upon the change.

15. The non-transitory machine readable medium of claim 14, wherein:

the second set of facial expressions comprises facial expressions of the group of users during a second time interval, and

the time interval corresponds to a first interval of time during which a first portion of the content is viewed and the second time interval corresponds to a second interval of time during which a second portion of the content is viewed.

16. The non-transitory machine readable medium of claim 14, wherein the sending the update comprises sending a set of landmark points corresponding to the most frequently occurring facial expression to the plurality of client devices.

17. The non-transitory machine readable medium of claim 11, the operations comprising:

identifying, at the user reaction distribution service, a second set of facial expressions of the group of users viewing the content;

ranking, at the user reaction distribution service, the second set of facial expressions based upon frequency; and

identifying, at the user reaction distribution service, a change to the most frequently occurring facial expression based upon the ranking the second set of facial expressions.

18. The non-transitory machine readable medium of claim 17, wherein:

the second set of facial expressions comprises facial expressions of the group of users during a second time interval, and

the time interval corresponds to a first interval of time during which a first portion of the content is viewed and the second time interval corresponds to a second interval of time during which a second portion of the content is viewed.