System and method for dynamically selecting supplemental content based on viewer emotions

• US 10,171,877 B1
• Filed: 10/30/2017
• Issued: 01/01/2019
• Est. Priority Date: 10/30/2017
• Status: Active Grant

• Current Assignee: Dish Network LLC (Dish Network Corporation)
• Original Assignee: Dish Network LLC (Dish Network Corporation)
• Inventors: Shah, Rima, Blake, Ryan, Day, Christopher
• Primary Examiner(s): Pierorazio, Michael B
• Application Number:
  US15/798,061
• Time in Patent Office: 428 Days
• Field of Search: 725 12
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4668 for recommending content, e.g. movies
• CPC Class Codes: H04N 21/812 involving advertisement data advertising per se G06Q30/02

Abstract

Embodiments are directed towards providing supplemental content to a viewer based on the viewer'"'"'s emotion and environment while the viewer is viewing content. Content is provided to a content receiver for presentation to the viewer. While the viewer is viewing the content, an image of the viewer is captured and analyzed to detect the viewer'"'"'s emotion and environment. Supplemental content is selected based on this detected viewer emotion and environment and presented to the viewer, which allows the supplemental content to be tailored to the emotional state of the viewer. One or more additional images of the viewer are captured while the viewer is viewing the supplemental content and the viewer'"'"'s emotion while viewing the supplemental content is detected. This subsequent viewer emotion is analyzed to determine if the viewer had an expected response to the supplemental content, which can be used to determine the efficiency of the supplemental content.

Claims

1. A system, comprising:

one or more cameras;

a database storing relationships between content categories and supplemental-content categories, and corresponding emotional responses;

one or more processors; and

a memory device storing computer instructions that, when executed by the one or more processors, cause the one or more processors to;

provide content having a content category to a content receiver for presentation to a viewer;

receive at least one first image of the viewer taken by the one or more cameras while the content is being presented to the viewer;

analyze the at least one first image to detect a first viewer emotion in reaction to the content currently being presented to the viewer;

select supplemental content having a supplemental-content category to provide to the viewer based on the detected first viewer emotion and the content category;

provide the supplemental content to the content receiver for presentation to the viewer;

receive at least one second image of the viewer taken by the one or more cameras while the supplemental content is being presented to the viewer;

analyze the at least one second image to detect a second viewer emotion in reaction to the supplemental content currently being presented to the viewer;

determine an emotional response of the viewer based on the detected first viewer emotion and the detected second viewer emotion; and

update a relationship between the content category and the supplemental-content category in the database based the determined emotional response of the viewer.

2. The system of claim 1, wherein the determination of the emotional response includes:

detecting if a change between the detected first viewer emotion and the detected second viewer emotion is an expected emotional response; and

updating the relationship between the content category and the supplemental-content category to indicate the positive emotional response of the viewer.

3. The system of claim 1, wherein the determination of the emotional response includes:

detecting if a change between the detected first viewer emotion and the detected second viewer emotion is a negative emotional response; and

updating the relationship between the content category and the supplemental-content category to indicate the negative emotional response of the viewer.

4. The system of claim 1, wherein the memory device stores further computer instructions that, when executed by the one or more processors, cause the one or more processors to:

analyze the at least one first image for first contextual information regarding an environment surrounding the viewer while the content is being presented to the viewer;

analyze the at least one second image for second contextual information regarding the environment surrounding the viewer while the supplemental content is being presented to the viewer;

detect one or more changes in the environment based on a comparison between the first and second contextual information; and

update the relationship between the content category and the supplemental-content category in the database based the one or more detected changes in the environment.

5. The system of claim 4, wherein the detection of the one or more changes includes identifying a product in the at least one second image that was not in the at least one first image and is related to the supplemental content.

6. The system of claim 1, wherein the content category indicates a current emotion of the content and the supplemental-content category indicate current emotion of the supplemental content.

7. The system of claim 1, wherein the memory device stores further computer instructions that, when executed by the one or more processors, cause the one or more processors to:

analyze the at least one first image to detect contextual information regarding an environment surrounding the viewer while the content is being presented to the viewer; and

select the supplemental content based on the contextual information of the environment.

8. The system of claim 7, wherein the memory device stores further computer instructions that, when executed by the one or more processors, cause the one or more processors to:

analyze the at least one second image for second contextual information regarding the environment surrounding the viewer while the supplemental content is being presented to the viewer;

detect one or more changes in the environment based on a comparison between the first and second contextual information; and

update the relationship between the content category and the supplemental-content category in the database based the one or more detected changes in the environment.

9. The system of claim 1, wherein the content is a television program and the supplemental content is an advertisement.

10. A computing device, comprising:

a memory device that stores relationships between content categories and supplemental-content categories, and corresponding emotional responses, and that stores computer instructions;

one or more processors that execute the computer instructions to perform actions, the actions including;

providing content having a content category to a content receiver for presentation to a viewer;

analyzing at least one first image of an environment surrounding the viewer taken while the content is being presented to the viewer to identify first contextual information associated with the environment;

selecting supplemental content having a supplemental-content category to provide to the viewer based on the first contextual information;

providing the supplemental content to the content receiver for presentation to the viewer;

analyzing at least one second image of the environment taken while the supplemental content is being presented to the viewer to identify second contextual information associated with the environment;

determining an emotional response of the viewer based on changes between the first contextual information and the second contextual information; and

updating a relationship between the content category and the supplemental-content category in a database based the determined emotional response of the viewer.

11. The computing device of claim 10, wherein the determining of the emotional response includes:

comparing the first contextual information with the second contextual information to identify a product in the at least one second image that was not in the at least one first image; and

updating the relationship between the content category and the supplemental-content category in the database to indicate an expected emotional response of the viewer.

12. The computing device of claim 11, wherein the updating of the relationship between the content category and the supplemental-content category includes:

determining if the product is associated with the supplemental content; and

in response to the product being associated with the supplemental content, updating the relationship between the content category and the supplemental-content category in the database to indicate an expected emotional response of the viewer.

13. The computing device of claim 10, wherein the determining of the emotional response includes:

comparing the first contextual information with the second contextual information to identify movement of a product within the environment; and

updating the relationship between the content category and the supplemental-content category in the database to indicate an expected emotional response of the viewer.

14. The computing device of claim 13, wherein the updating of the relationship between the content category and the supplemental-content category includes:

determining if the product is associated with the supplemental content; and

in response to the product being associated with the supplemental content, updating the relationship between the content category and the supplemental-content category in the database to indicate an expected emotional response of the viewer.

15. The computing device of claim 10, wherein the determining of the emotional response of the viewer, further includes:

analyzing the at least one first image to detect a first viewer emotion in reaction to the content currently being presented to the viewer;

analyzing the at least one second image to detect a second viewer emotion in reaction to the supplemental content currently being presented to the viewer; and

determining the emotional response of the viewer based on the detected first viewer emotion and the detected second viewer emotion.

16. The computing device of claim 15, wherein the determining of the emotional response of the viewer, further includes:

detecting if a change between the detected first viewer emotion and the detected second viewer emotion is an expected emotional response; and

updating the relationship between the content category and the supplemental-content category to indicate the positive emotional response of the viewer.

17. The computing device of claim 15, wherein the determining of the emotional response of the viewer, further includes:

detecting if a change between the detected first viewer emotion and the detected second viewer emotion is a negative emotional response; and

updating the relationship between the content category and the supplemental-content category to indicate the negative emotional response of the viewer.

18. A method executing on a content distribution server, comprising:

providing content to a content receiver for presentation to a user;

receiving at least one image of the user while the content is being presented to the user;

analyzing the at least one image to detect an emotion of the user in reaction to the content currently being presented to the user;

selecting at least one supplemental content to provide to the user in response to the detected user emotion;

providing the at least one supplemental content to the content receiver for presentation to the user; and

determining an emotional response of the user in reaction to the at least one supplemental content being presented to the user based on a change from the detected user emotion to a second detected emotion of the user.

19. The method of claim 18, further comprising:

analyzing the at least one image to identify contextual information associated with an environment surrounding the user; and

selecting the at least one supplemental content based on the contextual information.

20. The method of claim 18, wherein the determining of the emotional response of the user includes:

receiving at least one second image of the user while the supplemental content is being presented to the user;

analyzing the at least one second image to detect the second emotion of the user in reaction to the supplemental content currently being presented to the user;

determining the emotional response of the user based on changes from the emotion of the user in reaction to the content to the second emotion of the user in reaction to the supplemental content; and

providing the emotional response to a supplier of the supplemental content.