Systems and methods to detect and correlate user responses to media content

• US 10,187,690 B1
• Filed: 04/24/2017
• Issued: 01/22/2019
• Est. Priority Date: 04/24/2017
• Status: Active Grant

• Current Assignee: GoPro, Inc.
• Original Assignee: GoPro, Inc.
• Inventors: Garcia, Vincent, Medioni, Tom, Rouif, Matthieu, Lema, Gabriel, Santoni, Francescu
• Primary Examiner(s): Smith, Chenea
• Application Number:
  US15/495,099
• Time in Patent Office: 638 Days
• CPC Class Codes: G06V 10/17 using hand-held instruments
• CPC Class Codes: G06V 40/166 using acquisition arrangements
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: H04N 21/42201 biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user input arrangements for interaction with the human body based on nervous system activity detection G06F3/015
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4312 involving specific graphical features, e.g. screen layout, special fonts or colors, blinking icons, highlights or animations
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/44226 on social networks
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

Systems and method of automatically detecting user responses from users to media content and correlating the user responses to the media content include a determination of an emotional response by a user—for example a smile—to a presentation of media content. The determined emotional response is correlated to the media content. Subsequent presentation of the same media content includes presentation of an indicator—for example a smiley —that indicates the determined emotional response.

Claims

1. A system for automatically detecting user responses from users to media content and correlating the user responses to the media content, the system comprising:

one or more physical processors configured by machine-readable instructions to;

obtain images including facial expressions of users during presentation of video content to the users, the images including a first image and a second image, the first image including a facial expression of a first user during the presentation of the video content to the first user and the second image including a facial expression of a second user during the presentation of the video content to the second user;

perform classification of the facial expressions of the users based on the images such that the facial expression of the first user is classified based on the first image and the facial expression of the second user is classified based on the second image;

determine individual items of response information that reflect user responses to the presentation based on the classification of the facial expressions of the individual users, the items of response information including a first item of response information reflecting the first user'"'"'s response to the presentation and a second item of response information reflecting the second user'"'"'s response to the presentation, wherein the individual items of response information correspond to individual emojis such that the first item of response information corresponds to a first emoji and the second item of response information corresponds to a second emoji;

aggregate the items of response information into aggregated response information, the aggregated response information corresponding an aggregated emoji, wherein the first emoji and the second emoji are of a given type of emoji, and the aggregated emoji is of the given type of emoji and characterized by a different size or brightness than the first emoji and the second emoji; and

correlate the aggregated response information to the video content such that subsequent presentation of the video content includes simultaneous presentation of the aggregated emoji.

2. The system of claim 1, wherein the first image is captured by a front-facing camera.

3. The system of claim 1, wherein the aggregated response information is associated with a time-stamp, and wherein presentation of the aggregated emoji is timed in accordance with the time-stamp.

4. The system of claim 1, wherein the one or more physical processors are further configured by the machine-readable instructions to:

detect one or more moments of interest in the video content based on the aggregated response information; and

create a highlight reel including at least one of the one or more moments of interest.

5. The system of claim 1, wherein the aggregated emoji is animated, and wherein animation of the aggregated emoji is in accordance with changes in the facial expressions of the users.

6. The system of claim 3, wherein the images further include a third image including a second facial expression of the first user, and wherein the one or more physical processors are further configured to:

perform classification of the second facial expression;

determine the first item of response information further based on the classification of the second facial expression, wherein the first emoji changes based on the second facial expression.

7. The system of claim 1, wherein the first item of response information is determined based on classification of a biometric parameter of the first user, the biometric parameter determined based on biometric information conveyed by output signals of a biometric sensor.

8. The system of claim 1, wherein the items of response information are aggregated into aggregated response information based on individual user responses occurring at a same moment of the video content.

9. The system of claim 1, wherein the items of response information are aggregated into aggregated response information regardless of whether individual user responses occurred at a same moment or different moments of the video content.

10. The system of claim 1, wherein the aggregated response information is modified based on obtaining new images including new facial expressions of the users or other users.

11. A computer-implemented method to automatically detect user responses from users to media content and correlate the user responses to the media content, the method being implemented on a computing system that includes one or more processors configured by machine-readable instructions, the method comprising:

obtain images including facial expressions of users during presentation of video content to the users, the images including a first image and a second image, the first image including a facial expression of a first user during the presentation of the video content to the first user and the second image including a facial expression of a second user during the presentation of the video content to the second user;

performing classification of the facial expressions of the users based on the images such that the facial expression of the first user is classified based on the first image and the facial expression of the second user is classified based on the second image;

determining individual items of response information that reflect user responses to the presentation based on the classification of the facial expressions of the individual users, the items of response information including a first item of response information reflecting the first user'"'"'s response to the presentation and a second item of response information reflecting the second user'"'"'s response to the presentation, wherein the individual items of response information correspond to individual emojis such that the first item of response information corresponds to a first emoji and the second item of response information corresponds to a second emoji;

aggregating the items of response information into aggregated response information, the aggregated response information corresponding an aggregated emoji, wherein the first emoji and the second emoji are of a given type of emoji, and the aggregated emoji is of the given type of emoji and characterized by a different size or brightness than the first emoji and the second emoji; and

correlating the aggregated response information to the video content such that subsequent presentation of the video content includes simultaneous presentation of the aggregated emoji.

12. The method of claim 11, wherein the first image is captured by a front-facing camera.

13. The method of claim 11, wherein the aggregated emoji is animated, and wherein animation of the aggregated emoji is in accordance with changes in the facial expressions of the users.

14. The method of claim 11, wherein the first item of response information is determined based on classification of a biometric parameter of the first user, the biometric parameter determined based on biometric information conveyed by output signals of a biometric sensor.

15. The method of claim 11, wherein the aggregated response information is associated with a time-stamp, and wherein presentation of the aggregated emoji is timed in accordance with the time-stamp.

16. The method of claim 15, wherein the images further include a third image including a second facial expression of the first user, the method further comprising:

performing classification of the second facial expression;

determining the first item of response information further based on the classification of the second facial expression, wherein the first emoji changes based on the second facial expression.

17. The method of claim 11, wherein the items of response information are aggregated into aggregated response information based on individual user responses occurring at a same moment of the video content.

18. The method of claim 11, wherein the items of response information are aggregated into aggregated response information regardless of whether individual user responses occurred at a same moment or different moments of the video content.