System and method for segment relevance detection for digital content using multimodal correlations

• US 10,638,197 B2
• Filed: 11/21/2018
• Issued: 04/28/2020
• Est. Priority Date: 11/07/2011
• Status: Active Grant

• Current Assignee: Monet Networks, Inc.
• Original Assignee: Monet Networks, Inc.
• Inventors: Bist, Anurag, Pujol, Ramon Solves, Frankel, Eric Leopold
• Primary Examiner(s): Hance, Robert J
• Application Number:
  US16/198,503
• Publication Number:
  US 20190297384A1
• Time in Patent Office: 524 Days
• CPC Class Codes: G06F 16/436 using biological or physiological data of a human being, e.g. blood pressure, facial expression, gestures
• CPC Class Codes: G06F 16/437 Administration of user profiles, e.g. generation, initialisation, adaptation, distribution
• CPC Class Codes: G06F 16/7867 using information manually generated, e.g. tags, keywords, comments, title and artist information, manually generated time, location and usage information, user ratings
• CPC Class Codes: G06Q 30/02 Marketing; Price estimation or determination; Fundraising
• CPC Class Codes: G06Q 50/01 Social networking
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: H04N 21/23418 involving operations for analysing video streams, e.g. detecting features or characteristics television picture signal circuitry for scene change detection H04N5/147; filtering for image enhancement G06T5/00; methods or arrangements for recognising scenes G06V20/00; arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/25891 being end-user preferences retrieval of video data in a video database based on user preferences G06F16/739; arrangements for recognizing users' preferences H04H60/46; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4756 for rating content, e.g. scoring a recommended movie
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/8455 involving pointers to the content, e.g. pointers to the I-frames of the video stream
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/8549 Creating video summaries, e.g. movie trailer retrieval in video databases by using presentations in form of a video summary G06F16/739

Abstract

A system and method for media content evaluation based on combining multi-modal inputs from the audiences that may include reactions and emotions that are recorded in real-time on a frame-by-frame basis as the participants are watching the media content is provided. The real time reactions and emotions are recorded in two different campaigns with two different sets of people and which include different participants for each. For the first set of participants facial expression are captured and for the second set of participants reactions are captured. The facial expression analysis and reaction analysis of both set of participants are correlated to identify the segments which are engaging and interesting to all the participants.

Claims

1. A system for evaluating media content comprising:

a web-based application to stream a media content to a first set of participants and a second set of participants;

a server having a processor, and a facial detection engine, the server is configured to;

receive facial expression of one or more participants of the first set of participants in the form of video recordings and processing by the facial detection engine to identify one or more emotions of the first set of participants in frame-by-frame manner;

receive reactions of one or more participants in second set of participants, in which the reactions are captured by presenting one or more emojis to the second set of participants while the media content is playing, and asking the second set of participants to click emojis at different time of displayed media content to mark corresponding reactions;

wherein the server plots a graphical representation of the facial expression captured for the first set of participants and the reactions of second set of participants to identify one or more segment of the media content that are engaging for both the first set of participants and the second set of participants.

2. The system of claim 1, wherein an owner of the media content upload the media content on the server through a campaign using the web-based application.

3. The system of claim 2, wherein the owner of the media content specify the attributes for the first set of participants and the second set of participants, which includes, but are not limited to age, demography, ethnicity, gender, region.

4. The system of claim 1, wherein the one or more emotions identified by the facial detection includes but are not limited to Angry, sad, neutral, fear, surprise, joy and disgust.

5. The system of claim 1, wherein the reactions of the second set of participants include like, dislike, love, memorable and want.

6. The system of claim 1, wherein the server determines one or more segment of interest for the first set of participants by plotting a chart for the average of the emotions of the first set of participants with corresponding intensity in frame by frame manner and identifying slope trends, magnitude trends and peak trends in the chart to identify one or more segment of media content of interest to the first set of participants.

7. The system of claim 1, wherein the server determines one or more segment of interest for the second set of participants by plotting a chart with number of clicks in each time frame and determining one or more segments with highest number of clicks.

8. The system of claim 1, wherein the server determines one or more segments of interest to both the first set of participants and the second set of participants by correlating the chart for the average of emotions of the first set of participants and the chart with number of clicks in each time frame of the second set of participants;

and then identifying highest indicators in the chart of the first set of participants and the chart of the second set of participants.

9. A method for evaluating media content comprising:

streaming a media content to a first set of participants and a second set of participants;

providing a server having a processor, and a facial detection engine, the server is configured to;

receive facial expression of one or more participants of the first set of participants in form of video recordings and processing by the facial detection engine to identify one or more emotions of the first set of participants in frame-by-frame manner;

receive reactions of one or more participants in second set of participants, in which the reactions are captured by presenting one or more emojis to the second set of participants while the media content is playing, and asking the second set of participants to click emojis at different time of displayed media content to mark corresponding reactions;

wherein the server plots a graphical representation of the facial expression captured for the first set of participants and the reactions of second set of participants to identify one or more segment of the media content that are engaging for both the first set of participants and the second set of participants.

10. The method of claim 9, wherein an owner of the media content upload the media content on the server through a campaign using the web-based application.

11. The method of claim 10, wherein the owner of the media content specify the attributes for the first set of participants and the second set of participants, which includes, but are not limited to age, demography, ethnicity, gender, region.

12. The method of claim 9, wherein the one or more emotions identified by the facial detection includes but are not limited to Angry, sad, neutral, fear, surprise, joy and disgust.

13. The method of claim 9, wherein the reactions of the second set of participants include like, dislike, love, memorable and want.

14. The method of claim 9, wherein the server determines one or more segment of interest for the first set of participants by plotting a chart for the average of the emotions of the first set of participants with corresponding intensity in frame by frame manner and identifying slope trends, magnitude trends and peak trends in the chart to identify one or more segment of media content of interest to the first set of participants.

15. The method of claim 9, wherein the server determines one or more segment of interest for the second set of participants by plotting a chart with number of clicks in each time frame and determining one or more segments with highest number of clicks.

16. The method of claim 9, wherein the server determines one or more segments of interest to both the first set of participants and the second set of participants by correlating the chart for the average of emotions of the first set of participants and the chart with number of clicks in each time frame of the second set of participants;

and then identifying highest indicators in the chart of the first set of participants and the chart of the second set of participants.