Content categorization using facial expression recognition, with improved detection of moments of interest

• US 9,918,128 B2
• Filed: 04/05/2017
• Issued: 03/13/2018
• Est. Priority Date: 04/08/2016
• Status: Active Grant

• Current Assignee: Orange S.A.
• Original Assignee: Orange S.A.
• Inventors: Courouge, M. Guillaume, Lapiche, M. Antonin
• Primary Examiner(s): PIERORAZIO, MICHAEL
• Application Number:
  US15/479,542
• Publication Number:
  US 20170295402A1
• Time in Patent Office: 342 Days
• Field of Search: 725 12
• CPC Class Codes: G06V 20/46 Extracting features or characteristics from the video content, e.g. video fingerprints, representative shots or key frames
• CPC Class Codes: G06V 2201/10 Recognition assisted with metadata
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: H04N 21/252 Processing of multiple end-users' preferences to derive collaborative data
• CPC Class Codes: H04N 21/44204 Monitoring of content usage, e.g. the number of times a movie has been viewed, copied or the amount which has been watched monitoring of user activities for profile generation for accessing a video database G06F16/739; protecting generic digital content where the protection is independent of the precise nature of the content G06F21/10; arrangements for monitoring the use made of the broadcast services in broadcast systems H04H60/31
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/44222 Analytics of user selections, e.g. selection of programs or purchase activity monitoring of user selections in data processing systems G06F11/34; arrangements for monitoring the user's behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/4532 involving end-user characteristics, e.g. viewer profile, preferences monitoring of user activities for profile generation for accessing a video database G06F16/739; user profiles in network data switching protocols H04L67/306; processing of user preferences or user profiles in wireless networks H04W8/18
• CPC Class Codes: H04N 21/4668 for recommending content, e.g. movies
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38
• CPC Class Codes: H04N 21/84 Generation or processing of descriptive data, e.g. content descriptors systems specially adapted for using meta-information in broadcast systems H04H60/73
• CPC Class Codes: H04N 21/8456 by decomposing the content in the time domain, e.g. in time segments
• CPC Class Codes: H04N 21/854 Content authoring
• CPC Class Codes: H04N 21/8547 involving timestamps for synchronizing content

Abstract

The invention relates to content categorization and a method to that end comprises acts of:

• • rendering content within graphical user interfaces to a plurality of viewers;
  • receiving multiple signals related to feedbacks from the plurality of viewers relatively to the rendered content; and
  • tallying the multiple received signals from the plurality of viewers to categorize the content.

More particularly:

• • the content is fragmented into a plurality of fragments having a predetermined duration,
  • each signal comprises data of a facial expression recognition of each viewer, to sense a mood expression of the viewer during the content rendering,
  • a memory is provided for storing at least timestamps associated to fragments for which an intensity of the sensed mood expression of a viewer is above a first threshold, and
  • a processing circuit is further provided for implementing tallying acts of:
    • For all the viewers and for each given timestamp, using the memory for counting a first number of viewers for which the intensity is above the first threshold during the fragment corresponding to the given timestamp, and
    • When the first number is above a second threshold, labelling the fragment corresponding to the given timestamp as a moment of interest of the content.

Claims

1. A method of content categorization comprising acts of:

rendering content within graphical user interfaces to a plurality of viewers ;

receiving multiple signals related to feedbacks from the plurality of viewers relatively to the rendered content; and

tallying the multiple received signals from the plurality of viewers to categorize the content, wherein;

the content is fragmented into a plurality of fragments having a predetermined duration,

each signal comprises data of a facial expression recognition of each viewer, to sense a mood expression of the viewer during the content rendering,

a memory is provided for storing at least timestamps associated to fragments for which an intensity of the sensed mood expression of a viewer is above a first threshold, and

a processing circuit is further provided for implementing tallying acts of;

For all the viewers and for each given timestamp, using the memory for counting a first number of viewers for which the intensity is above the first threshold during the fragment corresponding to the given timestamp, and

When the first number is above a second threshold, labelling the fragment corresponding to the given timestamp as a moment of interest of the content.

2. The method of claim 1, wherein the memory is further arranged for storing timestamps along with corresponding intensities for each viewer, and the method further comprises tallying acts of:

for each moment labelled as a moment of interest of the content, calculating an average of the corresponding stored intensities,

when said average is above a third threshold, the moment of interest is further labelled as an emotional moment.

3. The method of claim 2, wherein:

if at least a first emotional moment and a second emotional moment are separated by less than a predetermined number of fragments, then a whole part of the content extending from the first emotional moment to the second emotional moment, is labeled as one emotional moment.

4. The method according to claim 1, wherein a number of moments of interest is counted over the content fragments and data of said number are included in a data set of recommendation of the content.

5. The method according to claim 1, wherein a cumulated duration of moments of interest is counted over the whole content duration and data of said cumulated duration are included in a data set of recommendation of the content.

6. The method according to claim 3, wherein a number of emotional moments is counted over the content fragments and data of said number are included in a data set of recommendation of the content.

7. The method according to claim 3, wherein at least a cumulated duration of emotional moments is counted over the whole content duration and data of said cumulated duration are included in a data set of recommendation of the content.

8. The method according to claim 3, further comprising a step of calculating a content recommendation for a user according to a profile of the user, and wherein said content recommendation is calculated on a basis of a weighted average, weights of the average depending on the user profile and being assigned to data of a data set of recommendation of the content, said data set comprising at least data of:

number of moments of interest counted over the content fragments,

cumulated duration of moments of interest counted over the whole content duration,

number of emotional moments counted over the content fragments, and

cumulated duration of emotional moments counted over the whole content duration.

9. The method according to claim 1, wherein the first threshold is estimated on the basis of metadata related to the content.

10. The method of claim 1, wherein a plurality of types of facial expressions is sensed and respective first thresholds are associated to each type of facial expression.

11. A non-transitory computer-readable storage medium with an executable program stored thereon, wherein the program instructs a processor to perform the method of claim 1.

12. A content categorization system, comprising:

a processing circuit,

an input interface connected to the processing circuit so as to receive signals related to feedbacks from a plurality of viewers relatively to a rendered content, each signal comprising data of a facial expression recognition of each viewer, each signal resulting from sensing a mood expression of the viewer during the content rendering,

a clock connected to the processing circuit for associating a timestamp to a fragment of the content for which an intensity of the sensed mood expression of a viewer is above a first threshold, and a memory connected to the processing circuit for storing at least the timestamp,

an output interface for delivering content recommendation data,

and wherein the processing circuit is arranged for performing, in cooperation with the input interface, the clock and the memory, tallying acts of;

For all the viewers and for each given timestamp, using the memory for counting a first number of viewers for which the intensity is above the first threshold during the fragment corresponding to the given timestamp, and

When the first number is above a second threshold, labelling the fragment corresponding to the given timestamp as a moment of interest of the content,

so as to feed the output interface with content recommendation data derived at least from moments of interest of the content.

13. A content categorization device, comprising:

an input interface for receiving timestamps associated to fragments of the content for which an intensity of a sensed mood expression of one or several viewers during a rendering said fragments is above a first threshold,

a memory for storing at least the received timestamps,

an output interface for delivering content recommendation data,

and a processing circuit arranged for performing, in cooperation with the input interface and the memory, tallying acts of;

For all the viewers and for each given timestamp, using the memory for counting a first number of viewers for which the intensity is above the first threshold during the fragment corresponding to the given timestamp, and

When the first number is above a second threshold, labelling the fragment corresponding to the given timestamp as a moment of interest of the content,

so as to feed the output interface with content recommendation data derived at least from moments of interest of the content.