Video recommendation via affect

• US 10,289,898 B2
• Filed: 11/21/2016
• Issued: 05/14/2019
• Est. Priority Date: 06/07/2010
• Status: Active Grant

• Current Assignee: Affectiva, Inc. (Smart Eye AB)
• Original Assignee: Affectiva, Inc. (Smart Eye AB)
• Inventors: el Kaliouby, Rana, Mahmoud, Abdelrahman, Turcot, Panu James
• Primary Examiner(s): Flynn, Nathan J
• Assistant Examiner(s): Fogg, Cynthia M
• Application Number:
  US15/357,585
• Publication Number:
  US 20170068847A1
• Time in Patent Office: 904 Days
• Field of Search: 725 9- 12
• CPC Class Codes: A61B 5/165 Evaluating the state of mind, e.g. depression, anxiety
• CPC Class Codes: G06Q 30/0631 Item recommendations
• CPC Class Codes: G06V 20/40 in video content extracting overlay text G06V20/62; video retrieval G06F16/70; processing of video elementary streams in video servers H04N21/234; processing of video elementary streams in video clients H04N21/44
• CPC Class Codes: G06V 40/174 Facial expression recognition
• CPC Class Codes: G06V 40/175 Static expression
• CPC Class Codes: G06V 40/176 Dynamic expression
• CPC Class Codes: G06V 40/179 metadata assisted face recognition
• CPC Class Codes: H04N 21/251 Learning process for intelligent management, e.g. learning user preferences for recommending movies details of learning user preferences for the retrieval of video data in a video database G06F16/739; computer systems using learning methods G06N3/08
• CPC Class Codes: H04N 21/252 Processing of multiple end-users' preferences to derive collaborative data
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4667 Processing of monitored end-user data, e.g. trend analysis based on the log file of viewer selections
• CPC Class Codes: H04N 21/4668 for recommending content, e.g. movies

Abstract

Analysis of mental state data is provided to enable video recommendations via affect. Analysis and recommendation is made for socially shared live-stream video. Video response is evaluated based on viewing and sampling various videos. Data is captured for viewers of a video, where the data includes facial information and/or physiological data. Facial and physiological information is gathered for a group of viewers. In some embodiments, demographic information is collected and used as a criterion for visualization of affect responses to videos. In some embodiments, data captured from an individual viewer or group of viewers is used to rank videos.

Claims

1. A computer-implemented method for affect based recommendations comprising:

playing a first media presentation to an individual on a video client;

capturing mental state data for the individual using the video client, wherein the mental state data includes video facial data from the individual during the first media presentation playing;

comparing, on an analysis server, the mental state data that was captured for the individual against a plurality of mental state event temporal signatures, wherein the mental state event temporal signatures include a shape of an intensity transition;

ranking, on the analysis server, the first media presentation relative to another media presentation based on the mental state data which was captured for the individual, wherein the ranking is for the individual;

correlating, on the analysis server, the mental state data which was captured for the individual to mental state data collected from other people who experienced the first media presentation; and

recommending, using one or more processors in at least one of the video client and the analysis server, a second media presentation to the individual based on the mental state data which was captured for the individual and the correlating, wherein the recommending the second media presentation to the individual is further based on the comparing of the mental state data to the plurality of mental state event temporal signatures.

2. The method of claim 1 further comprising matching a first event signature, from the plurality of mental state event temporal signatures, against the mental state data that was captured.

3. The method of claim 2 wherein the recommending the second media presentation is further based on the matching of the first event signature.

4. The method of claim 1 wherein the correlating is based on identifying similar likes.

5. The method of claim 1 wherein the correlating is based on identifying and using maximally dissimilar responses during part of the correlating.

6. The method of claim 1 further comprising identifying a mental state event type based on the comparing.

7. The method of claim 6 wherein the recommending of the second media presentation is further based on the mental state event type.

8. The method of claim 1 wherein the first media presentation includes a first socially shared live-stream video.

9. The method of claim 8 further comprising generating highlights for the first socially shared live-stream video, based on the mental state data that was captured.

10. The method of claim 8 wherein the first socially shared live-stream video includes an overlay with information on the mental state data that was captured.

11. The method of claim 8 wherein the mental state data that was captured for the first socially shared live-stream video is analyzed substantially in real time.

12. The method of claim 8 wherein the second media presentation includes a second socially shared live-stream video.

13. The method of claim 12 further comprising a recommendation for changing from the first socially shared live-stream video to the second socially shared live-stream video.

14. The method of claim 12 wherein the first socially shared live-stream video is broadcast to a plurality of people.

15. The method of claim 14 further comprising providing an indication to the individual that the second socially shared live-stream video is ready to be joined.

16. The method of claim 1 further comprising performing unsupervised clustering of features extracted from the mental state data.

17. The method of claim 1 further comprising analyzing the mental state data to produce mental state information.

18. The method of claim 17 wherein the analyzing of the mental state data is further based on a demographic basis.

19. The method of claim 1 wherein the ranking is based on anticipated preferences for the individual.

20. The method of claim 1 wherein the mental state data is captured from multiple people and further comprising aggregating the mental state data from the multiple people.

21. The method of claim 20 further comprising ranking the first media presentation relative to another media presentation based on the mental state data which was aggregated from the multiple people.

22. The method of claim 1 wherein the mental state data further includes physiological data or actigraphy data.

23. The method of claim 22 wherein the physiological data includes one or more of electrodermal activity, heart rate, heart rate variability, skin temperature, and respiration.

24. The method of claim 1 further comprising inferring mental states, based on the mental state data which was collected and analysis of the video facial data.

25. The method of claim 24 wherein the analysis of the video facial data is for at least brow furrows.

26. The method of claim 1 wherein the playing of the first media presentation is done on a mobile device and further comprising recording of facial images with the mobile device as part of the capturing of the mental state data.

27. The method of claim 1 wherein at least part of the mental state data collected from other people is captured while the other people view a third media presentation.

28. The method of claim 1 wherein at least part of the mental state data collected from other people is captured while the other people view the first media presentation.

29. The method of claim 1 further comprising determining a correlation vector for the individual based on the correlating.

30. The method of claim 29 further comprising analyzing a difference between the correlation vector for the individual and a correlation vector for at least one of the other people, wherein the recommending is further based on the difference.

31. The method of claim 1 wherein the plurality of mental state event temporal signatures is based on mental state data collected from the other people.

32. The method of claim 1 wherein the shape of the intensity transition included in the mental state event temporal signatures is an intensity transition from a low intensity to a peak intensity.

33. The method of claim 1 wherein the shape of the intensity transition included in the mental state event temporal signatures is an intensity transition from a peak intensity to a low intensity.

34. The method of claim 33 wherein the mental state event temporal signatures include a plurality of shapes of intensity transitions and the shape of at least one intensity transition included in the mental state event temporal signatures is an intensity transition from a low intensity to a peak intensity.

35. A computer program product stored on a non-transitory computer-readable medium for affect based recommendations, the computer program product comprising code which causes one or more processors to perform operations of:

playing a first media presentation to an individual on a video client;

capturing mental state data for the individual on the video client, wherein the mental state data includes video facial data from the individual during the first media presentation playing;

comparing, on an analysis server, the mental state data that was captured for the individual against a plurality of mental state event temporal signatures, wherein the mental state event temporal signatures include a shape of an intensity transition;

ranking, on the analysis server, the first media presentation relative to another media presentation based on the mental state data which was captured for the individual, wherein the ranking is for the individual;

correlating, on the analysis server, the mental state data which was captured for the individual to mental state data collected from other people who experienced the first media presentation; and

recommending, using at least one of the analysis server and the video client, a second media presentation to the individual based on the mental state data which was captured for the individual and the correlating, wherein the recommending the second media presentation to the individual is further based on the comparing of the mental state data to the plurality of mental state event temporal signatures.

36. A computer system for affect based recommendations comprising:

a memory for storing instructions;

one or more processors attached to the memory wherein the one or more processors are configured to;

play a first media presentation to an individual on a video client;

capture mental state data for the individual using the video client, wherein the mental state data includes video facial data from the individual during the first media presentation playing;

compare, on an analysis server, the mental state data that was captured for the individual against a plurality of mental state event temporal signatures wherein the mental state event temporal signatures include a shape of an intensity transition;

rank, on the analysis server, the first media presentation relative to another media presentation based on the mental state data which was captured for the individual, wherein the ranking is for the individual;

correlate, on the analysis server, the mental state data which was captured for the individual to mental state data collected from other people who experienced the first media presentation; and

recommend, using at least one of the video client and the analysis server, a second media presentation to the individual based on the mental state data which was captured for the individual and the correlation, wherein the recommending the second media presentation to the individual is further based on comparing of the mental state data to the plurality of mental state event temporal signatures.