Consumption of content with reactions of an individual

• US 10,045,077 B2
• Filed: 12/12/2016
• Issued: 08/07/2018
• Est. Priority Date: 01/10/2012
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Dang, Yingnong, Paolantonio, Sergio, Cao, Xiang, Lin, Xiumin, Zhang, Dongmei
• Primary Examiner(s): SALCE, JASON P
• Application Number:
  US15/376,255
• Publication Number:
  US 20170099519A1
• Time in Patent Office: 603 Days
• CPC Class Codes: G06F 2203/011 Emotion or mood input determined on the basis of sensed human body parameters such as pulse, heart rate or beat, temperature of skin, facial expressions, iris, voice pitch, brain activity patterns
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: H04N 21/42201 biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user input arrangements for interaction with the human body based on nervous system activity detection G06F3/015
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/44008 involving operations for analysing video streams, e.g. detecting features or characteristics in the video stream arrangements characterised by components specially adapted for monitoring, identification or recognition of video in broadcast systems H04H60/59
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4788 communicating with other users, e.g. chatting arrangements for providing for computer conferences, e.g. chat rooms, to substation in data switching networks H04L12/1813; distributed application using peer-to-peer [P2P] networks H04L67/104
• CPC Class Codes: H04N 7/157 defining a virtual conference space and using avatars or agents computer conference optimisation or adaptation H04L12/1827

Abstract

Techniques for obtaining a reaction of an individual to content and outputting the reaction of the individual with the content are described herein. The techniques may include obtaining the reaction of the individual as the content is displayed to the individual. The reaction may be obtained by capturing a video of the individual. The reaction of the individual and the content may be output at a same time to another individual. These techniques may also include obtaining the reaction of another individual as the content and the reaction of the individual are output to another individual.

Claims

1. A system comprising:

a rear facing camera to capture content;

one or more processors;

one or more sensors, including a forward facing camera, to capture reaction data while the content is being captured by the rear facing camera, the reaction data representative of one or more physical characteristics of the user or actions taken by the user that can be interpreted as one or more reactions of the user to the content;

computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

causing display of the content to the user during capture by the rear facing camera;

capturing the reaction data from the one or more sensors, including the forward facing camera;

analyzing the reaction data regarding the one or more reactions of the user to identify a particular reaction of the user, the particular reaction identified without receipt of external input beyond the reaction data captured from the one or more sensors;

analyzing, automatically using the one or more processors, the content to identify a section of the content that is associated with the particular reaction of the user; and

causing output of the section of the content and the particular reaction of the user.

2. The system as recited in claim 1, wherein the one or more sensors additionally comprise at least one of:

a microphone;

a heart rate monitor;

a blood pressure monitor;

or

a gaze tracking sensor.

3. The system as recited in claim 1, wherein the one or more sensors are configured to capture the reaction data by at least one of capturing a video of the user, capturing audio of the user, or monitoring a physical response of the user.

4. The system as recited in claim 1, wherein causing output of the section of the content and the particular reaction of the user comprises sending the section of the content and the particular reaction of the user to a computing device associated with another user.

5. The system as recited in claim 1, further comprising a display, and wherein:

causing output of the content to the user comprises presenting at least a portion of the content on the display; and

causing output of the section of the content and the particular reaction of the user comprises simultaneously presenting at least a portion of the section of the content and the particular reaction of the user on the display.

6. The system as recited in claim 1, wherein analyzing the reaction data regarding the one or more reactions of the user to identify the particular reaction of the user includes identifying, automatically using the one or more processors operating on the reaction data, at least one of a smile, a frown, a head motion, an arm motion, a laugh, a yell, a cry, a nerve response, a change in heart rate, or a change in blood pressure.

7. The system as recited in claim 1, wherein causing output of the section of the content and the particular reaction of the user comprises causing output of the section of the content and the particular reaction to another user that has a similar location;

interest, or age to the user.

8. A method comprising:

causing a computing device to output content to a user via a display associated with the computing device, wherein the content is simultaneously captured by a rear facing camera integrated into the computing device;

obtaining, by a sensor integrated into the computing device and while the content is being output to the user, reaction data representative of one or more physical characteristics of the user or actions taken by the user that can be interpreted as one or more reactions of the user to the content, the sensor comprising at least a forward facing camera;

analyzing the reaction data regarding the one or more reactions of the user to identify a particular reaction, the particular reaction identified without receipt of external input beyond the reaction data obtained from the sensor;

analyzing the content to identify a section of the content that is associated with the particular reaction of the user; and

causing output of the section of the content and the particular reaction of the user.

9. The method of claim 8, further comprising:

presenting a notification on the display requesting authorization from the user to obtain the one or more reactions of the user; and

receiving input, via an input device associated with the computing device, authorizing the computing device to obtain the one or more reactions of the user.

10. The method of claim 8, further comprising:

receiving additional reaction data regarding one or more additional reactions of another user, the one or more additional reactions being obtained while the content was output to the other user; and

causing output, by the computing device, of the one or more additional reactions of the other user simultaneously while causing output of the content to the user.

11. The method of claim 8, wherein obtaining the reaction data regarding the one or more reactions of the user comprises at least one of capturing a video of the user, capturing audio of the user, or monitoring a physical response of the user.

12. The method of claim 8, wherein analyzing the reaction data regarding the one or more reactions of the user to identify the particular reaction includes identifying at least one of a smile, a frown, a head motion, an arm motion, a laugh, a yell, a cry, a nerve response, a change in heart rate, or a change blood pressure.

13. The method of claim 8, wherein causing output of the section of the content and the particular reaction of the user comprises sending the section of the content and the particular reaction of the user to another computing device associated with a another user.

14. The method of claim 13, further comprising:

receiving one or more additional reactions of the another user obtained v the other computing device while the content was output to the another user; and

outputting, by the computing device, the one or more additional reactions of the another user.

15. A computing device comprising:

a rear facing camera to capture content;

one or more processors;

one or more sensors, including a forward facing camera, to capture reaction data while content is being output to a user, the reaction data representative of one or more physical characteristics of the user or actions taken by the user that can be interpreted as one or more reactions of the user to the content;

a display;

a speaker; and

computer-readable media storing computer-executable instructions that, when executed by the one or more processors, cause the one or more processors to perform acts comprising;

causing output of the content by at least one of the display or the speaker, the content captured by the rear facing camera;

capturing, using the one or more sensors, the reaction data as the content is output,

analyzing the reaction data regarding the one or more reactions of the user to identify a particular reaction of the user, the particular reaction identified without receipt of external input beyond the reaction data captured from the one or more sensors;

analyzing the content to identify a section of the content that is associated with the particular reaction of the user; and

causing output of the section of the content and the particular reaction of the user.

16. The computing device of claim 15, the acts further comprising receiving, from another computing device, additional reaction data regarding one or more additional reactions of another user, the one or more additional reactions being obtained while the content was output to the other user.

17. The computing device of claim 16, the acts further comprising causing output, by at least one of the display or the speaker, of the one or more additional reactions of the other user while causing the content to be output to the user.

18. The computing device of claim 15, wherein the one or more sensors additionally include at least one of:

a microphone;

a heart rate monitor;

a blood pressure monitor;

or

a gaze tracking sensor.

19. The computing device of claim 15, wherein analyzing the reaction data regarding the one or more reactions of the user to identify the particular reaction of the user includes identifying at least one of a smile, a frown, a head motion, an arm motion, a laugh, a yell, a cry, a nerve response, a change in heart rate, or a change in blood pressure.