Determining audience state or interest using passive sensor data

• US 9,100,685 B2
• Filed: 12/09/2011
• Issued: 08/04/2015
• Est. Priority Date: 12/09/2011
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Conrad, Michael J., Hulten, Geoffrey J, Krum, Kyle J., Mendhro, Umaimah A., Remington, Darren B.
• Primary Examiner(s): FLYNN, RANDY A
• Application Number:
  US13/316,351
• Publication Number:
  US 20130152113A1
• Time in Patent Office: 1,334 Days
• Field of Search: 725 9- 10, 725/12
• US Class Current: 1/1
• CPC Class Codes: H04N 21/42201 biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user input arrangements for interaction with the human body based on nervous system activity detection G06F3/015
• CPC Class Codes: H04N 21/42203 sound input device, e.g. microphone
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44213 Monitoring of end-user related data arrangements for monitoring the users' behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/4667 Processing of monitored end-user data, e.g. trend analysis based on the log file of viewer selections

Abstract

This document describes techniques and apparatuses for determining an audience'"'"'s state or interest using passive sensor data. The techniques receive sensor data that measures an audience during presentation of a media program. The techniques, based on this sensor data, then determine a state or interest of the audience members during the media program.

Claims

1. A system comprising:

one or more processors; and

one or more memories storing instructions that, response to execution by the one or more processors, cause the one or more processors to perform operations comprising;

receiving a media type describing a particular portion of a media program;

receiving sensor data for a person, the sensor data passively sensed during presentation of the media program to the person;

determining, based on the sensor data, a state of the person during the particular portion of the media program, the state being a sad, a related talking, an unrelated talking, a disgusted, an afraid, a smiling, a scowling, a placid, a surprised, an angry, a laughing, a screaming, a clapping, a waving, a cheering, a looking-away, a looking-toward, a leaning-away, a leaning-toward, an asleep, or a departed state;

adjusting the determined state based, at least in part, on historical sensor data;

responsive to the adjusting, deriving an interest level graph based, at least in part, on the determined state of the person during the particular portion of the media program; and

providing the interest level graph and the media type describing the particular portion of the media program effective to enable measurement of a value of the media program, assessment of a potential value of a similar media program or portion thereof, control of presentation of the media program, or automatic rating, for the person, of the media program.

2. A system as described in claim 1, wherein the sensor data includes a head orientation of the person and the operation of determining the state determines the state to be the looking-away or the looking-toward state.

3. A system as described in claim 1, wherein the operation of determining the state of the person determines a numerical probability for the state or multiple numerical probabilities for multiple states, respectively.

4. A system as described in claim 1, wherein:

the sensor data includes infrared data frames and the operation of determining the state determines the state based on the infrared data frames;

the sensor data includes skeletal movement of the person and the operation of determining the state determines the state to be the clapping state;

the sensor data includes skeletal movement of the person and no or low-amplitude audio of the person and the operation of determining the state determines the state to be the waving state;

the sensor data includes skeletal orientation or movement of the person and high-amplitude audio from the person and the operation of determining the state determines the state to be the cheering state;

the sensor data includes head movement of the person, facial movement or orientation of the person, and high-amplitude audio from the person and the operation of determining the state determines the state to be the laughing state or the screaming state;

the sensor data includes facial orientation of the person and the operation of determining the state determines the state to be the sad, disgusted, afraid, smiling, placid, scowling, surprised, or angry state;

the sensor data includes skeletal movement or a thermal reading of the person and the operation of determining the state determines the state to be the departed state;

the sensor data includes audio and the operation of determining the state determines, using speech recognition, the state to be the related talking state or the unrelated talking state;

or the sensor data includes facial orientation or respiration rate of the person and the operation of determining determines the state to be the asleep state.

5. A system as described in claim 1, wherein the sensor data is received for multiple periods during the presentation of additional portions of the media program, the operation of determining determines states for each of the multiple periods, and the operations further comprising providing the states for each of the multiple periods.

6. A system as described in claim 1, the operations further comprising determining, based on the sensor data, an identity of the person and providing the identity of the person.

7. A system as described in claim 1, the operations further comprising determining, based on the sensor data, a demographic of the person and providing the demographic of the person.

8. A system as described in claim 1, the operations further comprising determining, based on the media type and the state, an interest level of the person relative to the particular portion of the media program.

9. A computer-implemented method comprising:

receiving sensor data for an audience, the sensor data passively sensed during presentation of a media program to the audience;

determining, based on a media type describing a particular portion of the media program and the sensor data for the audience passively sensed during the presentation of the particular portion of the media program to the audience, an interest level of the audience to the particular portion of the media program;

normalizing the determined interest level of the audience based on historical interest levels for a person in the audience;

producing an interest level graph that indicates the determined and normalized interest level of the audience to the particular portion of the media program; and

providing the interest level graph of the audience, the interest level graph indicating a value of the media program, usable to assess a potential value of a similar media program or portion thereof, usable to control presentation of the media program, or indicating a rating for the media program.

10. A computer-implemented method as described in claim 9, wherein the normalizing the determined interest level of the audience is further based on historical interest levels for other persons not in the audience.

11. A computer-implemented method as described in claim 9, further comprising determining, based on the sensor data, an engagement of the audience to the particular portion of the media program, and where determining the interest level of the audience is based on the engagement of the audience and the media type.

12. A computer-implemented method as described in claim 9, further comprising determining, based on the sensor data, states of one or more persons in the audience during presentation of the particular portion of the media program, and wherein the determining the interest level of the audience is based on the states of the one or more persons and the media type.

13. A computer-implemented method as described in claim 9, wherein the sensor data for the audience includes multiple sensor readings over a time in which the media program is presented to the audience and determining the interest level of the audience determines multiple interest levels over the time.

14. A computer-implemented method as described in claim 9, wherein the sensor data is received from an audience-sensing device local to the audience.

15. A computer-implemented method as described in claim 14, wherein the audience-sensing device is a forward-facing camera integrated within a display on which the media program is rendered.

16. A computer-implemented method as described in claim 15, wherein the forward-facing camera is capable of sensing facial orientation and skeletal movement of the audience, and the sensor data includes facial orientation data and skeletal movement data for a person in the audience, the skeletal movement including movement of an arm of the person.

17. A computer-implemented method as described in claim 14, wherein the audience-sensing device includes a microphone capable of sensing audio generated by the audience and a location of the audio relative to persons in the audience.

18. A computer-implemented method as described in claim 9, wherein the audience includes two or more persons and the sensor data is received with or includes information indicating portions of the sensor data attributable separately to each of the two or more persons.

19. A computer-implemented method as described in claim 9, the method performed by a System-on-Chip (SoC) integrated within a media presentation device that presents the media program to the audience, the media presentation device including an audience-sensing device from which the sensor data is received.

20. A computer-implemented method comprising:

passively sensing or receiving sensor data for an audience, the sensor data passively sensed at time periods during which a media program is presented to the audience and from an audience-sensing device capable of sensing head or skeletal orientation or movement, the sensor data including the head or skeletal orientation or movement;

determining, based on the sensor data, multiple states of at least one person in the audience for the time periods during which the media program is presented to the audience;

normalizing one of the determined multiple states based, at least in part, on historical sensor data for the at least one person in the audience;

receiving one or more media types describing respective portions included in the media program;

determining, based on the one or more media types describing respective portions included in the media program and the multiple states of the person in the audience, multiple interest levels of the person for the time periods;

generating an interest graph that includes the determined multiple interest levels of the person for the time periods; and

providing the interest graph to an advertising entity effective to enable the advertising entity to measure a value of the media program, a media creator effective to enable the media creator to assess a potential value of a similar media program, a controller effective to enable the controller to control presentation of the media program, or a rating entity effective to enable the rating entity to automatically rate the media program for the person.