Automatic passive and anonymous feedback system

• US 8,667,519 B2
• Filed: 11/12/2010
• Issued: 03/04/2014
• Est. Priority Date: 11/12/2010
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Small, Sheridan Martin, Fuller, Andrew, Bar-Zeev, Avi, Perez, Kathryn Stone
• Primary Examiner(s): RABOVIANSKI, JIVKA A
• Application Number:
  US12/945,691
• Publication Number:
  US 20120124604A1
• Time in Patent Office: 1,208 Days
• Field of Search: 725/12, 725/9, 725/10, 725/25, 725/42, 725/86, 725/115
• US Class Current: 725/12
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44213 Monitoring of end-user related data arrangements for monitoring the users' behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45
• CPC Class Codes: H04N 21/6582 Data stored in the client, e.g. viewing habits, hardware capabilities, credit card number arrangements where receivers interact with the broadcast H04H20/38

Abstract

A system for generating passive and anonymous feedback of multimedia content viewed by users is disclosed. The multimedia content may include recorded video content, video-on-demand content, television content, television programs, advertisements, commercials, music, movies, video clips, and other on-demand media content. One or more of the users in a field of view of a capture device connected to the computing device are identified. An engagement level of the users to multimedia content being viewed by the users is determined by tracking movements, gestures, postures and facial expressions performed by the users. A report of the response to viewed multimedia content is generated based on the movements, gestures, postures and facial expressions performed by the users. The report is provided to rating agencies, content providers and advertisers. In one embodiment, preview content and personalized content related to the viewed multimedia content is received from the content providers and advertisers based on the report. The preview content and personalized content are displayed to the users.

Claims

1. A computer-implemented method for generating passive and anonymous feedback of multimedia content viewed by users, comprising the computer-implemented steps of:

receiving and displaying multimedia content associated with a current broadcast;

identifying one or more of the users in a field of view of a capture device connected to a computing device, the identifying comprising uniquely identifying the one or more users based on capturing at least one of a visual image, and a depth image associated with the one or more users;

tracking movements, gestures and postures performed by the one or more users in the field of view by a gesture recognition engine based on a three-dimensional skeletal model which models body parts as joints and bones;

tracking facial expressions performed by the one or more users viewing the multimedia content by a facial recognition engine based on stored facial expression filters,

tracking vocal responses from the one or more users while viewing the multimedia content,

automatically determining an engagement level of the one or more users to the multimedia content being viewed based on the movements, gestures, postures, the facial expressions, and the vocal responses performed by the one or more users in the field of view;

automatically generating a report of a response to viewed multimedia content, for the one or more users, based on the movements, gestures, postures, the facial expressions, and the vocal responses performed by the one or more users, the automatically tracking and automatically generating are performed by the computing device; and

transmitting the report of the response to the viewed multimedia content to a remote computing system.

2. The computer-implemented method of claim 1, wherein tracking the movements, gestures and postures performed by the one or more users further comprises:

detecting whether a user moves away from the field of view, stays within the field of view, faces an audio visual device connected to the computing device, leans forward and turns away from the audio visual device while viewing the multimedia content.

3. The computer-implemented method of claim 1,

wherein the facial expressions comprise one or more of smiles, laughter, cries, frowns, yawns and applauses from the one or more users.

4. The computer-implemented method of claim 1 further comprising:

polling responses to viewed multimedia content from multiple users, aggregating the responses and transmitting the aggregated responses of the multiple users to the remote computing system for analysis.

5. The computer-implemented method of claim 4, wherein determining the engagement level of the one or more users further comprises:

determining one of a positive, satisfactory or negative engagement level to the multimedia content viewed by the one or more users, based on the gestures, postures, movements facial expressions and vocal responses performed by the one or more users viewing the multimedia content.

6. The computer-implemented method of claim 1, wherein automatically generating a report of a response to viewed multimedia content comprises generating a user-specific report of a response to a viewed program, wherein the user-specific report further comprises:

one or more time intervals that comprise the length of a duration of the program viewed by the one or more users, the movements, gestures, postures, vocal responses and facial expressions performed by the one or more users during each time interval and the one or more users engagement level to the viewed program during each time interval.

7. The computer-implemented method of claim 1, wherein transmitting the report of the response to a remote computing system further comprises:

providing a user-specific report of the response to the viewed multimedia content to at least one or more of rating agencies, content providers and advertisers.

8. The computer-implemented method of claim 7, further comprising:

receiving preview content related to the viewed multimedia content based on the user-specific report and displaying the preview content to the one or more users via an audio visual device connected to the computing device.

9. The computer-implemented method of claim 7, further comprising:

receiving personalized content related to the viewed multimedia content based on the user-specific report and displaying the personalized content to the one or more users via an audio visual device connected to the computing device.

10. The computer-implemented method of claim 1, further comprising:

receiving at least one of preview content or personalized content related to a viewed program being viewed for the one or more users based on detecting the one or more users presence in the field of view of the capture device which is communicatively coupled to the computing device, the type of program being viewed by the one or more users, demographic information related to the one or more users and a program viewing history related to the one or more users.

11. The computer-implemented method of claim 1 further comprising:

the transmitting the report of the response to the viewed multimedia content to a remote computing system comprises providing the report to at least one or more of rating agencies, content providers and advertisers and receiving preview content related to the viewed multimedia content based on the report; and

the transmitting the report of the response to the viewed multimedia content to a remote computing system comprises providing the report to at least one or more of rating agencies, content providers and advertisers and receiving personalized content related to the viewed multimedia content based on the report.

12. An apparatus for generating passive and anonymous feedback of multimedia content viewed by users, comprising:

a depth camera;

a computing device connected to the depth camera to receive multimedia content associated with a current broadcast, identify one or more users in a field of view of a capture device, track movements, gestures, and postures performed by the one or more users in the field of view based a three-dimensional skeletal model which models body parts as joints and bones and depth images from the depth camera, track facial expressions of the one or more users in the field of view by a facial recognition engine based on stored facial expression filters, track vocal responses from the one or more users while viewing the multimedia content, determine an engagement level of the one or more users to viewed multimedia content based on tracking the movements, gestures, postures, vocal responses and facial expressions performed by the one or more users and receive preview content and personalized content related to the viewed multimedia content for the one or more users based on the engagement level of the one or more users to the viewed multimedia content.

13. The apparatus of claim 12, wherein:

the computing device generates a user-specific report of a response to the viewed multimedia content for the one or more users based on the movements, gestures, postures, vocal responses and facial expressions performed by the one or more users, provides the user-specific report of the response to the viewed multimedia content to a remote computing system and receives preview content and personalized content related to the viewed multimedia content for the one or more users, from the remote computing system.

14. The apparatus of claim 13, wherein:

the computing device receives a user eligibility to participate in a preview study related to the viewed multimedia content from the remote computing system, wherein the user eligibility is determined based on the user-specific report of the response to the viewed multimedia content.

15. The apparatus of claim 14, wherein:

the user eligibility is further determined based on at least one of a percentage of program episodes associated with the multimedia content viewed by the one or more users and an average response to each program episode viewed by the one or more users.

16. The apparatus of claim 14, wherein:

the computing device provides the one or more users with an option to participate in the preview study, receives an opt-in by the one or more users to the participate in the preview study and receives the preview content for the viewed multimedia content from the remote computing system.

17. The apparatus of claim 12, further comprising:

an audio visual device connected to the computing device, the audio visual device displays at least one of the preview content and the personalized content to the one or more users, the computing device determines the engagement level of the one or more users while viewing the preview content and the personalized content by tracking the movements, gestures, postures and facial expressions of the one or more users, provides the engagement level to one or more content providers and advertisers for analysis and receives at least one of new preview content, new personalized content, modified preview content or modified personalized content from the content providers and advertisers, based on the analysis.

18. One or more processor readable storage devices having processor readable code embodied on said one or more processor readable storage devices, the processor readable code for programming one or more processors to perform a method comprising:

tracking movements, gestures and postures performed by the one or more users in the a field of view of a capture device communicatively coupled to the one or more processors based on a three-dimensional skeletal model which models body parts as joints and bones during displaying of multimedia content;

tracking vocal responses from the one or more users while viewing the multimedia content,

automatically determining an engagement level of the one or more users to the multimedia content being viewed by the one or more users based on the tracking the movements, gestures, vocal responses and postures performed by the one or more users;

automatically generating a report about the response of the one or more users to the multimedia content based on the tracking the movements, gestures, vocal responses and postures performed by the one or more users;

transmitting the report to a remote computing system;

receiving personalized content related to the multimedia content for the one or more users based on the transmitted report; and

causing the personalized content to be displayed.

19. One or more processor readable storage devices according to claim 18, wherein tracking the movements, gestures, vocal responses and postures performed by the one or more users further comprises:

detecting whether a user turns away from an audio visual device displaying the multimedia content.