Device for acquisition of viewer interest when viewing content

• US 9,596,508 B2
• Filed: 03/15/2013
• Issued: 03/14/2017
• Est. Priority Date: 03/15/2013
• Status: Active Grant

• Current Assignee: Sony Corporation (Sony Group Corp.), Sony Network Entertainment International LLC (Sony Group Corp.)
• Original Assignee: Sony Corporation (Sony Group Corp.), Sony Interactive Entertainment LLC (Sony Group Corp.)
• Inventors: McCoy, Charles, Pedlow, Leo, Xiong, True
• Primary Examiner(s): Chin, Ricky
• Application Number:
  US13/832,510
• Publication Number:
  US 20140282646A1
• Time in Patent Office: 1,460 Days
• US Class Current: 1/1
• CPC Class Codes: G06V 40/18 Eye characteristics, e.g. of the iris
• CPC Class Codes: H04N 13/332 Displays for viewing with the aid of special glasses or head-mounted displays [HMD]
• CPC Class Codes: H04N 21/42201 biosensors, e.g. heat sensor for presence detection, EEG sensors or any limb activity sensors worn by the user input arrangements for interaction with the human body based on nervous system activity detection G06F3/015
• CPC Class Codes: H04N 21/4223 Cameras H04N23/00 takes precedence
• CPC Class Codes: H04N 21/44213 Monitoring of end-user related data arrangements for monitoring the users' behaviour or opinions in broadcast systems H04H60/33
• CPC Class Codes: H04N 21/44218 Detecting physical presence or behaviour of the user, e.g. using sensors to detect if the user is leaving the room or changes his face expression during a TV program methods or arrangements for recognising human body or animal bodies or body parts G06V40/10; methods or arrangements for acquiring or recognising human faces, facial parts, facial sketches, facial expressions G06V40/16; methods or arrangements for recognising movements or behaviour G06V40/20; arrangements for identifying users in broadcast systems H04H60/45

Abstract

In order to acquire a viewer'"'"'s enjoyment and interest when viewing three-dimensional contents on a consumer electronic device, a viewer'"'"'s eyes are monitored in real-time. Biometric data from the monitoring can be used for delivery of highly accurate and personalized viewing experiences for the viewer.

Claims

1. Glasses used for viewing an image on an associated internet protocol (IP) display device and for communication over the internet with at least an external server, comprising:

a frame; and

sensors that collect biometric data of a user of the glasses; and

a wireless transmitter that wirelessly transmits the biometric data to a consumer electronic device that controls the IP display device, wherein the consumer electronic device transmits the biometric data to the external server that connects with the consumer electronic device via the internet, the external server storing a user profile including biometric information of the user and being capable of overriding display settings of the IP display device via the internet,

wherein the sensors detect the user'"'"'s gaze location on an image displayed on a display screen of the IP display device and generate biometric data associated with the detected gaze location such that the user'"'"'s interests in an object in the displayed image is determined by correlating the biometric data of the gaze location with metadata of the image, the metadata including information about locations of objects in the image,

wherein personalized content, which is to be displayed to the user, is selected by the external server based on (a) the user profile stored on the external server, (b) identifying the user through eye recognition by communication with the external server, and (c) user preferences from the external server,

wherein the glasses provide an entry point to the personalize content from the external server for each user of the glasses viewing the associated IP display device.

2. The glasses according to claim 1, wherein the sensors are embedded in the frame.

3. The glasses according to claim 1, wherein the sensors are pluggable into the frame.

4. The glasses according to claim 1, wherein the sensors monitor the eye gaze.

5. The glasses according to claim 1, wherein the sensors monitor eye fixation.

6. The glasses according to claim 1, wherein the sensors monitor eye movement.

7. The glasses according to claim 1, wherein the sensors detect pupil dilation.

8. The glasses according to claim 1, wherein the biometric data obtained is for enabling display of personalized content on the IP display device.

9. The glasses according to claim 1 further comprising a plurality of sensors for monitoring additional biometric information.

10. The glasses according to claim 9, wherein the sensors include at least one of an eye tracking sensor, a spatial orientation sensor, a temperature sensor, audio sensor, microphone sensor, cerebral wavelength sensor, ambient light sensor, ambient sound sensor, heart sensor and a galvanic sensor.

11. The glasses according to claim 10, where the spatial orientation sensor includes one of a gyroscope, an accelerometer and a camera to detect one of a (1) screen of the display device and (2) visible or invisible signals emitted for detection by the glasses.

12. The glasses according to claim 10, wherein the ambient light sensor is for normalizing pupil dilation levels.

13. The glasses according to claim 10, wherein the ambient sound sensor is for determining user reactions to environmental distractions.

14. The glasses according to claim 1, wherein the sensors self calibrate passively on where the user'"'"'s eyes are focused on the IP display device over time to determine one of the user'"'"'s reaction and interest level.

15. The glasses according to claim 1, wherein the biometric data is used to automatically determine the user'"'"'s profile.

16. The glasses according to claim 1, wherein the biometric data is used to create a new user'"'"'s profile.

18. The glasses according to claim 1 wherein eye recognition of the user is based on visual patterns in the pupils of the user.

17. A method of obtaining biometric data using glasses used for (a) viewing an image on an associated internet protocol (IP) display device and (b) communication with at least an external server comprising the steps of:

providing the glasses with sensors; and

unobtrusively collecting biometric data of a user of the glasses; and

wirelessly transmits the biometric data to a consumer electronic device that controls the IP display device, wherein the consumer electronic device transmits the biometric data to the external server that connects with the consumer electronic device via the internet, the external server storing a user profile including biometric information of the user and being capable of overriding display settings of the IP display device via the internet,

wherein the collecting step detects the user'"'"'s gaze location on a frame of an image displayed on a display screen of the IP display device and generates biometric data associated with the detected gaze location such that the user'"'"'s interests in an object in the frame of the displayed image is determined by correlating the biometric data of the gaze location with metadata of the frame of the image, the metadata including information about locations of objects in the image,

wherein personalized content, which is to be displayed to the user, is selected by the external server based on (a) the user profile stored on the external server, (b) identifying user through eye recognition by communication with the external server, and (c) user preferences from the external server,

wherein the glasses provide an entry point to the personalize content from the external server for each user of the glasses viewing the associated display device.

19. The method according to claim 17 further comprising the step of monitoring eye gaze.

20. The method according to claim 17 further comprising the step of monitoring eye fixation.

21. The method according to claim 17 further comprising the step of monitoring eye movement.

22. The method according to claim 17 further comprising the step of monitoring pupil dilation.

23. The method according to claim 17 further comprising the step of enabling display of personalized content on the IP display device for the user based on the biometric data.

24. The method according to claim 17 further comprising the step of monitoring additional biometric information via a plurality of sensors.

25. The method according to claim 24 further comprising the step of monitoring at least a user'"'"'s eye biometrics, spatial orientation, temperature, audio, cerebral wavelength, ambient light, ambient sound, heartbeat and galvanic activity.

26. The method according to claim 25, wherein monitoring spatial orientation includes detecting a screen of the display device and detecting visible or invisible signals emitted for detection by the glasses.

27. The method according to claim 25, wherein monitoring ambient light is for normalizing pupil dilation levels.

28. The method according to claim 25, wherein monitoring ambient sound is for determining user reactions to environmental distractions.

29. The method according to claim 17, wherein the biometric data is obtained by passive self calibration on where the user'"'"'s eyes are focused on the IP display device over time to determine one of the user'"'"'s reaction and internet level.

30. The method according to claim 17 further comprising the step of automatically determining a user'"'"'s profile based on the biometric data.

31. The method according to claim 17 further comprising the step of using the biometric data to create a new user'"'"'s profile.

32. A system for acquiring viewer interest when viewing content on an associated display device and for communication over the interne, with at least a server comprising:

glasses containing sensors that collect biometric data of a user of the glasses;

the display device displays to the user;

a consumer electronic device that controls the display device; and

the server that stores a user profile including biometric information of the user and is capable of overriding display settings of the IP display device via the Internet;

wherein the glasses include a wireless transmitter that wirelessly transmits the biometric data to the consumer electronic device,

wherein the sensors detect the user'"'"'s gaze location on a frame of an image displayed on a display screen of the display device and generate biometric data associated with the detected gaze location such that the user'"'"'s interests in an object in the frame of the displayed image is determined by correlating the biometric data of the gaze location with metadata of the frame of the image, the metadata including information that indicate locations of objects in the image,

wherein the server determines, based on correlating the biometric data from the glasses to a particular time and location of an object displayed in an image on the display device, a user profile of the user'"'"'s preferences and interests based on the biometric data received,

wherein personalized content and commercials are selected by the server based on (a) the user profile stored on the server, (b) identifying the user through eye recognition by communication with the glasses and (c) user preferences from the server; and

wherein the display device displays the personalized content and personalized commercials consistent with the user profile of the user'"'"'s interests and preferences,

wherein the glasses provide an entry point to the personalize content from the server for each user of the glasses viewing the associated display device.

33. The system as claimed in claim 32, wherein the server stores user preferences based on the biometric data, and controls the display device based thereon, wherein the user preferences determined from the received biometric data include (a) degree of interest based on gaze, fixation, and eye movements/paths and (b) satisfaction levels based on gaze, fixation, eye movement/paths and pupil dilations.