Attention-based rendering and fidelity

• US 10,310,583 B2
• Filed: 07/25/2017
• Issued: 06/04/2019
• Est. Priority Date: 08/29/2013
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment LLC (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment America LLC (Sony Group Corp.)
• Inventors: Timm, Paul, Cevallos, Andres Ramos, Halvorson, Ryan
• Primary Examiner(s): Haley, Joseph R
• Assistant Examiner(s): Frank, Emily J
• Application Number:
  US15/659,254
• Publication Number:
  US 20170357308A1
• Time in Patent Office: 679 Days
• Field of Search: None
• CPC Class Codes: G06F 1/3231 Monitoring the presence, absence or movement of users
• CPC Class Codes: G06F 1/3265 Power saving in display device
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/015 Input arrangements based on nervous system activity detection, e.g. brain waves [EEG] detection, electromyograms [EMG] detection, electrodermal response detection
• CPC Class Codes: G09G 2330/021 Power management, e.g. power saving
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/363 Graphics controllers
• CPC Class Codes: G09G 5/391 Resolution modifying circuits, e.g. variable screen formats
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Methods and systems for attention-based rendering on an entertainment system are provided. A tracking device captures data associated with a user, which is used to determine that a user has reacted (e.g., visually or emotionally) to a particular part of the screen. The processing power is increased in this part of the screen, which increases detail and fidelity of the graphics and/or updating speed. The processing power in the areas of the screen that the user is not paying attention to is decreased and diverted from those areas, resulting in decreased detail and fidelity of the graphics and/or decreased updating speed.

Claims

1. A method for attention-based rendering, the method comprising:

detecting via one or more sensors when a user is exhibiting a reaction at a point in time during display of content on a display screen, wherein the sensors include at least an eye-tracking device that detects eye-tracking data corresponding to the point in time when the user reaction is detected, the detected eye-tracking data indicative of reflected light from one or more eyes of the user at the point in time during display of the content on the display screen;

analyzing the eye-tracking data corresponding to the point in time when the user reaction is detected;

identifying a gaze direction based on a vector of the reflected light and an eye rotation based on changes in the reflected light;

locating a focus area on the display screen corresponding to the time when the user reaction is detected, wherein identifying a location of the focus area is based on the identified gaze direction and eye rotation; and

increasing processing power used to render the located focus area relative to processing power used to render a remainder of the display screen based on the detected user reaction while the user is currently focusing on the located focus area.

2. The method of claim 1, wherein the eye-tracking device is a camera that is integrated with a peripheral device to an electronic entertainment system.

3. The method of claim 1, wherein identifying the gaze direction comprises:

using light to create corneal reflections from a center of a pupil of the eyes; and

identifying the vector between the center of the pupil and the created corneal reflections.

4. The method of claim 1, further comprising calibrating the eye-tracking device for the user prior to receiving the eye-tracking data regarding the eyes of the user.

5. The method of claim 1, wherein the eye-tracking data further includes information regarding retinal blood vessels.

6. The method of claim 1, wherein the increased processing power is diverted from processing power used to render at least one portion of the remainder of the display screen.

7. The method of claim 1, further comprising initiating a power-save mode that shuts down the display screen when the gaze direction is no longer directed to the display screen.

16. The method of claim 1, wherein the processing power used to render the located focus area is increased in real-time.

17. The method of claim 6, wherein diverting the processing power is further based on identifying that the user is not currently focusing on the at least one portion of the remainder of the display screen.

18. The method of claim 17, wherein identifying that the user is not currently focusing on the at least one portion is based on identifying the at least one portion corresponds to a peripheral vision of the user.

19. The method of claim 1, wherein the display of content on the display screen is rendered at a predetermined standard level prior to detection of the user reaction, and wherein increasing the processing power includes rendering the located focus area at a level that is higher than the standard level.

20. The method of claim 19, wherein increasing the processing power includes rendering the remainder of the display screen at a level that is lower than the standard level.

8. A system for attention-based rendering, the system comprising:

a display screen;

a graphics processing unit that generates a display of content at a point in time, wherein the generated display is displayed on the display screen;

one or more sensors that detect that a user is exhibiting a reaction at the point in time to the content displayed on the display screen, wherein the sensors include at least a tracking device that detects eye-tracking data corresponding to the point in time when the user reaction is detected, the detected eye-tracking data of the user indicative of reflected light from one or more eyes of the user at the point in time during display of the content on the display screen;

a memory; and

a processor that executes instructions stored in the memory, wherein execution of the instructions by the processor;

analyzes the eye-tracking data corresponding to the point in time when the user reaction is detected,

identifies a gaze direction based on a vector of the reflected light and an eye rotation based on changes in the reflected light,

locates a focus area on the display screen corresponding to the time when the user reaction is detected, wherein identifying a location of the focus area is based on the identified gaze direction and eye rotation, and

increases processing power used to render the located focus area relative to processing power used to render a remainder of the display screen based on the detected user while the user is currently focusing on the located focus area.

9. The system of claim 8, wherein the eye-tracking device is a camera that is integrated with a peripheral device to an electronic entertainment system.

10. The system of claim 8, wherein the eye-tracking device uses light to create corneal reflections from a center of a pupil of the eyes, and wherein the processor identifies the gaze direction by calculating the vector between the center of the pupil and the created corneal reflections.

11. The system of claim 8, wherein the processor further calibrates the eye-tracking device for the user prior to receiving the eye:

tracking data regarding the eyes of the user.

12. The system of claim 8, wherein the eye-tracking data further includes information regarding retinal blood vessels.

13. The system of claim 8, wherein the increased processing power is diverted from processing power used to render at least one portion of the remainder of the display screen.

14. The system of claim 8, wherein the processor further initiates a power-save mode that shuts down the display device when the gaze direction is no longer directed to the display screen.

15. A non-transitory computer-readable storage medium having embodied thereon a program, the program being executable by a processor to perform a method for attention-based rendering, the method comprising:

detecting via one or more sensors when a user is exhibiting a reaction at a point in time during display of content on a display screen, wherein the sensors include at least an eye-tracking device that detects eye-tracking data corresponding to the point in time when the user reaction is detected, the eye-tracking data indicative of reflected light from one or more eyes of the user at the point in time during display of the content on the display screen;

analyzing the eye-tracking data corresponding to the point in time when the user reaction is detected;

identifying a gaze direction based on a vector of the reflected light and an eye rotation based on changes in the reflected light;

locating a focus area on the display screen corresponding to the time when the user reaction is detected, wherein identifying a location of the focus area is based on the identified gaze direction and eye rotation; and

increasing processing power used to render the located focus area relative to processing power used to render a remainder of the display screen based on the detected user reaction while the user is currently focusing on the located focus area.