Real-time user adaptive foveated rendering

• US 10,720,128 B2
• Filed: 01/29/2019
• Issued: 07/21/2020
• Est. Priority Date: 03/31/2016
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Young, Andrew Roger, Stafford, Jeffrey
• Primary Examiner(s): Polo, Gustavo
• Application Number:
  US16/261,476
• Publication Number:
  US 20190156794A1
• Time in Patent Office: 539 Days
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G02B 27/0101 characterised by optical features G02B27/0172 takes precedence
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 1/3265 Power saving in display device
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 5/391 Resolution modifying circuits, e.g. variable screen formats
• CPC Class Codes: Y02D 10/00 Energy efficient computing, e.g. low power processors, power management or thermal management

Abstract

Foveated rendering based on user gaze tracking may be adjusted to account for the realities of human vision. Gaze tracking error and state parameters may be determined from gaze tracking data representing a user'"'"'s gaze with respect to one or more images presented to a user. Adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to a user may be generated based on the one or more gaze tracking error or state parameters. Foveated image data representing one or more foveated images may be generated with the adjusted foveation data. The foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest. The foveated images may then be presented to the user.

Claims

1. A graphics processing method comprising:

determining one or more gaze tracking parameters from gaze tracking data representing a user'"'"'s gaze with respect to one or more images presented to a user, wherein the gaze tracking parameters include one or more gaze tracking state parameters, wherein determining the one or more gaze tracking state parameters from the gaze tracking data includes determining one or metrics related to a saccade of a user t;

generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking parameters including the one or more gaze tracking error parameters;

generating foveated image data representing one or more foveated images using the adjusted foveation data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail in one or more other regions outside the one or more regions of interest; and

transmitting or storing the foveated image data or presenting the one or more foveated images to the user.

2. The method of claim 1, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user'"'"'s eye with respect to one or more axes.

3. The method of claim 1, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a rate of rotation of a user'"'"'s eye with respect to one or more axes and determining an error in a fixation of the user'"'"'s eye on a region of interest of the one or more regions of interest.

4. The method of claim 1, wherein determining one or more gaze tracking error parameters from the gaze tracking data includes determining a confidence interval regarding the current and/or predicted future gaze position.

5. The method of claim 1, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining the metrics of a blink of a user.

6. The method of claim 1, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining a transition in the gaze direction of a user as a result of a change in the depth of field of the presented image.

7. The method of claim 1, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining whether a user is color blind.

8. The method of claim 1, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining a rate of rotation of a user'"'"'s eye with respect to one or more axes and determining the level of gaze stability of the user'"'"'s eye.

9. The method of claim 1, wherein determining one or more gaze tracking parameters from the gaze tracking data includes determining a rate of rotation of a user'"'"'s eye with respect to one or more axes and determining whether the user'"'"'s eye movement is a precursor to the user'"'"'s head movement.

10. The method of claim 1, wherein the adjusted foveation data includes data characterizing a particular region of interest of the one or more regions of interest, wherein the data characterizing the particular region of interest includes a foveation region and a transition region, wherein the foveation region is characterized by a higher level of detail than the transition region and the transition region is characterized by a higher level of detail than regions of a corresponding image outside the particular region of interest.

11. The method of claim 10, wherein generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest includes adjusting a size of the transition region.

12. The method of claim 1, wherein the adjusted foveation data includes data used to determine geometric density in the one or more regions of interest.

13. The method of claim 1, wherein the adjusted foveation data includes data used to determine pixel resolution in the one or more regions of interest.

14. The method of claim 1, wherein the adjusted foveation data includes data used to determine pixel shading quality in the one or more regions of interest.

15. The method of claim 1, further comprising creating a standard 2D compliant image from the foveated image data for presentation on additional displays.

16. A system, comprising:

a processor;

a memory; and

computer-readable instructions embodied in the memory, the computer-readable instructions being configured to implement a graphics processing method when executed, the graphics processing method comprising;

determining one or more gaze tracking parameters from gaze tracking data representing a user'"'"'s gaze with respect to one or more images presented to a user, wherein the gaze tracking parameters include one or more gaze tracking state parameters, wherein determining the one or more gaze tracking state parameters from the gaze tracking data includes determining one or metrics related to a saccade of the user;

generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking parameters;

generating foveated image data representing one or more foveated images using the adjusted foveation data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest; and

presenting the one or more foveated images to the user.

17. A non-transitory computer-readable medium having computer-readable instructions embodied therein, the computer-readable instructions being configured to implement a graphics processing method when executed, the graphics processing method comprising:

determining one or more gaze tracking parameters from gaze tracking data representing a user'"'"'s gaze with respect to one or more images presented to a user, wherein the gaze tracking parameters include one or more gaze tracking state parameters, wherein determining the one or more gaze tracking state parameters from the gaze tracking data includes determining one or metrics related to a saccade of the user;

generating adjusted foveation data representing an adjusted size and/or shape of one or more regions of interest in one or more images to be subsequently presented to the user based on the one or more gaze tracking error parameters;

generating foveated image data representing one or more foveated images using the adjusted foveation data, wherein the one or more foveated images are characterized by level of detail within the one or more regions of interest and lower level of detail outside the one or more regions of interest; and

transmitting or storing the foveated image data or presenting the one or more foveated images to the user.