Temporal supersampling for foveated rendering systems

• US 10,713,752 B2
• Filed: 02/26/2018
• Issued: 07/14/2020
• Est. Priority Date: 06/09/2017
• Status: Active Grant

• Current Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Original Assignee: Sony Interactive Entertainment Inc. (Sony Group Corp.)
• Inventors: Young, Andrew, Ho, Chris, Stafford, Jeffrey Roger
• Primary Examiner(s): Guo, Xilin
• Application Number:
  US15/905,801
• Publication Number:
  US 20180357749A1
• Time in Patent Office: 869 Days
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 11/40 Filling a planar surface by adding surface attributes, e.g. colour or texture
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
• CPC Class Codes: G09G 2310/0232 Special driving of display border areas
• CPC Class Codes: G09G 2340/0428 Gradation resolution change
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2360/08 Power processing, i.e. workload management for processors involved in display operations, such as CPUs or GPUs
• CPC Class Codes: G09G 2360/18 Use of a frame buffer in a display terminal, inclusive of the display panel
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 5/001 Arbitration of resources in a display system, e.g. control of access to frame buffer by video controller and/or main processor
• CPC Class Codes: G09G 5/391 Resolution modifying circuits, e.g. variable screen formats
• CPC Class Codes: G09G 5/42 characterised by the display of patterns using a display memory without fixed position correspondence between the display memory contents and the display position on the screen

Abstract

Methods and systems are provided for using temporal supersampling to increase a displayed resolution associated with peripheral region of a foveated rendering view. A method for enabling reconstitution of higher resolution pixels from a low resolution sampling region for fragment data is provided. The method includes an operation for receiving a fragment from a rasterizer of a GPU and for applying temporal supersampling to the fragment with the low resolution sampling region over a plurality of prior frames to obtain a plurality of color values. The method further includes an operation for reconstituting a plurality of high resolution pixels in a buffer that is based on the plurality of color values obtained via the temporal supersampling. Moreover, the method includes an operation for sending the plurality of high resolution pixels for display.

Claims

1. A method for drawing higher resolution pixels from a low resolution sampling region, comprising:

processing a fragment of a frame within a graphics pipeline when executing a video game;

applying temporal supersampling to the fragment within the low resolution sampling region over the frame and a plurality of prior frames for obtaining a plurality of color values, wherein the temporal supersampling includes sampling at different locations within the low resolution sampling region for the frame and the plurality of prior frames, wherein each of the frame and the plurality of prior frames includes low resolution pixel data for the low resolution sampling region;

reconstituting a plurality of high resolution pixels associated with the low resolution sampling region in a buffer based on the plurality of color values obtained via the temporal supersampling; and

sending, from the buffer, the plurality of high resolution pixels for presentation on a display.

2. The method of claim 1, wherein the temporal supersampling includes sampling the different locations for the low resolution sampling region as determined by pixel reprojection.

3. The method of claim 1, wherein the temporal supersampling includes sampling the different locations for the low resolution sampling region as determined by jitter.

4. The method of claim 1, wherein the plurality of high resolution pixels is greater in number than the number of the plurality of prior frames.

5. The method of claim 1, wherein the reconstituting of the plurality of high resolution pixels further includes blending of the plurality of color values.

6. The method of claim 1, wherein the plurality of high resolution pixels is associated with a native resolution of the display or a resolution that is greater than the native resolution of the display.

7. The method claim 1, wherein the display is associated with a head mounted display (HMD).

8. The method of claim 1, wherein the low resolution sampling region is associated with a peripheral region of a foveated rendering view.

9. A graphics system, comprising:

a graphics processing unit (GPU) for processing a fragment of a frame when executing a video game, the GPU applying temporal supersampling to the fragment within a low resolution sampling region over the frame and a plurality of prior frames to obtain a plurality of color values, wherein the temporal supersampling includes sampling at different locations within the low resolution sampling region for the frame and the plurality of prior frames;

a frame buffer for storing the plurality of color values generated for the frame and the plurality of prior frames that are rendered by the GPU; and

a display buffer for storing color values for a plurality of high resolution pixels in the low resolution sampling region that is reconstituted based on the plurality of color values obtained via the temporal supersampling of prior frames,

wherein the plurality of high resolution pixels is configured for presentation of the frame on a display.

10. The graphics system of claim 9, wherein the temporal supersampling defines the different locations for the low resolution sampling region as determined by pixel reprojection.

11. The graphics system of claim 9, wherein the temporal supersampling defines the different locations for the low resolution sampling region as determined by jitter.

12. The graphics system of claim 9, wherein the plurality of high resolution pixels is greater in number that the number of the plurality of prior frames.

13. The graphics system of claim 9, wherein the plurality of high resolution pixels that is reconstituted includes blending of the plurality of color values.

14. The graphics system of claim 9, wherein the plurality of high resolution pixels is associated with a native resolution of the display or a resolution that is greater than the native resolution of the display.

15. The graphics system of claim 9, wherein the display is associated with a head mounted display (HMD).

16. The graphics system of claim 9, wherein the low resolution sampling region is associated with a peripheral region of a foveated rendering view.

17. A non-transitory computer-readable storage medium storing a computer program executable by a processor-based system, comprising:

program instructions for processing a fragment of a frame within a graphics pipeline when executing a video game, at least a portion of the fragment being located within a low resolution sampling region;

program instructions for applying temporal supersampling to the fragment over the frame and a plurality of prior frames for obtaining a plurality of color values, wherein the temporal supersampling includes sampling at different locations within the low resolution sampling region for the frame and the plurality of prior frames, wherein each of the frame and the plurality of prior frames includes low resolution pixel data for the low resolution sampling region;

program instructions for reconstituting, in a buffer, a plurality of high resolution pixels associated with the low resolution sampling region, the plurality of high resolution pixels based on the plurality of color values obtained via the temporal supersampling; and

program instructions for sending, from the buffer, the plurality of high resolution pixels for presentation on a display.

18. The non-transitory computer-readable medium of claim 17, wherein the temporal supersampling defines the different locations for the low resolution sampling region over the frame and the plurality of prior frames as determined by pixel reprojection or as determined by jitter, and

wherein the plurality of high resolution pixels is associated with a native resolution of the display or a resolution that is greater than the native resolution of the display.