Motion based adaptive rendering

• US 9,928,610 B2
• Filed: 06/18/2015
• Issued: 03/27/2018
• Est. Priority Date: 06/27/2014
• Status: Active Grant

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Golas, Abhinav, Ramani, Karthik, Cheng, Christopher T., Brothers, John W., Zhang, Liangjun, Abraham, Santosh, Chow, Ki Fung
• Primary Examiner(s): Sajous, Wesner
• Application Number:
  US14/743,700
• Publication Number:
  US 20150379727A1
• Time in Patent Office: 1,013 Days
• CPC Class Codes: G06T 1/60 Memory management
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 11/40 Filling a planar surface by adding surface attributes, e.g. colour or texture
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 2200/12 involving antialiasing
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/20192 Edge enhancement; Edge preservation
• CPC Class Codes: G06T 5/00 Image enhancement or restoration
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06T 7/32 using correlation-based methods
• CPC Class Codes: G06T 7/337 involving reference images or patches
• CPC Class Codes: G09G 5/363 Graphics controllers

Abstract

An apparatus, system and method is provided to determine a motion of pixels in local regions of a scene, classify the motion into a speed category, and make decisions on how to render blocks of pixels. In one implementation the motion in a tile is classified into at least three different speed regimes. If the pixels in a tile are in a quasi-static speed regime, a determination is made whether or not to reuse a fraction of pixels from the previous frame. If the pixels are determined to be in a high speed regime, a decision is made whether or not a sampling rate may be reduced.

Claims

1. A method of graphics processing utilizing a graphics processing unit (GPU), the method comprising:

determining, by the GPU, on a tile-by-tile basis, a relative speed of objects as measured in screen space co-ordinates or pixels in a current frame relative to a previous frame, the frames corresponding to images to be displayed on a display device;

classifying, by the GPU, each tile within the current frame into one of at least three speed categories based on the determined relative speed, the at least three categories including a quasi-static speed category, a medium speed category, and a high speed category;

making, by the GPU, a sampling decision for each tile, based at least in part on the speed category associated with each tile, including whether the tile is to be sampled at a full resolution sampling rate of at least one sample per pixel in the current frame or sampled at a lower rate in the current frame; and

rendering, by the GPU, tiles of the current frame.

2. The method of claim 1, wherein for a tile classified in the quasi-static speed category the method further comprises reusing pixel data from the previous frame by mapping a location of at least one pixel in the current frame to a corresponding pixel in the previous frame and copying pixel data for the corresponding pixel of the previous frame to the current frame.

3. The method of claim 2, wherein a subset of the pixels from a tile of the previous frame are copied to a tile of the current frame based on a pre-selected ratio of rendered to unrendered pixels.

4. The method of claim 2, further comprising performing a discrepancy check with respect to pixel color values of the previous frame as a condition for reusing pixel data from the previous frame.

5. The method of claim 1, wherein the speed is determined from a difference in vertex position data for rendered geometry between the current frame and a previous frame.

6. The method of claim 1, wherein, for a tile classified in the high speed category, making a sampling decision includes selecting a sampling pattern in which a number of rendered samples is less than the number of pixels associated with the tile.

7. The method of claim 1, further comprising performing edge detection in depth/Z and wherein the making a sampling decision for each tile is further based on whether edges are detected within individual tiles.

8. The method of claim 7, wherein:

for a tile in the high speed category not including an edge, selecting a sampling pattern in which a number of samples is less than the number of pixels associated with the tile; and

for a tile in the high speed category including an edge, selecting a sampling pattern in which a number of samples is at least equal to one sample per pixel.

9. The method of claim 1, wherein making a sampling decision further comprises making a decision for each tile whether or not to perform super-sampling.

10. The method of claim 9, wherein super-sampling is performed for a tile including an edge.

11. A method of graphics processing utilizing a graphics processing unit (GPU), the method comprising:

determining, by the GPU, on a tile-by-tile basis, a speed of objects as measured in screen space pixels in a current frame relative to a previous frame, the frames to be displayed on a display device;

detecting, by the GPU, on a tile-by-tile basis, whether a tile includes an edge;

classifying, by the GPU based on the determined speed, each tile within the current frame into one of at least three speed categories including a quasi-static speed category, an intermediate speed regime, and at least one high speed regime;

making, by the GPU, a sampling rate decision for individual tiles based on the speed category associated with a tile and whether the tile includes an edge, the sampling decision including determining whether the tile is to be sampled at a full resolution sampling rate of at least one sample per pixel in the current frame or sampled at a lower rate in the current frame; and

rendering, by the GPU, tiles of the current frame.

12. The method of claim 11, wherein for a tile classified in the quasi-static speed category the method further comprises reusing pixel data from the previous frame by mapping a location of at least one pixel in a current frame to a corresponding pixel in the previous frame and copying pixel data for the corresponding pixel of the previous frame to the current frame.

13. The method of claim 12, wherein a subset of the pixels from a tile of the previous frame are copied to a tile of the current frame based on a pre-selected ratio of rendered to unrendered pixels.

14. The method of claim 12, further comprising performing a discrepancy check with respect to pixel color values of the previous frame as a condition for reusing pixel data from the previous frame.

15. The method of claim 11, wherein for a tile classified in the intermediate speed regime, every pixel is sampled at least once.

16. The method of claim 11, wherein for a tile classified in the at least one high speed regime selecting a sampling pattern in which a number of samples is less than the number of pixels associated with the tile and performing interpolation to determine a color at pixel locations not sampled.

17. The method of claim 11, further comprising:

for a tile in the at least one high speed regime not including an edge, selecting a sampling pattern in which a number of samples is less than the number of pixels associated with the tile; and

for a tile in the at least one high speed regime including an edge, selecting a sampling pattern in which a number of samples is at least equal to one sample per pixel.

18. A graphics system including a graphics pipeline, the graphics system comprising:

an adaptive sampling generator; and

a pixel shader coupled to the adaptive sampling generator,

wherein the adaptive sampling generator is configured to determine a required sample rate for each tile from among tiles of an image based at least in part on a speed of pixels of objects in each tile, to select a sample pattern based on the required sample rate, and to determine a sample pattern and a sample rate for an individual tile based on a combination of the speed of the pixels of the objects in the individual tile and whether the individual tile includes an edge, and

wherein a reduced sampling rate is selected for each tile not including an edge and having a speed above a threshold speed.

19. The graphics system of claim 18, further comprising:

a velocity buffer to compare per-vertex coordinates of vertices of objects in a current frame to corresponding per-vertex coordinates in a previous frame, and to determine a per-pixel velocity of pixels in each tile;

an edge detector to detect a presence of edges in each tile; and

a sample generator to determine, for each tile to be shaded, a sample pattern and sampling rate based on a speed associated with pixels in each tile and the presence of edges.

20. The graphics system of claim 18, further comprising an advection unit and a reconstruction unit, wherein, for a tile having a velocity below a quasi-static speed limit, a sample pattern having a reduced sampling rate is selected and the reconstruction unit fills in missing pixel data by reusing pixel data from a previous frame via advection.

21. The graphics system of claim 20, wherein the reconstruction unit identifies a location of a pixel in a previous frame by mapping a change in pixel coordinates.

22. The graphics system of claim 20, wherein for the tile having a velocity below the quasi-static speed limit a preselected subset of the pixels of the tile are rendered and the remainder of the pixels are reused from the previous frame based on a pre-selected ratio of rendered to unrendered pixels.

23. A method of graphics processing utilizing a graphics processing unit (GPU), the method comprising:

adaptively rendering, by the GPU, at least one tile of a current frame based on speed determined by;

comparing the current frame to a previous frame;

determining, by the GPU, for the at least one tile of the current frame a relative speed of objects as measured in screen space co-ordinates or pixels in the current frame relative to a previous frame;

classifying, by the GPU based on the determined relative speed, each tile within the current frame into one of at least three speed categories including a quasi-static speed category, a medium speed category, and a high speed category;

making, by the GPU, a sampling decision for each tile, based at least in part on the speed category associated with each tile, including whether the tile is to be sampled at a full resolution sampling rate of at least one sample per pixel in the current frame or sampled at a lower rate in the current frame; and

rendering, by the GPU, tiles of the current frame,

wherein the frames correspond to images to be displayed on a display device.