DUAL-PATH FOVEATED GRAPHICS PIPELINE

• US 20180136720A1
• Filed: 11/14/2016
• Published: 05/17/2018
• Est. Priority Date: 11/14/2016
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Spitzer, Mark, Vieri, Carlin, Turner, Eric, Bastani, Behnam
• Granted Patent: US 10,564,715 B2
• Time in Patent Office: Days
• CPC Class Codes: G02B 27/0093 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/147 using display panels
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 3/40 Scaling of whole images or parts thereof, e.g. expanding or contracting
• CPC Class Codes: G09G 2320/0261 in the context of movement of objects on the screen or movement of the observer relative to the screen
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2340/0407 Resolution change, inclusive of the use of different resolutions for different screen areas
• CPC Class Codes: G09G 2340/0457 Improvement of perceived resolution by subpixel rendering
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2352/00 Parallel handling of streams of display data
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092
• CPC Class Codes: G09G 5/006 Details of the interface to the display terminal specific for a display terminal using a CRT G09G1/167; using a flat panel G09G3/2096; circuits for interfacing with colour displays G09G5/04
• CPC Class Codes: G09G 5/391 Resolution modifying circuits, e.g. variable screen formats

Abstract

A foveated display system includes a rendering device including at least one graphics processing unit (GPU) to render a foveal region and a peripheral region of a first image, wherein the foveal region has a higher resolution than the peripheral region. The system further includes a display device coupled to the rendering device via at least one physical layer. The display device includes a pixel array and a display controller coupled to the pixel array. The display controller includes a scaling component to upscale the first peripheral region to generate a scaled first peripheral region and a blending component to blend the foveal region with the scaled first peripheral region to generate a second image.

Claims

1. A foveated display system comprising:

a rendering device comprising at least one graphics processing unit (GPU) to render a foveal region and a first peripheral region of a first image, the foveal region having a higher resolution than the first peripheral region; and

a display device coupled to the rendering device via at least one physical layer, the display device comprising;

a pixel array; and

a display controller coupled to the pixel array, the display controller comprising;

a scaling component to upscale the first peripheral region to generate a scaled first peripheral region; and

a blending component to blend the foveal region with the scaled first peripheral region to generate a second image.

2. The foveated display system of claim 1, wherein the rendering device is to transmit the foveal region and the first peripheral region to the display device via the same physical layer.

3. The foveated display system of claim 2, wherein:

foveal region is composed of a first array of pixels in a first number of rows and a second number of columns;

the first peripheral region is composed of a second array of pixels in a third number of rows and a fourth number of columns; and

the rendering device further includes;

a pixel stream combination component to combine the foveal region and first peripheral region for transmission as a third array of pixels having a fifth number of rows and a sixth number of columns, whereby the fifth number of rows is equal to the larger of the first number and the third number and the sixth number is based on the second number and the fourth number.

4. The foveated display system of claim 3, wherein the pixel stream combination component is to generate a row of the third array of pixels by determining a seventh number based on the third number and appending a number of pixels equal to the seventh number from one or more rows of the one of the first array or the second array having the smaller number of rows to a row of pixels of the one of the first array or the second array having the larger number of rows.

5. The foveated display system of claim 3, wherein the display controller further comprises:

a pixel stream separation component to receive the third array of pixels and to separate the third array of pixels into the first array of pixels and the second array of pixels.

6. The foveated display system of claim 1, wherein the rendering device is to transmit the foveal region to the display device via a first physical layer and to transmit the first peripheral region to the display device via a second physical layer distinct from the first physical layer.

7. The foveated display system of claim 1, wherein:

the rendering device further is configured to compress the foveal region to generate a compressed foveal region and to compress the first peripheral region to generate a compressed peripheral region; and

the display controller further is to decompress the compressed foveal region and to decompress the compressed first peripheral region.

8. The foveated display system of claim 1, wherein the display controller further comprises:

a sub-pixel rendering component to sub-pixel render the foveal region to generate the foveal region and to sub-pixel render the first peripheral region to generate the first peripheral region.

9. The foveated display system of claim 1, wherein:

the GPU further is to render a second peripheral region of the first image, the second peripheral region having a lower resolution than the first peripheral region;

the scaling component further is to upscale the second peripheral region to generate a scaled second peripheral region; and

the blending component further is to blend the foveal region, the scaled first peripheral region, and the scaled second peripheral region to generate the second image.

10. The foveated display system of claim 1, wherein:

the display device is a head mounted display device; and

the pixel array is implemented in one of a left-eye display panel or a right-eye display panel.

11. The foveated display system of claim 10, wherein:

the HMD device comprises a gaze tracking system to track a gaze direction of at least one eye of a user; and

the GPU is to determine a position for the foveal region for the first image based on the gaze direction.

12. In a foveated display system comprising a rendering device connected to a display device via at least one physical layer, a method comprising:

rendering a foveal region and a first peripheral region of a first image using at least one graphics processing unit (GPU) of the display device, the foveal region having a higher resolution of the first peripheral region;

transmitting the foveal region and the first peripheral region to the display device via the at least one physical layer;

upscaling, at a display controller of the display device, the first peripheral region to generate a scaled first peripheral region; and

blending, at the display controller, the foveal region and the scaled first peripheral region to generate a second image.

13. The method of claim 12, wherein:

transmitting the foveal region and the first peripheral region comprises transmitting the foveal region and the first peripheral region to the display device via the same physical layer.

14. The method of claim 13, wherein:

foveal region is composed of a first array of pixels in a first number of rows and a second number of columns;

the first peripheral region is composed of a second array of pixels in a third number of rows and a fourth number of columns; and

the method further comprises;

combining the foveal region and first peripheral region for transmission as a third array of pixels having a fifth number of rows and a sixth number of columns, whereby the fifth number of rows is equal to the larger of the first number and the third number and the sixth number is based on the second number and the fourth number.

15. The method of claim 14, wherein combining the foveal region and first peripheral region comprises:

generating a row of the third array of pixels by determining a seventh number based on the third number and appending a number of pixels equal to the seventh number from one or more rows of the one of the first array or the second array having the smaller number of rows to a row of pixels of the one of the first array or the second array having the larger number of rows.

16. The method of claim 14, further comprising:

receiving, at the display controller, the third array of pixels via the same physical layer; and

separating, at the display controller, the third array of pixels into the first array of pixels and the second array of pixels.

17. The method of claim 16, wherein:

transmitting the foveal region and the first peripheral region comprises transmitting the foveal region from the rendering device to the display device via a first physical layer and transmitting the first peripheral region from the rendering device to the display device via a second physical layer distinct from the first physical layer.

18. The method of claim 12, further comprising:

compressing the foveal region at the rendering device to generate a compressed foveal region;

compressing the first peripheral region at the rendering device to generate a compressed peripheral region;

decompressing the compressed foveal region at the display controller to generate the foveal region;

decompressing the compressed first peripheral region at the display controller to generate the first peripheral region; and

wherein transmitting the foveal region and the first peripheral region comprises transmitting the compressed foveal region and the compressed first peripheral region.

19. The method of claim 18, further comprising:

transforming, using at least one GPU of the rendering device, a pixel format of the foveal region and the first peripheral region from a red-green-blue (RGB) pixel format to a non-RGB pixel format.

20. The method of claim 12, further comprising:

transforming, using at least one GPU of the rendering device, a format of the foveal region and the first peripheral region from a red-green-blue (RGB) pixel format to a non-RGB pixel format.

21. The method of claim 12, further comprising:

rendering, using at least one GPU of the rendering device, a second peripheral region of the first image, the second peripheral region having a lower resolution than the first peripheral region;

transmitting the second peripheral region from the rendering device to the display device via the at least one physical layer;

upscaling, at the display controller, the second peripheral region to generate a scaled second peripheral region; and

blending the foveal region and the first peripheral region further comprises blending the foveal region, the first peripheral region, and the second peripheral region to generate the second image.

22. The method of claim 12, wherein the display device is a head mounted display (HMD) device.

23. The method of claim 22, further comprising:

tracking a gaze direction of at least one eye of a user; and

determining, at the rendering device, a position for the foveal region in the first image based on the gaze direction.

24. A system comprising:

a head mounted display (HMD) device comprising;

a pixel array; and

a display controller coupled to the pixel array

a rendering device comprising at least one graphics processing unit (GPU);

first and second physical layers coupling the rendering device to the HMD device; and

wherein the system implements;

a first data path for generating a first foveal region of a first image and preparing the first foveal region for display to an eye of a user using the pixel array, the first data path comprising the at least one GPU, the first physical layer, and the display controller; and

a second data path for generating a first peripheral region of the first image and preparing the first peripheral region for display to the eye of the user using the pixel array, the second data path comprising the at least one GPU, the second physical layer, and the display controller.

25. The system of claim 24, wherein:

the first data path comprises a compression component at the rendering device for compressing the first foveal region to generate a compressed first foveal region and a decompression component at the display controller for decompressing the compressed first foveal region; and

the second data path comprises a compression component at the rendering device for compressing the first peripheral region to generate a compressed first peripheral region and a decompression component at the display controller for decompressing the compressed first peripheral region.

26. The system of claim 24, wherein:

the second data path comprises a scaling component for upscaling the first peripheral region to generate a scaled first peripheral region; and

the display controller further comprises a blending component to blend the first foveal region with the scaled first peripheral region to generate a representation of the first image.

27. The system of claim 24, wherein:

the first data path comprises a sub-pixel transform component of the rendering device to transform a format of the first foveal region from a red-green-blue (RGB) format to a non-RGB pixel format; and

the second data path comprises a sub-pixel transform component of the rendering device to transform a format of the first peripheral region from the RGB pixel format to the non-RGB pixel format.