Super-resolution based foveated rendering

• US 10,643,307 B2
• Filed: 11/10/2017
• Issued: 05/05/2020
• Est. Priority Date: 11/10/2017
• Status: Active Grant

• Current Assignee: Intel Corporation
• Original Assignee: Intel Corporation
• Inventors: Chen, Fan, Li, Zhengmin
• Primary Examiner(s): Du, Haixia
• Application Number:
  US15/809,344
• Publication Number:
  US 20190026864A1
• Time in Patent Office: 907 Days
• Field of Search: 345428
• CPC Class Codes: G06T 3/4046 using neural networks
• CPC Class Codes: G06T 3/4053 based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Abstract

An embodiment of a semiconductor package apparatus may include technology to identify a region of interest portion of a first image, and render the region of interest portion with super-resolution. Other embodiments are disclosed and claimed.

Claims

1. An electronic processing system comprising:

a graphics processor;

memory communicatively coupled to the graphics processor; and

logic communicatively coupled to the graphics processor to;

identify a region of interest portion of a first image,

provide the region of interest portion of the first image to a super-resolution neural network to generate a super-resolution enhanced image which corresponds to an increase of resolution relative to a resolution of the first image,

up-sample the first image to generate an up-sampled image,

combine the super-resolution enhanced image with the up-sampled image to provide a foveated image, and

train the super-resolution neural network to provide a blended transition between the region of interest portion of the first image and other portions of the first image.

2. The system of claim 1, wherein the logic is further to:

crop a training image based on the region of interest to generate a cropped image;

down-sample the cropped image to generate a down-sampled image;

up-sample the down-sampled image to generate an up-sampled second image; and

blend the up-sampled second image with the cropped image to generate a target image.

3. The system of claim 2, wherein the logic is further to:

train the super-resolution neural network with the down-sampled image as an input image for the super-resolution network and the target image as a target output image for the super-resolution network.

4. The system of claim 1, wherein the resolution of the first image is lower than a resolution of a target display device.

5. A semiconductor package apparatus comprising:

one or more substrates; and

logic coupled to the one or more substrates, wherein the logic is at least partly implemented in one or more of configurable logic and fixed-functionality hardware logic, the logic coupled to the one or more substrates to;

identify a region of interest portion of a first image,

provide the region of interest portion of the first image to a super-resolution neural network to generate a super-resolution enhanced image which corresponds to an increase of resolution relative to a resolution of the first image,

up-sample the first image to generate an up-sampled image,

combine the super-resolution enhanced image with the up-sampled image to provide a foveated image, and

train the super-resolution neural network to provide a blended transition between the region of interest portion of the first image and other portions of the first image.

6. The apparatus of claim 5, wherein the logic is further to:

crop a training image based on the region of interest to generate a cropped image;

down-sample the cropped image to generate a down-sampled image;

up-sample the down-sampled image to generate an up-sampled second image; and

blend the up-sampled second image with the cropped image to generate a target image.

7. The apparatus of claim 6, wherein the logic is further to:

train the super-resolution neural network with the down-sampled image as an input image for the super-resolution network and the target image as a target output image for the super-resolution network.

8. The apparatus of claim 5, wherein the resolution of the first image is lower than a resolution of a target display device.

9. A method of processing an image, comprising:

cropping a training image to generate a cropped image;

down-sampling the cropped image to generate a down-sampled image;

up-sampling the down-sampled image to generate an up-sampled image;

blending the up-sampled image with the cropped image to generate a target image; and

training a super-resolution network with the down-sampled image as an input image for the super-resolution network and the target image as a target output image for the super-resolution network.

10. The method of claim 9, further comprising:

identifying a region of interest portion of a first image;

providing the region of interest portion of the first image to the trained super-resolution network to generate a super-resolution enhanced image which corresponds to an increase of resolution relative to a resolution of the first image;

up-sampling the first image to generate an up-sampled second image; and

combining the super-resolution enhanced image with the up-sampled second image to provide a foveated image.

11. The method of claim 10, wherein the resolution of the first image is lower than a resolution of a target display device.

12. The method of claim 9, wherein the super-resolution network comprises a super-resolution neural network.

13. The method of claim 12, wherein the super-resolution neural network comprises a super-resolution convolutional neural network.

14. At least one non-transitory computer readable medium, comprising a set of instructions, which when executed by a computing device, cause the computing device to:

crop a training image to generate a cropped image;

down-sample the cropped image to generate a down-sampled image;

up-sample the down-sampled image to generate an up-sampled image;

blend the up-sampled image with the cropped image to generate a target image; and

train a super-resolution network with the down-sampled image as an input image for the super-resolution network and the target image as a target output image for the super-resolution network.

15. The at least one non-transitory computer readable medium of claim 14, comprising a further set of instructions, which when executed by the computing device, cause the computing device to:

identify a region of interest portion of a first image;

provide the region of interest portion of the first image to the trained super-resolution network to generate a super-resolution enhanced image which corresponds to an increase of resolution relative to a resolution of the first image;

up-sample the first image to generate an up-sampled second image; and

combine the super-resolution enhanced image with the up-sampled second image to provide a foveated image.

16. The at least one non-transitory computer readable medium of claim 15, wherein the resolution of the first image is lower than a resolution of a target display device.

17. The at least one non-transitory computer readable medium of claim 14, wherein the super-resolution network comprises a super-resolution neural network.

18. The at least one non-transitory computer readable medium of claim 17, wherein the super-resolution neural network comprises a super-resolution convolutional neural network.