Machine learning applied to textures compression or upscaling

• US 10,504,248 B2
• Filed: 05/31/2018
• Issued: 12/10/2019
• Est. Priority Date: 03/30/2018
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Inventors: Fuller, Martin Jon Irwin, Kennett, Daniel Gilbert
• Primary Examiner(s): Chen, Frank S
• Application Number:
  US15/994,464
• Publication Number:
  US 20190304140A1
• Time in Patent Office: 558 Days
• Field of Search: None
• CPC Class Codes: G06F 12/023 Free address space management
• CPC Class Codes: G06F 2212/401 Compressed data
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 11/001 Texturing; Colouring; Generation of texture or colour inpainting G06T5/77
• CPC Class Codes: G06T 2200/28 involving image processing hardware
• CPC Class Codes: G06T 9/00 Image coding bandwidth or redundancy reduction for static pictures H04N1/41; coding or decoding of static colour picture signals H04N1/64; methods or arrangements for coding, decoding, compressing or decompressing digital video signals H04N19/00
• CPC Class Codes: G06T 9/002 using neural networks

Abstract

Methods and devices for generating hardware compatible compressed textures may include accessing, at runtime of an application program, graphics hardware incompatible compressed textures in a format incompatible with a graphics processing unit (GPU). The methods and devices may include converting the graphics hardware incompatible compressed textures directly into hardware compatible compressed textures usable by the GPU using a trained machine learning model.

Claims

1. A computer device, comprising:

a graphics processing unit (GPU);

a memory to store data and instructions including an application and graphics hardware incompatible compressed textures in a format incompatible with the GPU;

at least one processor in communication with the memory;

an operating system in communication with the memory, the at least one processor, the GPU, and the application, wherein the application is operable to;

receive, at runtime or installation of the application, the graphics hardware incompatible compressed textures;

determine that the graphics hardware incompatible compressed textures are incompatible with the GPU; and

convert the graphics hardware incompatible compressed textures directly into hardware compatible compressed textures usable by the GPU using a trained machine learning model, wherein the trained machine learning model uses metadata that provides configurations for a block compression of the graphics hardware incompatible compressed textures to use during the conversion so that the hardware compatible compressed textures closely resemble original raw images of the application.

2. The computer device of claim 1, wherein the hardware compatible compressed textures are in a format of a block compressed number (BCN) texture, and wherein the GPU is further configured to render textures from the hardware compatible compressed textures to present on a display.

3. The computer device of claim 1, wherein the trained machine learning model is trained offline by performing at least one machine learning search on the graphics hardware incompatible compressed textures to determine the configurations for a block compression of the graphics hardware incompatible compressed textures.

4. The computer device of claim 3, wherein different machine learning searches are trained for one or more of different texture types and different content types.

5. The computer device of claim 1, wherein the application is further operable to:

access a modified MIP chain that includes a plurality of images with varying resolutions for use with textures of the application, wherein the modified MIP chain removed a highest resolution image from the original MIP chain and the plurality of images correspond to an original MIP chain for use with the textures of the application.

6. The computer device of claim 5, wherein the application is further operable to:

reconstruct the highest resolution image by applying the trained machine learning model to a first image of the modified MIP chain;

generate a hardware compatible compressed reconstructed MIP chain with the reconstructed highest resolution image and the plurality of images; and

transmit the hardware compatible compressed reconstructed MIP chain to the GPU.

7. The computer device of claim 6, wherein the hardware compatible compressed reconstructed MIP chain includes images in a format of a block compressed number (BCN) texture.

8. The computer device of claim 5, wherein the modified MIP chain also removes one or more lower resolution image from the original MIP chain.

9. The computer device of claim 1, wherein at least one of the GPU and a central processing unit (CPU) decompresses the hardware incompatible compressed textures directly into the hardware compatible compressed textures.

10. A method for generating hardware compatible compressed textures, comprising:

receiving, by an application executing on the computer device, at runtime or installation of an application program, graphics hardware incompatible compressed textures in a format incompatible with a graphics processing unit (GPU);

determining that the graphics hardware incompatible compressed textures are incompatible with the GPU; and

converting the graphics hardware incompatible compressed textures directly into hardware compatible compressed textures usable by the GPU using a trained machine learning model, wherein the trained machine learning model uses metadata that provides configurations for a block compression of the graphics hardware incompatible compressed textures to use during the conversion so that the hardware compatible compressed textures closely resemble original raw images of the application.

11. The method of claim 10, wherein the hardware compatible compressed textures are in a format of a block compressed number (BCN) texture, and wherein the GPU is further configured to render textures from the hardware compatible compressed textures to present on a display.

12. The method of claim 10, wherein the trained machine learning model is trained offline by performing at least one machine learning search on the graphics hardware incompatible compressed textures to determine the configurations for a block compression of the graphics hardware incompatible compressed textures.

13. The method of claim 12, wherein different machine learning searches are trained for one or more of different texture types and different content types.

14. The method of claim 10, further comprising:

accessing a modified MIP chain that includes a plurality of images with varying resolutions for use with textures of the application, wherein the modified MIP chain removed a highest resolution image from the original MIP chain and the plurality of images correspond to an original MIP chain for use with the textures of the application.

15. The method of claim 14, further comprising:

reconstructing the highest resolution image by applying the trained machine learning model to a first image of the modified MIP chain;

generating a hardware compatible compressed reconstructed MIP chain with the reconstructed highest resolution image and the plurality of images; and

transmitting the hardware compatible compressed reconstructed MIP chain to the GPU.

16. The method of claim 15, wherein the hardware compatible compressed reconstructed MIP chain includes images in a format of a block compressed number (BCN) texture.

17. The method of claim 14, wherein the modified MIP chain also removes one or more lower resolution image from the original MIP chain.

18. The method of claim 10, wherein at least one of the GPU and a central processing unit (CPU) decompresses the hardware incompatible compressed textures directly into the hardware compatible compressed textures.

19. A non-transitory computer-readable medium storing instructions executable by a computer device, comprising:

at least one instruction for causing the computer device to receive, at runtime of an application program, graphics hardware incompatible compressed textures in a format incompatible with a graphics processing unit (GPU);

at least one instruction for causing the computer device to determine that the graphics hardware incompatible compressed textures are incompatible with the GPU; and

at least one instruction for causing the computer device to convert the hardware incompatible compressed textures directly into hardware compatible compressed textures usable by the GPU using a trained machine learning model, wherein the trained machine learning model uses metadata that provides configurations for a block compression of the graphics hardware incompatible compressed textures to use during the conversion so that the hardware compatible compressed textures closely resemble original raw images of the application.