Method and mechanism for delivering applications over a wan

• US 9,001,135 B2
• Filed: 09/16/2011
• Issued: 04/07/2015
• Est. Priority Date: 09/18/2010
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Dharmapurikar, Makarand
• Primary Examiner(s): Zarka, David
• Assistant Examiner(s): PRINGLE-PARKER, JASON A
• Application Number:
  US13/234,948
• Publication Number:
  US 20120069036A1
• Time in Patent Office: 1,299 Days
• Field of Search: None
• US Class Current: 345/522
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 15/30 Clipping
• CPC Class Codes: G06T 15/40 Hidden part removal
• CPC Class Codes: G06T 2200/16 involving adaptation to the client's capabilities
• CPC Class Codes: G06T 2210/36 Level of detail
• CPC Class Codes: G09G 2340/02 Handling of images in compressed format, e.g. JPEG, MPEG
• CPC Class Codes: G09G 2350/00 Solving problems of bandwidth in display systems
• CPC Class Codes: G09G 2360/08 Power processing, i.e. workload management for processors involved in display operations, such as CPUs or GPUs
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 3/003 to produce spatial visual effects

Abstract

An improved approach for a remote graphics rendering system that can utilize both server-side processing and client-side processing for the same display frame. Some techniques for optimizing a set of graphics command data to be sent from the server to the client include: eliminating some or all data, that is not needed by a client GPU to render one or more images, from the set of graphics command data to be transmitted to the client; applying precision changes to the set of graphics command data to be transmitted to the client; and performing one or more data type compression algorithms on the set of graphics command data.

Claims

1. A computer implemented method for implementing remote

graphics rendering, comprising:

a) generating, at a server CPU, a set of graphics commands to be transmitted from a server to a client, wherein the set of graphics commands are commands to be executed by a client GPU to generate pixels for display;

b) optimizing the set of graphics commands at the server CPU by;

1) eliminating some or all primitive data, that need not be executed by the client GPU to generate pixels for rendering one or more images, from the set of graphics commands to be transmitted to the client;

2) applying precision changes to the set of graphics commands to be transmitted to the client; and

3) performing one or more data type compression algorithms on the set of graphics commands, including;

i) separating the set of graphics commands into a plurality of data streams based on individual data types, the individual data types comprising vertex position data, API calls, and texture coordinate data; and

ii) performing a different customized compression algorithm on each of the plurality of data streams to generate a corresponding plurality of compressed data streams representing the set of graphics commands; and

c) transmitting the plurality of compressed data streams representing the optimized set of graphics commands to the client, wherein the plurality of compressed data streams representing the optimized set of graphics commands is executed by the client GPU to render the one or more images.

2. The computer implemented method of claim 1, wherein eliminating some or all primitive data that does not need to be executed by the client GPU to generate pixels for rendering the one or more images comprises:

identifying data to be eliminated from the set of graphics commands to be sent to the client device;

marking the data to be eliminated in the set of graphics commands to be sent to the client device;

removing the data to be eliminated from the set of graphics commands to be sent to the client device.

3. The computer implemented method of claim 2, wherein identifying data to be eliminated comprises identifying a primitive that is facing away from a camera.

4. The computer implemented method of claim 2, wherein identifying data to be eliminated comprises identifying a primitive that is occluded from a camera.

5. The computer implemented method of claim 2, wherein identifying data to be eliminated comprises identifying a primitive that falls outside of a view frustum of a camera.

6. The computer implemented method of claim 2, wherein identifying data to be eliminated comprises identifying vertex information for a primitive that is to be sent to the client device but never rendered.

7. The computer implemented method of claim 2, wherein identifying data to be eliminated comprises identifying an entire render target that is not changing.

8. The computer implemented method of claim 1, wherein applying precision changes to the set of graphics commands comprises:

evaluating an object associated with the set of graphics commands to determine a ratio between a screen area of the object and an amount of data in the set of graphics commands used to create the object;

comparing the ratio to a threshold value; and

reducing a precision requirement for the object if the ratio is above the threshold value.

9. The computer implemented method of claim 8, wherein reducing a precision requirement for the object comprises reducing vertex attribute precision.

10. The computer implemented method of claim 8, wherein reducing a precision requirement for the object comprises reducing texture image precision.

11. The computer implemented method of claim 8, wherein reducing a precision requirement for the object comprises reducing object mesh precision.

12. The method of claim 1, wherein performing a different customized compression algorithm on each of the plurality of data streams comprises performing lossless compression on API calls.

13. The method of claim 1, wherein performing a different customized compression algorithm on each of the plurality of data streams comprises performing lossy compression on texture coordinate data.

14. The method of claim 1, wherein performing a different customized compression algorithm on each of the plurality of data streams comprises performing transform matrix based compression on vertex position data.

15. The computer implemented method of claim 1, wherein the method further comprises:

initializing a reference device; and

generating a reference image based on the set of graphics commands at the server CPU using the reference device to derive information to be used by the server CPU for;

eliminating primitive data, that does not need to be executed by the client GPU to render the one or more images, from the set of graphics commands;

or

applying precision changes to the set of graphics commands.

16. The computer implemented method of claim 1, wherein the set of graphics commands corresponds to a single display frame.

17. A computer implemented method for implementing remote graphics rendering, comprising:

a) receiving, at a client, a set of optimized graphics commands from a server, the set of optimized graphics commands including a plurality of compressed data streams corresponding to individual data types, the individual data types comprising vertex position data, API calls, and texture coordinate data, wherein the set of optimized graphics commands is executed by a client GPU at the client to generate pixels for rendering one or more images;

b) parsing the received set of optimized graphics commands to separate the set of optimized graphics commands into the individual data types; and

c) applying a decompression algorithms based at least in part on data type to each of the plurality of data streams representing the set of optimized graphics commands to form a set of uncompressed optimized graphics commands; and

d) assembling API streams for the uncompressed optimized graphics commands.

18. The computer implemented method of claim 17, further comprising:

sending the uncompressed optimized graphics commands to the client GPU;

processing the uncompressed optimized graphics commands with the client GPU; and

displaying an image associated with the uncompressed optimized graphics commands.

19. The computer implemented method of claim 17, wherein the set of graphics commands corresponds to a single display frame.

20. A remote graphics rendering system, comprising:

a server comprising a server CPU, wherein the server is configured to perform a method for implementing remote graphics rendering, comprising;

a) generating, at the server CPU, a set of graphics commands to be transmitted from the server to a client, wherein the set of graphics commands are commands to be executed by a GPU to generate pixels for display;

b) optimizing the set of graphics commands at the server CPU by;

1) eliminating some or all primitive data, that does not need to be executed by a client GPU to generate pixels for rendering one or more images, from the set of graphics commands to be transmitted to the client;

2) applying precision changes to the set of graphics commands to be transmitted to the client; and

3) performing one or more data type compression algorithms on the set of graphics commands, including performing a different customized compression algorithm on each of a plurality of individual data streams of the set of graphics commands, the plurality of individual data streams comprising vertex position data, API calls, and texture coordinate data, to generate a plurality of compressed data streams representing the set of graphics commands; and

c) transmitting the plurality of compressed data streams representing the optimized set of graphics commands to the client, wherein the optimized set of graphics commands is executed by the client GPU to generate pixels for rendering the one or more images.

21. The remote graphics rendering system of claim 20, wherein eliminating some or all primitive data that does not need to be executed by the client GPU to generate pixels for rendering the one or more images comprises:

identifying data to be eliminated from the set of graphics commands to be sent to the client device;

marking the data to be eliminated in the set of graphics commands to be sent to the client device;

removing the data to be eliminated from the set of graphics commands to be sent to the client device.

22. The remote graphics rendering system of claim 20, wherein applying precision changes to the set of graphics commands comprises:

evaluating an object associated with the set of graphics commands to determine a ratio between a screen area of the object and an amount of data in the set of graphics commands used to create the object;

comparing the ratio to a threshold value; and

reducing a precision requirement for the object if the ratio is above the threshold value.

23. The remote graphics rendering system of claim 20, wherein the server further comprises a server software layer connected to the server CPU, the server software layer being configured to facilitate optimizing the set of graphics commands at the server CPU.

24. The remote graphics rendering system of claim 23, wherein the server software layer facilitates optimizing the set of graphics commands data by:

initializing a reference device; and

generating a reference image based on the set of graphics commands using the reference device to derive information to be used by the server CPU for;

eliminating some or all primitive data, that does not need to be executed by the client GPU to render the one or more images, from the set of graphics commands;

or

applying precision changes to the set of graphics commands.

25. A remote graphics rendering system, comprising:

a client, comprising;

a client CPU; and

a client GPU, wherein the client is configured to perform a method for implementing remote graphics rendering, comprising;

a) receiving a set of optimized graphics commands from a server, wherein the set of optimized graphics commands includes a plurality of data streams based on data types, the plurality of data streams comprising vertex position data, API calls, and texture coordinate data, the plurality of data streams to be executed by the client GPU at the client to generate pixels for rendering one or more images;

b) parsing the received set of optimized graphics commands using the client CPU to separate the set of optimized graphics commands into the plurality of data streams based on data types; and

c) applying one or more decompression algorithms based at least in part on data type to the plurality of data streams representing the set of optimized graphics commands using the client CPU to form a set of uncompressed optimized graphics commands; and

d) assembling API streams for the uncompressed optimized graphics commands data using the client CPU.

26. The remote graphics rendering system of claim 25, wherein the method for implementing remote graphics rendering, further comprises:

sending the uncompressed optimized graphics commands to the client GPU;

processing the uncompressed optimized graphics commands with the client GPU; and

displaying an image associated with the uncompressed optimized graphics commands data.

27. A computer program product that includes a non-transitory computer readable medium, the non-transitory computer readable medium comprising a plurality of computer instructions which, when executed by at least one processor, cause the at least one processor to execute a method for implementing remote graphics rendering, the method comprising:

a) generating, at a server CPU, a set of graphics commands to be transmitted from a server to a client, wherein the set of graphics commands are commands to be executed by a GPU to generate pixels for display;

b) optimizing the set of graphics commands at the server CPU by;

1) eliminating some or all primitive data, that need not be executed by a client GPU to generate pixels for rendering one or more images, from the set of graphics commands to be transmitted to the client;

2) applying precision changes to the set of graphics commands to be transmitted to the client; and

3) performing one or more data type compression algorithms on the set of graphics commands, including performing a different customized compression algorithm on each of a plurality of data streams of the set of graphics commands, the plurality of data streams comprising vertex position data, API calls, and texture coordinate data, to generate a plurality of compressed data streams representing the set of graphics commands; and

c) transmitting the plurality of compressed data streams representing the optimized set of graphics commands to the client, wherein the optimized set of graphics commands is executed by the client GPU to generate pixels for rendering the one or more images.

28. A computer program product that includes a non-transitory computer readable medium, the non-transitory computer readable medium comprising a plurality of computer instructions which, when executed by at least one processor, cause the at least one processor to execute a method for implementing remote graphics rendering, the method comprising:

a) receiving, at a client, a set of optimized graphics commands from a server, wherein the set of optimized graphics commands includes a plurality of data streams each corresponding to a different data type, the plurality of data streams comprising vertex position data, API calls, and texture coordinate data, the optimized set of graphics commands to be executed by a client GPU at the client to generate pixels for rendering one or more images;

b) parsing the received set of optimized graphics commands using the client CPU to separate the set of optimized graphics commands into the plurality of data streams based on data types; and

c) applying one or more decompression algorithms based at least in part on data type to the plurality of data streams representing the set of optimized graphics commands using the client CPU to form a set of uncompressed optimized graphics commands; and

d) assembling API streams for the uncompressed optimized graphics commands using the client CPU.