System, Method, and Computer Program Product for Remote Graphics Processing

• US 20130215129A1
• Filed: 03/25/2013
• Published: 08/22/2013
• Est. Priority Date: 07/31/2000
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Silicon Graphics International Corporation (Hewlett-Packard Enterprise Company)
• Inventors: Keslin, Phillip C.
• Granted Patent: US 8,581,917 B2
• Time in Patent Office: Days
• US Class Current: 345/522
• CPC Class Codes: G06F 2209/509 Offload
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 9/5044 considering hardware capabilities
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 15/005 General purpose rendering architectures
• CPC Class Codes: G06T 2200/16 involving adaptation to the client's capabilities
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 5/36 characterised by the display of a graphic pattern, e.g. using an all-points-addressable [APA] memory
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality

Abstract

A system, method, and computer program product are provided for remote rendering of computer graphics. The system includes a graphics application program resident at a remote server. The graphics application is invoked by a user or process located at a client. The invoked graphics application proceeds to issue graphics instructions. The graphics instructions are received by a remote rendering control system. Given that the client and server differ with respect to graphics context and image processing capability, the remote rendering control system modifies the graphics instructions in order to accommodate these differences. The modified graphics instructions are sent to graphics rendering resources, which produce one or more rendered images. Data representing the rendered images is written to one or more frame buffers. The remote rendering control system then reads this image data from the frame buffers. The image data is transmitted to the client for display or processing. In an embodiment of the system, the image data is compressed before being transmitted to the client. In such an embodiment, the steps of rendering, compression, and transmission can be performed asynchronously in a pipelined manner.

Claims

1. A system comprising:

a server system including a processor and memory storing one or more programs executable by the processor, the one or more programs including instructions for;

receiving, at a graphics application on the server system, a command from a client, the command related to a graphics rendering operation of the graphics application;

responsive to the command, generating an application programming interface call for producing rendered image data;

receiving, at a remote rendering control system on the server system, the application programming interface call from the graphics application;

modifying a parameter of the application programming interface call based upon a difference between the client and the server in at least one of graphics processing capabilities and graphics contexts;

performing the rendering operation using the modified parameters; and

transmitting the rendered image data to the client for display.

2. The system of claim 1, wherein the application programming interface is an OPENGL application programming interface.

3. The system of claim 1, wherein the command corresponds to a command requesting the rendered image data for display on the client.

4. The system of claim 1, the one or more programs further including instructions for:

receiving, at the server system, a list of graphics processing capabilities of the client;

merging, with the remote rendering control system, the list of graphics processing capabilities of the client and a list of graphics capabilities of the server system to create a merged list of graphics processing capabilities; and

generating, using the merged list of graphics processing capabilities, a routine for converting graphics processing capabilities of the server to graphics processing capabilities of the client.

5. The system of claim 4, wherein the list of graphics processing capabilities of the client are selected from a group consisting of:

color mapping range;

ability to represent opacity; and

ability to perform stenciling.

6. The system of claim 1, the one or more programs further including instructions for compressing the image data by a compression process of the remote rendering control system, wherein the rendering process of a first block of data occurs during a first time period, the compression process of a second block of data occurs during a second time period, and the transmission process of a third block of data occurs during a third time period so that the rendering process, the compression process, and the transmission process are pipelined.

7. The system of claim 6, the one or more programs further including instructions for:

discarding, by the remote rendering control system, the first block of data when the rendering process of the first block of data is completed before the compression process of the second block of data; and

discarding, by the remote rendering control system, the second block of data when the compression process of the second block of data is completed before the transmission process of the third block of data.

8. The system of claim 1, the one or more programs further including instructions for modifying the parameter of the application programming interface call by converting a graphics processing capability of the server system appearing in the application programming interface call to a graphics processing capability of the client.

9. The system of claim 1, the one or more programs further including instructions for modifying the parameter of the application programming interface call by converting a graphics context of the server system appearing in the application programming interface call to a graphics context of the client.

10. A method for rendering image data, performed at an electronic device having a processor and memory storing one or more programs executable by the processor, the method comprising:

receiving, at a graphics application on a server system, a command from a client, the command related to a graphics rendering operation performable by the graphics application;

responsive to the command, generating an application programming interface call for producing rendered image data;

receiving, at a remote rendering control system on the server system, the application programming interface call from the graphics application;

modifying a parameter of the application programming interface call based upon a difference between the client and the server in at least one of graphics processing capabilities and graphics contexts;

performing the rendering operation using the modified parameters; and

transmitting the rendered image data to the client for display.

11. The method of claim 10, wherein the application programming interface is an OPENGL application programming interface.

12. The method of claim 10, wherein the command corresponds to a command requesting the rendered image data for display on the client.

13. The method of claim 10, further comprising:

receiving, at the server system, a list of graphics processing capabilities of the client;

merging, with the remote rendering control system, the list of graphics processing capabilities of the client and a list of graphics capabilities of the server system to create a merged list of graphics processing capabilities; and

generating, using the merged list of graphics processing capabilities, a routine for converting graphics processing capabilities of the server to graphics processing capabilities of the client.

14. The method of claim 13, wherein the list of graphics processing capabilities of the client are selected from a group consisting of:

color mapping range;

ability to represent opacity; and

ability to perform stenciling.

15. The method of claim 10, further comprising compressing the image data by a compression process of the remote rendering control system, wherein the rendering process of a first block of data is occurs during a first time period, the compression process of a second block of data occurs during a second time period, and the transmission process of a third block of data occurs during a third time period so that the rendering process, the compression process, and the transmission process are pipelined.

16. The method of claim 15, further comprising:

discarding, by the remote rendering control system, the first block of data when the rendering process of the first block of data is completed before the compression process of the second block of data; and

discarding, by the remote rendering control system, the second block of data when the compression process of the second block of data is completed before the transmission process of the third block of data.

17. The method of claim 10, wherein modifying the parameter of the application programming interface call includes converting a graphics processing capability of the server system appearing in the application programming interface call to a graphics processing capability of the client.

18. The method of claim 10, wherein modifying the parameter of the application programming interface call includes converting a graphics context of the server system appearing in the application programming interface call to a graphics context of the client.

19. A non-transitory computer readable storage medium having stored thereon computer-executable instructions that, responsive to execution by a server system, cause the server system to perform operations comprising:

receiving, at a graphics application, a command from a client, the command related to a graphics rendering operation performable by the graphics application;

responsive to the command, generating an application programming interface call for producing rendered image data;

receiving, at a remote rendering control system, the application programming interface call from the graphics application;

modifying a parameter of the application programming interface call based upon a difference between the client and the server in at least one of graphics processing capabilities and graphics contexts;

performing the rendering operation using the modified parameters; and

transmitting the rendered image data to the client for display.

20. The non-transitory computer readable storage medium of claim 19, wherein the application programming interface is an OPENGL application programming interface.

21. The non-transitory computer readable storage medium of claim 19, wherein the command corresponds to a command requesting the rendered image data for display on the client.

22. The non-transitory computer readable storage medium of claim 19, the operations further comprising:

receiving, at the server system, a list of graphics processing capabilities of the client;

merging, with the remote rendering control system, the list of graphics processing capabilities of the client and a list of graphics capabilities of the server system to create a merged list of graphics processing capabilities; and

generating, using the merged list of graphics processing capabilities, a routine for converting graphics processing capabilities of the server to graphics processing capabilities of the client.

23. The non-transitory computer readable storage medium of claim 21, wherein the list of graphics processing capabilities of the client are selected from a group consisting of:

color mapping range;

ability to represent opacity; and

ability to perform stenciling.

24. The non-transitory computer readable storage medium of claim 19, the operations further comprising compressing the image data by a compression process of the remote rendering control system, wherein the rendering process of a first block of data occurs during a first time period, the compression process of a second block of data occurs during a second time period, and the transmission process of a third block of data occurs during a third time period so that the rendering process, the compression process, and the transmission process are pipelined.

25. The non-transitory computer readable storage medium of claim 24, the operations further comprising:

discarding, by the remote rendering control system, the first block of data when the rendering process of the first block of data is completed before the compression process of the second block of data; and

discarding, by the remote rendering control system, the second block of data when the compression process of the second block of data is completed before the transmission process of the third block of data.

26. The non-transitory computer readable storage medium of claim 19, wherein modifying the parameter of the application programming interface call includes converting a graphics processing capability of the server system appearing in the application programming interface call to a graphics processing capability of the client.

27. The non-transitory computer readable storage medium of claim 19, wherein modifying the parameter of the application programming interface call includes converting a graphics context of the server system appearing in the application programming interface call to a graphics context of the client.