Graphics scenegraph rendering for web applications using native code modules

• US 9,824,418 B1
• Filed: 03/15/2017
• Issued: 11/21/2017
• Est. Priority Date: 07/02/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Labour, Antoine, Papakipos, Matthew
• Primary Examiner(s): Thompson, James A
• Application Number:
  US15/460,168
• Time in Patent Office: 251 Days
• Field of Search: None
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: G06F 2221/2119 Authenticating web pages, e.g. with suspicious links
• CPC Class Codes: G06F 9/44526 Plug-ins; Add-ons
• CPC Class Codes: G06F 9/44589 Program code verification, e.g. Java bytecode verification, proof-carrying code high-level semantic checks G06F8/43; testing and debugging software G06F11/36
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 1/60 Memory management

Abstract

A first code module is loaded into a secure runtime environment that prevents the first code module from accessing a graphics-processor unit (GPU). Rendering commands are generated using the code module based on a scene representation, and transmitted from the first code module within the secure runtime environment to a second code module outside the secure runtime environment. The second code module is configured to communicate with the GPU to provide graphics hardware acceleration, and a rendered image is rendered using the second code module and the GPU based on execution of the rendering commands outside the secure runtime environment.

Claims

1. A computer-implemented method, comprising:

executing a first code module in a secure runtime environment, the secure runtime environment preventing the code module from accessing a graphics-processor unit (GPU);

generating, using the first code module, rendering commands based on a graphical scene representation;

transmitting the rendering commands from the first code module within the secure runtime environment to a second code module outside the secure runtime environment, the second code module configured to communicate with the GPU to provide graphics hardware acceleration; and

rendering, using the second code module and the GPU, a rendered image based on execution of the rendering commands outside the secure runtime environment.

2. The computer-implemented method of claim 1, wherein transmitting the rendering commands comprises:

writing, using the first code module, the rendering commands to a command buffer shared with the first code module and the second code module, and writing image data comprising one or more of vertex data, texture data, and effects data to a data buffer shared with the first code module and the second code module; and

wherein the method further comprises the second code module reading the rendering commands and the image data, and executing the rendering commands using the GPU.

3. The computer-implemented method of claim 2, further comprising:

generating an animation of the rendered image based on updating parameters in the data buffer for a same set of the rendering commands, without regenerating the same set of the rendering commands.

4. The computer-implemented method of claim 1, wherein the graphical scene representation is a data structure comprising interconnected nodes, each node for an object or a transformation for an object, the data structure.

5. The computer-implemented method of claim 4, wherein the graphical scene representation is a scenegraph, the method further comprising:

creating, using the first code module, the scenegraph based on a graphics model comprising a set of shapes, one or more light sources, or one or more rendering effects.

6. The computer-implemented method of claim 5, wherein the scenegraph comprises a shape node, and wherein generating the rendering commands comprises:

transforming the shape node into a set of resource management commands and a set of drawing commands, wherein the resource management commands include commands for creating resources to be used by the second code module and copying data stored in the shape node to the created resources.

7. The computer-implemented method of claim 5, wherein the runtime environment is associated with a web application, wherein the graphics model is for the web application, and wherein the graphics hardware acceleration is provided to the web application using the executing, generating, transmitting, and rendering steps.

8. The computer-implemented method of claim 7, wherein the web application is associated with a web browser, the method further comprising:

providing the rendered image for display in the web browser.

9. The computer-implemented method of claim 1, wherein the generating the rendering commands comprises transforming high-level shape nodes into low-level rendering commands.

10. The computer-implemented method of claim 1, further comprising:

validating first code module before the first code module is loaded into the secure runtime environment.

11. The computer-implemented method of claim 1, wherein the first code module is software platform independent and comprises binary code that runs directly on hardware.

12. A system, comprising:

a graphics processor unit (GPU);

one or more processors; and

a memory having instructions stored thereon that, when executed by the one or more processors, perform operations comprising;

executing a first code module in a secure runtime environment, the secure runtime environment preventing the first code module from accessing the GPU;

generating, using the first code module, rendering commands based on a graphical scene representation;

providing the rendering commands from the first code module within the secure runtime environment to a second code module outside the secure runtime environment, the second code module configured to communicate with the GPU to provide graphics hardware acceleration; and

rendering, using the second code module and the GPU, a rendered image based on execution of the rendering commands outside the secure runtime environment.

13. The system of claim of 12, wherein providing the rendering commands comprises:

writing, using the first code module, the rendering commands to a command buffer shared with the first code module and the second code module, and writing image data comprising one or more of vertex data, texture data, and effects data to a data buffer shared with the first code module and the second code module; and

wherein the method further comprises the second code module reading the rendering commands and the image data, and executing the rendering commands using the GPU.

14. The system of claim 13, wherein the operations further comprise:

generating an animation of the rendered image based on updating parameters in the data buffer for a same set of the rendering commands, without regenerating the same set of the rendering commands.

15. The system of claim 12, wherein the graphical scene representation is a scenegraph, the operations further comprising:

creating, using the first code module, the scenegraph based on a graphics model comprising a set of shapes, one or more light sources, or one or more rendering effects.

16. The system of claim 15, wherein the scenegraph comprises a shape node, and wherein generating the rendering commands comprises:

transforming the shape node into a set of resource management commands and a set of drawing commands, wherein the resource management commands include commands for creating resources to be used by the second code module and copying data stored in the shape node to the created resources.

17. The system of claim 15, wherein the runtime environment is associated with a web application, wherein the graphics model is for the web application, and wherein the graphics hardware acceleration is provided to the web application using the executing, generating, providing, and rendering operations.

18. The system of claim 17, wherein the web application is associated with a web browser, the operations further comprising:

providing the rendered image for display in the web browser.

19. A non-transitory computer-readable storage medium storing instructions thereon that, when executed by a computer system, cause the computer system to perform a method, comprising:

loading a first code module into a secure runtime environment, the secure runtime environment preventing the code module from accessing a graphics-processor unit (GPU);

generating, using the first code module, rendering commands based on a graphical scene representation;

providing the rendering commands from the first code module within the secure runtime environment to a second code module outside the secure runtime environment, the second code module configured to communicate with the GPU to provide graphics hardware acceleration; and

rendering, using the second code module and the GPU, a rendered image based on execution of the rendering commands outside the secure runtime environment.

20. The non-transitory computer-readable storage medium of claim 19, wherein the graphical scene representation is a scenegraph, and the runtime environment is associated with a web application loaded by a web browser, wherein the method further comprises:

creating, using the first code module, the scenegraph based on a graphics model for the web application, the graphics model comprising a set of shapes, one or more light sources, or one or more rendering effects, wherein the graphics hardware acceleration is provided to the web application using the loading, generating, providing, and rendering steps; and

providing the rendered image for display in the web browser.