Graphics scenegraph rendering for web applications using native code modules

• US 10,026,147 B1
• Filed: 11/14/2017
• Issued: 07/17/2018
• Est. Priority Date: 07/02/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Labour, Antoine, Papakipos, Matthew
• Primary Examiner(s): Thompson, James A
• Application Number:
  US15/813,025
• Time in Patent Office: 245 Days
• Field of Search: None
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: G06F 2221/2119 Authenticating web pages, e.g. with suspicious links
• CPC Class Codes: G06F 9/44526 Plug-ins; Add-ons
• CPC Class Codes: G06F 9/44589 Program code verification, e.g. Java bytecode verification, proof-carrying code high-level semantic checks G06F8/43; testing and debugging software G06F11/36
• CPC Class Codes: G06T 1/20 Processor architectures; Processor configuration, e.g. pipelining
• CPC Class Codes: G06T 1/60 Memory management

Abstract

A native code module is executed in a secure runtime environment that prevents the first code module from accessing a graphics rendering interface outside of the secure runtime environment. Rendering commands are generated using the native code module based on a scene representation, and transmitted from the native code module within the secure runtime environment to a trusted code module outside the secure runtime environment. The trusted code module is configured to communicate with the graphics rendering interface to provide graphics acceleration, and a rendered image is rendered using the trusted second code module and graphics hardware based on execution of the rendering commands outside the secure runtime environment.

Claims

1. A computer-implemented method, comprising:

executing a native code module in a secure runtime environment, the secure runtime environment preventing the native code module from accessing a graphics rendering interface outside of the secure runtime environment;

generating, using the native code module, rendering commands based on a graphical scene representation; and

transmitting the rendering commands from the native code module within the secure runtime environment to a trusted code module outside the secure runtime environment for execution of the rendering commands by graphics hardware associated with the graphics rendering interface, the trusted code module configured to communicate with the graphics rendering interface to provide graphics acceleration by the graphics hardware.

2. The computer-implemented method of claim 1, wherein transmitting the rendering commands comprises:

writing, using the native code module, the rendering commands to a command buffer shared with the native code module and the trusted code module, and writing image data comprising one or more of vertex data, texture data, and effects data to a data buffer shared with the native code module and the trusted code module.

3. The computer-implemented method of claim 2, further comprising:

generating an animation based on updating parameters in the data buffer for a same set of the rendering commands provided to the command buffer, without regenerating the same set of the rendering commands.

4. The computer-implemented method of claim 1, wherein the graphical scene representation is a scenegraph, the method further comprising:

creating, using the native code module, the scenegraph based on a graphics model comprising a set of shapes, one or more light sources, or one or more rendering effects.

5. The computer-implemented method of claim 4, wherein the scenegraph comprises a shape node, and wherein generating the rendering commands comprises:

transforming the shape node into a set of resource management commands and a set of drawing commands, wherein the resource management commands include commands for creating resources to be used by the trusted code module and for copying data stored in the shape node to the created resources.

6. The computer-implemented method of claim 4, wherein the secure runtime environment is associated with a web application, wherein the graphics model is for the web application, and wherein the graphics acceleration is provided to the web application using the executing, generating, and transmitting steps.

7. The computer-implemented method of claim 6, wherein the web application is associated with a web browser, and wherein a rendered image is generated for display in the web browser using graphics hardware and the rendering commands transmitted from the native code module within the secure runtime environment to a trusted code module outside the secure runtime environment.

8. The computer-implemented method of claim 1, wherein the graphics hardware comprises a graphics-processor unit (GPU) and the graphics rendering interface is an interface of the GPU, and wherein a rendered image is generated for display based on the trusted code module transmitting the rendering commands to the graphics rendering interface for execution using the GPU.

9. The computer-implemented method of claim 1, wherein the graphics rendering interface comprises a low-level rendering application programming interface (API) of a rendering engine not available to the secure runtime environment, and wherein a rendered image is generated for display based on the trusted code module transmitting the rendering commands to the graphics rendering interface for execution using the rendering engine.

10. The computer-implemented method of claim 1, further comprising:

validating native code module before the native code module is loaded into the secure runtime environment; and

loading the native code module in the secure runtime environment after the native code module is validated.

11. The computer-implemented method of claim 1, wherein the native code module comprises binary code that runs directly on hardware.

12. A system, comprising:

one or more processors; and

a memory having instructions stored thereon that, when executed by the one or more processors, perform operations comprising;

executing a native code module in a secure runtime environment, the secure runtime environment preventing the native code module from accessing a graphics rendering interface outside of the secure runtime environment;

generating, using the native code module, rendering commands based on a graphical scene representation; and

transmitting the rendering commands from the native code module within the secure runtime environment to a trusted code module outside the secure runtime environment for execution of the rendering commands outside the secure runtime environment, the trusted code module configured to communicate with the graphics rendering interface to provide graphics acceleration.

13. The system of claim of 12, wherein transmitting the rendering commands comprises:

writing, using the native code module, the rendering commands to a command buffer shared with the native code module and the trusted code module, and writing image data comprising one or more of vertex data, texture data, and effects data to a data buffer shared with the native code module and the trusted code module.

14. The system of claim 13, wherein the operations further comprise:

generating an animation based on updating parameters in the data buffer for a same set of the rendering commands, without regenerating the same set of the rendering commands.

15. The system of claim 12, wherein the graphical scene representation is a scenegraph, the operations further comprising:

creating, using the native code module, the scenegraph based on a graphics model comprising a set of shapes, one or more light sources, or one or more rendering effects.

16. The system of claim 15, wherein the secure runtime environment is associated with a web application, wherein the graphics model is for the web application, and wherein the graphics acceleration is provided to the web application using the executing, generating, and transmitting steps.

17. The system of claim 16, wherein the web application is associated with a web browser, and wherein a rendered image is generated for display in the web browser using graphics hardware and the rendering commands transmitted from the native code module within the secure runtime environment to a trusted code module outside the secure runtime environment.

18. The system of claim 17, wherein the graphics hardware comprises a graphics-processor unit (GPU) and the graphics rendering interface is an interface of the GPU.

19. A non-transitory computer-readable storage medium storing instructions thereon that, when executed by a computer system, cause the computer system to perform a method, comprising:

executing a native code module in a secure runtime environment, the secure runtime environment preventing the native code module from accessing a graphics rendering interface outside of the secure runtime environment;

generating, using the native code module, rendering commands based on a graphical scene representation; and

transmitting the rendering commands from the native code module within the secure runtime environment to a trusted code module outside the secure runtime environment for execution of the rendering commands by graphics hardware associated with the graphics rendering interface, the trusted code module configured to communicate with the graphics rendering interface to provide graphics acceleration by the graphics hardware.

20. The non-transitory computer-readable storage medium of claim 19, wherein the graphical scene representation is a scenegraph, and the secure runtime environment is associated with a web application loaded by a web browser, wherein the method further comprises:

creating, using the native code module, the scenegraph based on a graphics model for the web application, wherein a rendered image is generated for display in the web browser using graphics hardware and the rendering commands transmitted from the native code module within the secure runtime environment to a trusted code module outside the secure runtime environment.