Web browser remoting using network vector rendering

• US 10,452,868 B1
• Filed: 07/03/2019
• Issued: 10/22/2019
• Est. Priority Date: 02/04/2019
• Status: Active Grant

• Current Assignee: CloudFlare Incorporated
• Original Assignee: S2 Systems Corporation (CloudFlare Incorporated)
• Inventors: Sundberg, Trevor, Koenig, Killian, Remington, Darren, Buzbee, Benjamin, Conrad, Michael, Harnett, David
• Primary Examiner(s): Schwartz, Darren B
• Application Number:
  US16/503,388
• Time in Patent Office: 111 Days
• CPC Class Codes: G06F 11/3656 using a specific debug interface
• CPC Class Codes: G06F 21/51 at application loading time, e.g. accepting, rejecting, starting or inhibiting executable software based on integrity or source reliability
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: G06F 21/6245 Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 21/6281 at program execution time, where the protection is within the operating system
• CPC Class Codes: G06F 21/629 to features or functions of an application
• CPC Class Codes: G06F 21/71 to assure secure computing or processing of information
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: H04L 63/1416 Event detection, e.g. attack signature detection
• CPC Class Codes: H04L 63/1441 Countermeasures against malicious traffic countermeasures against attacks on cryptographic mechanisms H04L9/002

Abstract

Methods, systems, and techniques for application isolation by remote-enabling applications are provided. Example embodiments provide an Adaptive Rendering Application Isolation System (“ARAIS”), which transparently and dynamically enables applications to run in an isolated execution environment yet be rendered locally in a manner that minimizes the amount of data to be transferred and the latency caused by expensive computation and/or by overburdening available bandwidth by remoting rendering using draw commands over rendering using pixel pushing or other techniques. In one embodiment, the ARAIS includes an orchestrator server which comprises remoting level determination logic and rules engine, pre-computed graphics libraries, connection support logic, data repositories for objects such as a render cache, whitelists, blacklists, client privileges, and application information, and one or more secure containers running remote application instances. These components cooperate to deliver isolation-ready technology to client applications.

Claims

1. A computer-implemented method for improving one or more computer technologies, comprising:

under control of an execution computing device that is remotely located, separate, and distinct from a rendering computing device;

receiving from the rendering computing device a request to initiate a corresponding remote application in the execution computing device while the rendering computing device executes a web browser application;

instantiating an instance of the remote application in the execution computing device, the remote application instance having a compositor and a region renderer;

causing a connection between the remote application instance and the web browser application to facilitate the rendering computing device to display output of the remote application instance;

under control of the connected remote application instance;

intercepting first draw commands from the compositor of the remote application instance, the first draw commands corresponding to one or more tiles to be rendered and representing only a portion of all draw commands used to output a current page being rendered;

capturing first position information from the region renderer of the remote application instance for the one or more tiles that correspond to the first draw commands;

intercepting second draw commands from the region renderer of the remote application instance that correspond to one or more regions to be rendered and capturing second position information for the one or more corresponding regions; and

providing to the rendering computing device the first and second draw commands with the first and second captured position information to cause the web browser application to render one or more portions of the output of the remote application instance based on the first and second draw commands and the first and second captured position information without sending bitmap rasterizations of the first and second draw commands from the execution computing device to render the one or more portions of the output of the remote application instance.

2. The method of claim 1, further comprising:

under control of the connected remote application instance;

receiving from the rendering computing device an indication of a user action; and

causing the web browser application to update on the rendering computing device the output received from the remote application instance by providing to the rendering computing device one or more third draw commands with position information based on the received indication of the user action.

3. The method of claim 1 wherein the one or more tiles are devoid of bitmaps to reduce a computational load on the executing computing device.

4. The method of claim 1 wherein the remote application instance has a layout engine that generates a picture layer, and wherein intercepting first draw commands from the compositor of the remote application instance includes intercepting draw commands associated with the generated picture layer.

5. The method of claim 4 wherein intercepting draw commands associated with the generated picture layer only intercepts draw commands that correspond to the one or more tiles to be rendered.

6. The method of claim 1 wherein the first and second draw commands are configured to cause the web browser application to render one or more layers to a cache based on the first and second draw commands and the first and second captured position information received from the executing computing device, wherein one or more portions of the cache are caused to be rendered on a display of the rendering computing device based on a portion of the received first and second draw commands, and wherein one or more other portions of the cache are not rendered on the display of the rendering computing device concurrently with the one or more portions of the cache and are based on another portion of the received first and second draw commands.

7. The method of claim 6 wherein the cache includes one or more of an image cache, a texture cache, a font cache, a shape cache, a curve cache, or a draw command cache.

8. The method of claim 6 wherein, upon notification of a scrolling event or another event associated with a user action, the web browser application is caused to render on the display of the rendering computing device some part of the one or more other portions of the cache without receiving further draw commands from the execution computing device.

9. The method of claim 1, further comprising:

determining that the user action is associated with an event that causes one or more portions of a cache of the remote application instance to be updated;

intercepting third draw commands from one or more of the compositor or the region renderer of the remote application instance, the third draw commands being associated with updating the one or more portions of the cache of the remote application instance;

updating the one or more portions of the cache of the remote application instance; and

providing to the rendering computing device the third draw commands to cause the rendering computing device to render an update to one or more portions of a cache of the web browsing application, the one or more portions of the cache of the web browsing application corresponding to the one or more updated portions of the cache of the remote application instance.

10. The method of claim 1, further comprising:

evaluating one or more characteristics of one or more of the remote application instance or a data channel connecting the execution computing device and the rendering computing device;

determining whether providing to the rendering computing device fourth draw commands associated with one or more portions of the current page being rendered facilitates higher performance than providing to the rendering computing device pixel information associated with the one or more portions of the current page being rendered based on the evaluation of the one or more characteristics; and

providing to the rendering computing device a selected one of the fourth draw commands or the pixel information based on the determination of whether providing the fourth draw commands facilitates higher performance than providing the pixel information.

11. The method of claim 10 wherein the evaluating includes determining a data bandwidth and/or a latency and comparing the data bandwidth and/or the latency to one or more thresholds to determine whether providing the fourth draw commands or the pixel information facilitates higher performance.

12. A computer-implemented method for improving one or more computer technologies, comprising:

under control of a code execution computing device that is remotely located, separate, and distinct from an output rendering computing device;

receiving from the output rendering computing device a request to initiate a corresponding remote application in the code execution computing device while the output rendering computing device executes a web browser application, the request indicating an operating system of the output rendering computing device and a browser type of the web browser application;

instantiating an instance of the remote application in the code execution computing device, the remote application instance having a compositor, a region renderer, and a rules engine;

executing the rules engine to determine one or more of a current compositor, a current region renderer, or a current graphics library based on one or more of the operating system of the output rendering computing device and/or the browser type of the web browser application in the output rendering computing device; and

determining and providing a set of files to the output rendering computing device on demand based on the received request and the one or more of the current compositor, the current region renderer, or the current graphics library, the set of files containing compiled draw commands and information that modifies the web browser application executing on the output rendering computing device to display output of the remote application instance,

wherein one or more files in the set of files is configured to interface with an application programming interface (“

API”

) of the web browser application to cause the web browser application to load one or more files in the set of files, is configured to intercept events from an event loop to listen for events associated with the web browser application and invoke draw handling code, and is configured to initiate a connection from the web browser application to the remote application instance,

wherein one or more files in the set of files includes the draw handling code in a compiled instance of a graphics library of the remote application instance, the compiled instance of the graphics library being configured to cause draw commands to be rendered in the same manner on the web browser application as on the remote application instance to ensure consistent rendering on the web browser application and on the remote application instance, and

wherein one or more files in the set of files is configured to cause the web browser application to generate a drawing canvas;

causing a connection between the remote application instance and the web browser application to facilitate the output rendering computing device to display output of the remote application instance; and

under control of the connected remote application instance, providing to the output rendering computing device draw commands to facilitate the one or more loaded files to cause the web browser application to render one or more portions of the output of the remote application instance based on the draw commands without sending bitmap rasterizations of the draw commands from the code execution computing device to render the one or more portions of the output of the remote application instance.

13. The method of claim 12, further comprising:

implementing the current compositor as the compositor of the instantiated remote application;

implementing the current region renderer as the region renderer of the instantiated remote application; and

implementing the current graphics library as the graphics library of the instantiated remote application.

14. The method of claim 12 wherein the set of files include a script file, a rendering support file, and a canvas file, the script file being configured to (i) interface with the API of the web browser application, (ii) intercept events from the event loop, and (iii) initiate the connection from the web browser application to the remote application instance, the rendering support file including the draw handling code, and the canvas file being configured to cause the web browser application to generate the drawing canvas.

15. The method of claim 12 wherein one or more files in the set of files is configured to cause the web browser application to render one or more layers to a cache based on the draw commands received from the code execution computing device without receiving bitmap rasterizations of the received draw commands, wherein one or more portions of the cache are caused to be rendered on a display of the output rendering computing device based on a portion of the received draw commands, and wherein one or more other portions of the cache are not rendered on the display of the output rendering computing device concurrently with the one or more portions of the cache and are based on another portion of the received draw commands.

16. The method of claim 15 wherein the cache includes one or more of an image cache, a texture cache, a font cache, a shape cache, a curve cache, or a draw command cache.

17. The method of claim 15 wherein one or more files in the set of files, upon notification of a scrolling event or another event associated with a user action, causes the web browser application to render on the display of the output rendering computing device some part of the one or more other portions of the cache without receiving further draw commands from the code execution computing device.

18. A computer-implemented method for improving one or more computer technologies, comprising:

under control of a server computing device that is remotely located, separate, and distinct from a client computing device;

receiving from the client computing device a request to initiate a corresponding remote application in the server computing device while the client computing device executes a client web browser application, the request indicating an operating system of the client computing device and a browser type of the client web browser application;

instantiating an instance of the remote application in the server computing device, the remote application instance having a compositor, a region renderer, and a rules engine;

executing the rules engine to determine one or more of a current compositor, a current region renderer, or a current graphics library based on one or more of the operating system of the client computing device and/or the browser type of the client web browser application in the client computing device;

determining and providing a set of files to the client computing device on demand based on the received request and the one or more of the current compositor, the current region renderer, or the current graphics library, the set of files including a script file, an executable code file, and a Hypertext Markup Language (“

HTML”

) file that include information that modifies the client web browser application executing on the client computing device to display output of the remote application instance,

wherein the script file is configured to interface with an application programming interface (“

API”

) of the client web browser application to cause the client web browser application to load the executable code file, is configured to intercept events from an event loop to listen for events associated with the client web browser application and invoke draw handling code, and is configured to initiate a connection from the client web browser application to the remote application instance,

wherein the executable code file includes the draw handling code in a compiled instance of a graphics library of the remote application instance, the compiled instance of the graphics library being configured to cause draw commands to be rendered in the same manner on the client web browser application as on the remote application instance to ensure consistent rendering on the client web browser application and on the remote application instance, and

wherein the HTML file is configured to cause the client web browser application to generate a drawing canvas;

under control of the connected remote application instance;

intercepting first draw commands from the compositor of the remote application instance, the first draw commands corresponding to one or more tiles to be rendered and representing only a portion of all draw commands used to output a current page being rendered;

capturing first position information from the region renderer of the remote application instance for the one or more tiles that correspond to the first draw commands;

intercepting second draw commands from the region renderer of the remote application instance that correspond to one or more regions to be rendered and capturing second position information for the one or more corresponding regions;

providing to the client computing device the first and second draw commands with the first and second captured position information to facilitate the loaded executable code file to cause the client web browser application to render one or more portions of the output of the remote application instance based on the first and second draw commands and the first and second captured position information without sending bitmap rasterizations of the first and second draw commands from the server computing device to render the one or more portions of the output of the remote application instance;

receiving from the client computing device an indication of a user action based on an event intercepted by the script file; and

causing the client web browser application to update on the client computing device the output received from the remote application instance by providing to the client computing device one or more third draw commands with position information based on the received indication of the user action.

19. The method of claim 18, further comprising:

implementing the current compositor as the compositor of the instantiated remote application;

implementing the current region renderer as the region renderer of the instantiated remote application; and

implementing the current graphics library as the graphics library of the instantiated remote application.

20. The method of claim 18 wherein the one or more tiles are devoid of bitmaps.

21. The method of claim 18 wherein the remote application instance has a layout engine that generates a picture layer, wherein intercepting first draw commands from the compositor of the remote application instance includes intercepting draw commands associated with the generated picture layer, and wherein intercepting draw commands associated with the generated picture layer only intercepts draw commands that correspond to the one or more tiles to be rendered.

22. The method of claim 18 wherein the loaded executable code file is configured to cause the client web browser application to render one or more layers to a cache based on the first and second draw commands and the first and second captured position information received from the server computing device, wherein one or more portions of the cache are caused to be rendered on a display of the client computing device based on a portion of the received first and second draw commands, and wherein one or more other portions of the cache are not yet caused to be rendered on the display of the client computing device and are based on another portion of the received first and second draw commands.

23. The method of claim 22 wherein the cache includes one or more of an image cache, a texture cache, a font cache, a shape cache, a curve cache, or a draw command cache.

24. The method of claim 22 wherein the loaded executable code file, upon notification of a scrolling event or another event associated with a user action, causes the client web browser application to render on the display of the client computing device some part of the one or more other portions of the cache not yet caused to be rendered without receiving further draw commands from the server computing device.

25. The method of claim 18, further comprising:

determining that the user action is associated with an event that causes one or more portions of a cache of the remote application instance to be updated;

intercepting the third draw commands from one or more of the compositor or the region renderer of the remote application instance, the third draw commands being associated with updating the one or more portions of the cache of the remote application instance;

updating the one or more portions of the cache of the remote application instance; and

providing to the client computing device the third draw commands to cause the client computing device to render an update to one or more portions of a cache of the client web browsing application, the one or more portions of the cache of the client web browsing application corresponding to the one or more updated portions of the cache of the remote application instance.

26. The method of claim 18, further comprising:

evaluating one or more characteristics of one or more of the remote application instance or a data channel connecting the server computing device and the client computing device;

determining whether providing to the client computing device fourth draw commands associated with one or more portions of the current page being rendered facilitates higher performance than providing to the client computing device pixel information associated with the one or more portions of the current page being rendered based on the evaluation of the one or more characteristics; and

providing to the client computing device a selected one of the fourth draw commands or the pixel information based on the determination of whether providing the fourth draw commands or the pixel information facilitates higher performance.

27. The method of claim 26 wherein the evaluating includes determining a data bandwidth and/or a latency and comparing the data bandwidth and/or the latency to one or more thresholds to determine whether providing the fourth draw commands facilitates higher performance than providing the pixel information.

28. A computer-implemented method for improving one or more computer technologies, comprising:

under control of a server computing device that is remotely located, separate, and distinct from a rendering computing device;

receiving from the rendering computing device a request to initiate a corresponding remote application in the executing computing device while the rendering computing device executes a web browser application;

instantiating an instance of the remote application in the execution computing device, the remote application instance having multiple portions of code for rendering output;

causing a connection between the remote application instance and the web browser application to facilitate the rendering computing device to display output of the remote application instance;

under control of the connected remote application instance;

intercepting first draw commands from a first portion of the multiple portions of code of the remote application instance, the first draw commands corresponding to one or more first portions of a current page to be rendered and representing only a first portion of all draw commands used to output the current page being rendered;

capturing first position information from the remote application instance that corresponds to the first draw commands;

intercepting second draw commands from a second portion of the multiple portions of code of the remote application instance that correspond to one or more second portions of the current page to be rendered and representing only a second portion of all draw commands used to output the current page being rendered;

capturing second position information from the remote application instance that corresponds to the second draw commands; and

providing to the rendering computing device the first and second draw commands with the corresponding first and second captured position information to cause the web browser application to render one or more portions of the output of the remote application instance based on the first and second draw commands and the corresponding first and second captured position information without sending bitmap rasterizations of the first and second draw commands from the execution computing device to render the one or more portions of the output of the remote application instance.

29. The method of claim 28 wherein the remote application instance has a logical rendering flow and the second portion of the multiple portions of code of the remote application instance is downstream in the logical rendering flow from the first portion of the multiple portions of code of the remote application instance.

30. The method of claim 28 wherein the first portion of the multiple portions of code of the remote application instance is a compositor of the remote application instance, and the second portion of the multiple portions of code of the remote application instance is a region renderer of the remote application instance.