Application remoting using network vector rendering

• US 10,558,824 B1
• Filed: 07/03/2019
• Issued: 02/11/2020
• Est. Priority Date: 02/04/2019
• Status: Active Grant

• Current Assignee: CloudFlare Incorporated
• Original Assignee: S2 Systems Corporation (CloudFlare Incorporated)
• Inventors: Remington, Darren, Sundberg, Trevor, Koenig, Killian, Buzbee, Benjamin, Conrad, Michael, Harnett, David
• Primary Examiner(s): Schwartz, Darren B
• Application Number:
  US16/503,393
• Time in Patent Office: 223 Days
• CPC Class Codes: G06F 21/53 by executing in a restricted environment, e.g. sandbox or secure virtual machine
• CPC Class Codes: G06F 21/6245 Protecting personal data, e.g. for financial or medical purposes
• CPC Class Codes: G06F 21/6281 at program execution time, where the protection is within the operating system
• CPC Class Codes: G06F 21/629 to features or functions of an application
• CPC Class Codes: G06F 21/71 to assure secure computing or processing of information
• CPC Class Codes: G06F 2221/032 Protect output to user by software means
• CPC Class Codes: H04L 63/145 the attack involving the propagation of malware through the network, e.g. viruses, trojans or worms
• CPC Class Codes: H04L 63/1483 service impersonation, e.g. phishing, pharming or web spoofing detection of rogue wireless access points H04W12/12
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/564 Enhancement of application control based on intercepted application data
• CPC Class Codes: H04L 67/565 Conversion or adaptation of application format or content adding application control or application functional data H04L67/561

Abstract

Methods, systems, and techniques for application isolation by remote-enabling applications are provided. Example embodiments provide an Adaptive Rendering Application Isolation System (“ARAIS”), which transparently and dynamically enables applications to run in an isolated execution environment yet be rendered locally in a manner that minimizes the amount of data to be transferred and the latency caused by expensive computation and/or by overburdening available bandwidth by remoting rendering using draw commands over rendering using pixel pushing or other techniques. In one embodiment, the ARAIS includes an orchestrator server which comprises remoting level determination logic and rules engine, pre-computed graphics libraries, connection support logic, data repositories for objects such as a render cache, whitelists, blacklists, client privileges, and application information, and one or more secure containers running remote application instances. These components cooperate to deliver isolation-ready technology to client applications.

Claims

1. A computer-implemented method for improving one or more computer technologies, comprising:

under control of a server computing device that is remotely located, separate, and distinct from a client computing device;

receiving from the client computing device a request to initiate a first remote application in the server computing device while the client computing device executes a client application, the request indicating one or more characteristics of the client computing device or the client application;

instantiating an instance of the first remote application in the server computing device, the server computing device having a rules engine;

executing the rules engine to determine a first appropriate interception technique for intercepting rendering commands from the first remote application instance based on evaluation of one or more hierarchical rules against one or more characteristics of the first remote application, the one or more characteristics of the client computing device, or the client application;

intercepting first render commands associated with the first remote application instance;

providing to the client computing device the first render commands to cause the client application to render one or more portions of the output of the first remote application instance based on the first render commands;

receiving from the client computing device an indication of a user action based on an event intercepted by the one or more files; and

causing the client application to update on the client computing device the output received from the first remote application instance by providing to the client computing device one or more second render commands based on the received indication of the user action.

2. The method of claim 1 wherein one or more data objects define the one or more hierarchical rules in a tabular format that defines relationships between (a) one or more appropriate interception techniques or available rendering libraries and (b) one or more characteristics of one or more remote applications, client computing devices, or client applications.

3. The method of claim 1 wherein executing the rules engine to determine a first interception technique comprises:

evaluating a plurality of candidate graphics levels for intercepting rendering commands used to display the output of the first remote application instance;

determining a highest-level rendering library available in the plurality of candidate graphics levels based on the one or more hierarchical rules and the one or more characteristics of the first remote application; and

determining an interception technique in a plurality of interception techniques as the first interception technique for intercepting rendering commands from the first remote application instance based on the determined highest-level rendering library.

4. The method of claim 1 wherein the one or more hierarchical rules define a logical hierarchy for determining a highest performing interception technique in a plurality of candidate interception techniques as the first appropriate interception technique for intercepting rendering commands from the first remote application instance, wherein the plurality of interception techniques include a first candidate interception technique that includes modifying source code of the first remote application to intercept vector draw commands.

5. The method of claim 4 where in the plurality of interception techniques further comprise one or more of:

a second candidate interception technique that includes placing hooks or detours in a graphics library of the first remote application to intercept application rendering functions;

a third candidate interception technique that includes placing hooks or detours in an operating-system-level rendering library to intercept operating system rendering functions;

or

a fourth candidate interception technique that includes intercepting pixel information.

6. The method of claim 1 wherein the one or more hierarchical rules define a logical hierarchy for determining a highest performing interception technique in a plurality of candidate interception techniques as the first appropriate interception technique for intercepting rendering commands from the first remote application instance, wherein the rules engine is configured to determine a highest-level rendering library available in a plurality of candidate graphics levels based on the one or more hierarchical rules and the one or more characteristics of the first remote application, the rules engine is configured to determine a first candidate interception technique in the plurality of candidate interception techniques as the highest performing interception technique based on the determined highest-level rendering library available, and the first candidate interception technique causes the client computing device to display output of the first remote application instance at a higher resolution or consumes less computational resources of the server computing device than one or more other available interception techniques in the plurality of candidate interception techniques.

7. The method of claim 1 wherein the first render commands include draw commands, and providing to the client computing device the draw commands causes the client application to render the one or more portions of the output of the first remote application instance without sending bitmap rasterizations of the draw commands from the server computing device to render the one or more portions of the output of the first remote application instance.

8. The method of claim 1, further comprising:

receiving from the client computing device a request to initiate a second remote application in the server computing device;

instantiating an instance of the second remote application in the server computing device;

executing the rules engine to determine a second appropriate interception technique for intercepting rendering commands from the second remote application instance based on evaluation of one or more hierarchical rules against one or more characteristics of the second remote application or the one or the one or more characteristics of the client computing device or the client application, the second appropriate interception technique being different than the first appropriate interception technique;

intercepting third render commands associated with the second remote application instance; and

providing to the client computing device the third render commands to cause the client application to render one or more portions of the output of the second remote application instance based on the third render commands and the second captured position information.

9. The method of claim 8 wherein the first interception technique includes modifying source code of the first remote application to intercept vector draw commands, and the second interception technique includes placing hooks or detours in a graphics library of the second remote application to intercept application rendering or in an operating-system-level rendering library to intercept operating system rendering functions.

10. The method of claim 8 wherein the first interception technique includes modifying source code of the first remote application to intercept vector draw commands, and the second interception technique includes intercepting pixel information.

11. The method of claim 1 wherein the first remote application instance includes multiple portions of code, and the first interception technique includes:

intercepting first draw commands from a first portion of the multiple portions of code of the first remote application instance, the first draw commands corresponding to one or more first portions of the current view to be rendered and representing only a first portion of all draw commands used to output the current view being rendered; and

intercepting second draw commands from a second portion of the multiple portions of code of the first remote application instance, the second draw commands corresponding to one or more second portions of the current view to be rendered representing only a second portion of all draw commands used to output the current view being rendered.

12. The method of claim 11, further comprising capturing first position information from the first remote application instance that corresponds to the first draw commands and/or capturing second position information from the first remote application instance that corresponds to the second draw commands.

13. The method of claim 11 wherein the first remote application instance has a logical rendering flow, and the second portion of the multiple portions of code of the first remote application instance is downstream in the logical rendering flow from the first portion of the multiple portions of code of the first remote application instance.

14. The method of claim 11 wherein the first portion of the multiple portions of code of the first remote application instance is a compositor of the remote application instance, and the second portion of the multiple portions of code of the first remote application instance is a region renderer of the first remote application instance.

15. The method of claim 1 wherein the first remote application instance has a compositor and a region renderer, and the first interception technique includes:

intercepting first draw commands from the compositor of the first remote application instance, the first draw commands corresponding to one or more portions to be rendered and representing only a portion of all draw commands used to output the current view being rendered;

capturing the first position information from the region renderer of the first remote application instance for the one or more portions that correspond to the first draw commands; and

intercepting second draw commands from the region renderer of the first remote application instance that correspond to one or more regions to be rendered and capturing second position information for the one or more corresponding regions.

16. A computer-implemented method for improving one or more computer technologies, comprising:

under control of a server computing device that is remotely located, separate, and distinct from a client computing device;

receiving from the client computing device a request to initiate a first remote application in the server computing device while the client computing device executes a client application, the request indicating one or more characteristics of the client computing device or the client application;

instantiating an instance of the first remote application in the server computing device, the server computing device having a rules engine;

executing the rules engine to determine a first appropriate interception technique for intercepting rendering commands from the first remote application instance based on evaluation of one or more hierarchical rules against one or more characteristics of the first remote application or the one or more characteristics of the client computing device or the client application;

determining and providing one or more files to the client computing device on demand based on the received request and the first determined appropriate interception technique, the one or more files including application remoting information that modifies the client application executing on the client computing device to display output of the first remote application instance,

wherein the one or more files are configured to be automatically loaded in the client application, and the application remoting information is configured to intercept events from an event loop to listen for events associated with the client application and invoke render handling code, is configured to initiate a connection from the client application to the first remote application instance, includes the render handling code in a compiled instance of a graphics library that is logically associated with the first determined appropriate interception technique, the compiled instance of the graphics library being configured to cause render commands to be rendered in the same manner on the client application as on the first remote application instance to ensure consistent rendering on the client application and on the first remote application instance;

intercepting first render commands associated with the first remote application instance;

providing to the client computing device the first render commands to facilitate the one or more loaded files to cause the client application to render one or more portions of the output of the first remote application instance based on the first render commands;

receiving from the client computing device an indication of a user action based on an event intercepted by the one or more files; and

causing the client application to update on the client computing device the output received from the first remote application instance by providing to the client computing device one or more second render commands based on the received indication of the user action.

17. The method of claim 16 wherein executing the rules engine to determine a first interception technique comprises:

evaluating a plurality of candidate graphics levels for intercepting rendering commands used to display the output of the first remote application instance;

determining a highest-level rendering library available in the plurality of candidate graphics levels based on the one or more hierarchical rules and the one or more characteristics of the first remote application; and

determining an interception technique in a plurality of interception techniques as the first interception technique for intercepting rendering commands from the first remote application instance based on the determined highest-level rendering library.

18. The method of claim 16 wherein the one or more hierarchical rules define a logical hierarchy for determining a highest performing interception technique in a plurality of candidate interception techniques as the first appropriate interception technique for intercepting rendering commands from the first remote application instance, wherein the plurality of interception techniques include a first candidate interception technique that includes modifying source code of the first remote application to intercept vector draw commands.

19. The method of claim 18 wherein the plurality of interception techniques further comprise one or more of:

a second candidate interception technique that includes placing hooks or detours in a graphics library of the first remote application to intercept application rendering functions;

a third candidate interception technique that includes placing hooks or detours in an operating-system-level rendering library to intercept operating system rendering functions;

or

a fourth candidate interception technique that includes intercepting pixel information.

20. The method of claim 16 wherein the one or more hierarchical rules define a logical hierarchy for determining a highest performing interception technique in a plurality of candidate interception techniques as the first appropriate interception technique for intercepting rendering commands from the first remote application instance, wherein the rules engine is configured to determine a highest-level rendering library available in a plurality of candidate graphics levels based on the one or more hierarchical rules and the one or more characteristics of the first remote application, the rules engine is configured to determine a first candidate interception technique in the plurality of candidate interception techniques as the highest performing interception technique based on the determined highest-level rendering library available, and the first candidate interception technique causes the client computing device to display the output of the first remote application instance at a higher resolution or consumes less computational resources of the server computing device than one or more other available interception techniques in the plurality of candidate interception techniques.

21. The method of claim 16 wherein the first render commands include draw commands, and providing to the client computing device the draw commands causes the client application to render the one or more portions of the output of the first remote application instance without sending bitmap rasterizations of the draw commands from the server computing device to render the one or more portions of the output of the first remote application instance.

22. The method of claim 16, further comprising:

receiving from the client computing device a request to initiate a second remote application in the server computing device;

instantiating an instance of the second remote application in the server computing device;

executing the rules engine to determine a second appropriate interception technique for intercepting rendering commands from the second remote application instance based on evaluation of one or more hierarchical rules against one or more characteristics of the second remote application or the one or the one or more characteristics of the client computing device or the client application, the second appropriate interception technique being different than the first appropriate interception technique;

determining and providing another one or more files to the client computing device on demand based on the received request to initiate the second remote application and the second determined appropriate interception technique, the other one or more files including further application remoting information that modifies the client application executing on the client computing device to display output of the second remote application instance;

intercepting third render commands associated with the second remote application instance; and

providing to the client computing device the third render commands to facilitate the other one or more files to cause the client application to render one or more portions of the output of the second remote application instance based on the third render commands and the second captured position information.

23. The method of claim 22 wherein the first interception technique includes modifying source code of the first remote application to intercept vector draw commands, and the second interception technique includes placing hooks or detours in a graphics library of the second remote application to intercept application rendering or in an operating-system-level rendering library to intercept operating system rendering functions.

24. The method of claim 22 wherein the first interception technique includes modifying source code of the first remote application to intercept vector draw commands, and the second interception technique includes intercepting pixel information.

25. The method of claim 16 wherein the first remote application instance includes multiple portions of code, and the first interception technique includes:

intercepting first draw commands from a first portion of the multiple portions of code of the first remote application instance, the first draw commands corresponding to one or more first portions of the current view to be rendered and representing only a first portion of all draw commands used to output the current view being rendered; and

intercepting second draw commands from a second portion of the multiple portions of code of the first remote application instance that correspond to one or more second portions of the current view to be rendered and representing only a second portion of all draw commands used to output the current view being rendered.

26. The method of claim 25 wherein the first remote application instance has a logical rendering flow, and the second portion of the multiple portions of code of the first remote application instance is downstream in the logical rendering flow from the first portion of the multiple portions of code of the first remote application instance.

27. The method of claim 25 wherein the first portion of the multiple portions of code of the first remote application instance is a compositor of the remote application instance, and the second portion of the multiple portions of code of the first remote application instance is a region renderer of the first remote application instance.

28. The method of claim 16 wherein the first remote application instance has a compositor and a region renderer, and the first interception technique includes:

intercepting first draw commands from the compositor of the first remote application instance, the first draw commands corresponding to one or more tiles to be rendered and representing only a portion of all draw commands used to output the current view being rendered;

capturing the first position information from the region renderer of the first remote application instance for the one or more tiles that correspond to the first draw commands; and

intercepting second draw commands from the region renderer of the first remote application instance that correspond to one or more regions to be rendered and capturing second position information for the one or more corresponding regions.

29. The method of claim 16 wherein the one or more files include a JavaScript file that includes one or more portions of the application remoting information,

wherein the JavaScript file is configured to interface with an application programming interface (“

API”

) of the client web browser application to cause the client web browser application to load a second file that includes the compiled instance of the graphics library, is configured to intercept the events from the event loop to listen for the events associated with the client application and invoke the render handling code, and is configured to initiate the connection from the client application to the first remote application instance.

30. The method of claim 16 wherein the one or more files include a WebAssembly file that includes one or more portions of the application remoting information, and the WebAssembly file includes the render handling code in the compiled instance of the graphics library.