Instant remote rendering

• US 9,047,102 B2
• Filed: 09/27/2011
• Issued: 06/02/2015
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Z124 LLC (Flexis)
• Original Assignee: Z124 LLC (Flexis)
• Inventors: Liu, Wuke
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Flora, Nurun N
• Application Number:
  US13/246,665
• Publication Number:
  US 20120086717A1
• Time in Patent Office: 1,344 Days
• Field of Search: 719/319
• US Class Current: 1/1
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G09G 5/393 Arrangements for updating the contents of the bit-mapped memory
• CPC Class Codes: G09G 5/395 Arrangements specially adapted for transferring the contents of the bit-mapped memory to the screen G09G5/399 takes precedence
• CPC Class Codes: G09G 5/399 using two or more bit-mapped memories, the operations of which are switched in time, e.g. ping-pong buffers

Abstract

Cross-environment rendering and user interaction support provide a seamless computing experience in a multi-operating system computing environment. The multi-operating system computing environment may include a mobile operating system and a desktop operating system running concurrently and independently on a mobile computing device. Real-time or instant display of an application running in the mobile operating system within an environment of the desktop operating system is provided by rendering application graphics for the application within the desktop operating system. A console application of the desktop operating system may access surface information for the application from shared memory and render the application within a console window of the computing environment associated with the desktop operating system. The mobile computing device may be a smartphone running the Android mobile operating system and a full desktop Linux distribution on a modified Android kernel.

Claims

1. A method comprising:

maintaining display of application graphics for a first application and a second application, both compiled for and in active concurrent execution within a first operating system, by iteratively performing steps comprising;

calculating updates to surfaces of the first application using the first operating system;

rendering the updated surfaces of the first application using the first operating system to generate a first graphics frame in a first memory location of anonymous shared memory comprising named memory blocks, the anonymous shared memory accessible by name through a file descriptor by all processes running on the first operating system and a second operating system and not accessible by name by other processes, the named memory blocks being freed by a single shared kernel, the first operating system and the second operating system running concurrently and independently on the single shared kernel, wherein each of the first and second operating systems makes system calls through the shared kernel without virtualized memory;

calculating updates to surfaces of the second application using the first operating system;

storing the updated surfaces of the second application in a second memory location, the second memory location being a shared memory location of the anonymous shared memory accessible by both the first operating system and the second operating system; and

rendering the updated surfaces of the second application using a console application of the second operating system to generate a second graphics frame in a third memory location of the anonymous shared memory, wherein the surfaces of the first application are associated with a mobile device and the surfaces of the second application are associated with a desktop computing system, wherein the single shared kernel includes an Inter Process Communication (“

IPC”

) driver which passes the file descriptor to processes in the first operating system and the second operating system to allow communication between the first operating system and the second operating system across process boundaries, and wherein device drivers in the single shared kernel implement hardware interfaces.

2. The method of claim 1, further comprising:

displaying the first graphics frame from the first memory location to a display of a first computing environment associated with the first operating system; and

displaying the second graphics frame from the third memory location to a display of a second computing environment associated with the second operating system.

3. The method of claim 2, wherein the second graphics frame is displayed from the third memory location to a display of a second computing environment associated with the second operating system via the console application.

4. The method of claim 1, further comprising receiving, by the console application, a draw notification indicating that updated surfaces are available at the second memory location.

5. The method of claim 4, wherein the receiving of the draw notification includes receiving a file descriptor to the second memory location.

6. The method of claim 4, wherein the draw notification is received through the inter-process communication channel.

7. The method of claim 4, further comprising receiving a reference to the second memory location by mapping the file descriptor to the process space of the console application.

8. The method of claim 7, wherein rendering the updated surfaces of the second application includes directly reading the second memory location through the reference without copying the updated surfaces.

9. The method of claim 1, further comprising:

instantiating a renderer object in the console application; and

registering, in a graphics server of the first operating system, a remotable interface of the renderer object to receive draw notifications for the surfaces of the second application.

10. The method of claim 1, wherein the first operating system comprises a mobile operating system and the second operating system comprises a desktop operating system.

11. The method of claim 1, wherein the second application is incompatible with a graphics server of the second operating system.

12. The method of claim 1, wherein the second application comprises a mobile application and a graphics server of the second operating system comprises a desktop graphics server.

13. The method of claim 1, wherein rendering the updated surfaces of the second application includes compositing the updated surfaces by a layer order.

21. The method of claim 1, wherein the console application is configured to, upon receipt of a first input event from an input device connected to the mobile computing device, generate a second input event based on the first input event, wherein the second input event from the console application is passed to a virtual input device, wherein the first operating system accesses and passes the second input event from the virtual input device to an application running in the first operating system associated with the virtual display.

22. The method of claim 1, wherein the console application is configured to, upon receipt of a first input event from an input device connected to the mobile computing device, generate a second input event based on the first input event, wherein a virtual input device for input events to the console application is generated and associated with a virtual display of the first operating system, wherein the second input event from the console application is passed to the virtual input device, wherein the first operating system accesses and passes the second input event from the virtual input device to an application running in the first operating system associated with the virtual display.

14. A mobile computing device comprising:

a first application and a second application in active concurrent execution within a first operating system;

a first graphics surface of a first display associated with the first application;

a graphics server of the first operating system operable to render the first graphics surface on the first display to generate a first graphics frame in a first memory location;

anonymous shared memory comprising named memory blocks accessible by name through a file descriptor by all processes running on the first operating system and a second operating system and not accessible by name by other processes, the named memory blocks being freed by a single shared kernel, the first operating system and the second operating system running concurrently and independently on the single shared kernel, wherein device drivers in the single shared kernel implement hardware interfaces, wherein each of the first and second operating systems makes system calls through the shared kernel without virtualized memory;

a second graphics surface of a second display associated with the second application, the second graphics surface stored in a second memory location, the second memory location being a shared memory location of the anonymous shared memory;

a console application running within the second operating system, the console application operable to render the second graphics surface to generate a second graphics frame in a third memory location of the anonymous shared memory wherein the first graphics surface of the first application is associated with a mobile device and the second graphic surface of the second application is associated with a desktop computing system, wherein the first operating system and the second operating system execute on the mobile device, and wherein the single shared kernel includes an Inter Process Communication (“

IPC”

) driver which passes the file descriptor to processes in the first operating system and the second operating system to allow communication between the first operating system and the second operating system across process boundaries.

15. The mobile computing device of claim 14, further comprising a renderer object instantiated by the console application that includes a remotable method for rendering the second graphics surface to generate the second graphics frame.

16. The mobile computing device of claim 14, wherein the console application receives a draw notification from the graphics server of the first operating system indicating that the second graphics surface has been updated, wherein the console application renders the updated surfaces of the second application in real-time without copying the updated surfaces of the second application.

17. The mobile computing device of claim 14, wherein a graphics server of the second operating system is a desktop graphics server and wherein the second application uses a graphics library of the first operating system that is incompatible with the desktop graphics server of the second operating system.

18. A computing device including a non-transitory computer-readable medium storing instructions for a physical processor, the instructions, when executed, causing the processor to perform steps comprising:

maintaining display of application graphics for a first application and a second application, both compiled for and in active concurrent execution within a first operating system, by iteratively performing steps comprising;

calculating updates to surfaces of the first application using the first operating system;

rendering the updated surfaces of the first application using the first operating system to generate a first graphics frame in a first memory location of anonymous shared memory comprising named memory blocks, the anonymous shared memory accessible by name through a file descriptor by all processes running on the first operating system and a second operating system and not accessible by name by other processes, the named memory blocks being freed by a single shared kernel, the first operating system and the second operating system running concurrently and independently on the single shared kernel, wherein each of the first and second operating systems makes system calls through the shared kernel without virtualized memory, wherein device drivers in the single shared kernel implement hardware interfaces;

calculating updates to surfaces of the second application using the first operating system;

storing the updated surfaces of the second application in a second memory location, the second memory location being a shared memory location of the anonymous shared memory accessible by both the first operating system and the second operating system; and

rendering the updated surfaces of the second application using a console application of the second operating system to generate a second graphics frame in a third memory location of the anonymous shared memory, wherein the surfaces of the first application are associated with a mobile device and the surfaces of the second application are associated with a desktop computing system.

19. The computing device of claim 18, wherein the first operating system and the second operating system execute on the mobile device.

20. The computing device of claim 18, wherein the shared kernel includes an IPC driver which allows processes in the first operating system and the second operating system to communicate with one another, wherein the console application renders the updated surfaces of the second application in real-time without copying the updated surfaces of the second application.