Application mirroring using multiple graphics contexts

• US 9,060,006 B2
• Filed: 09/27/2011
• Issued: 06/16/2015
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Z124 LLC (Flexis)
• Original Assignee: Z124 LLC (Flexis)
• Inventors: Yusupov, Alisher, Reeves, Paul E., Chincisan, Octavian, Liu, Wuke
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Flora, Nurun N
• Application Number:
  US13/246,128
• Publication Number:
  US 20120081353A1
• Time in Patent Office: 1,358 Days
• US Class Current: 1/1
• CPC Class Codes: G06F 2209/545 Gui
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45537 Provision of facilities of other operating environments, e.g. WINE I/O emulation G06F13/105
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G09G 5/399 using two or more bit-mapped memories, the operations of which are switched in time, e.g. ping-pong buffers
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 69/32 Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

Cross-environment rendering and user interaction support provide a seamless computing experience in a multi-operating system computing environment. Cross-environment rendering provides real-time display of applications running in a mobile operating system to be displayed within an environment of a desktop operating system. The mobile operating system and the desktop operating system may be running concurrently and independently on a shared kernel of a mobile computing device. A graphics server of the mobile operating system tears down and rebuilds the rendering context for each application as it composites the surface information. The rendering context may be established to match the resolution of the associated display, so that graphics will be appropriately rendered for that resolution. The mobile computing device may be a smartphone running the Android mobile operating system and a full desktop Linux distribution on a modified Android kernel.

Claims

1. A method comprising:

maintaining display of application graphics for a first application and a second application, both compiled for and in active concurrent execution within a first operating system, by iteratively performing steps comprising;

establishing a first rendering context of the first operating system;

calculating updates to surfaces of the first application using the first operating system;

rendering the updated surfaces of the first application in the first rendering context of the first operating system to generate a first graphics frame in a first memory location of anonymous shared memory comprising named memory blocks, the anonymous shared memory accessible by name through a file descriptor by all processes running on the first operating system and a second operating system and not accessible by name by other processes, the named memory blocks freed by a shared kernel, the first operating system and the second operating system running concurrently and independently on the shared kernel, the shared kernel including an interprocess communication (“

IPC”

) driver which passes the file descriptor to processes in the first operating system and the second operating system to allow communication between the first operating system and the second operating system across process boundaries, wherein each of the first and second operating systems makes system calls through the shared kernel without virtualized memory, wherein the first rendering context defines a first display associated with a mobile device;

disestablishing the first rendering context;

establishing a second rendering context of the first operating system;

calculating updates to surfaces of the second application using the first operating system;

rendering the updated surfaces of the second application in the second rendering context of the first operating system to generate a second graphics frame in a second memory location of the anonymous shared memory, wherein the second rendering context defines a second display associated with a desktop computing system, the second memory location being accessible by both the first operating system and a second operating system; and

disestablishing the second rendering context,

wherein the surfaces of the first application are associated with the mobile device and the surfaces of the second application are associated with the desktop computing system, wherein the first operating system and the second operating system execute on the mobile device, and

wherein device drivers in the single shared kernel implement hardware interfaces.

2. The method of claim 1, wherein the maintaining display of application graphics for the first application and the second application is by iteratively performing steps further comprising:

displaying the first graphics frame from the first memory location to the first display of a first computing environment associated with the first operating system; and

displaying the second graphics frame from the second memory location to a display of a second computing environment associated with the second operating system,

wherein the first computing environment is the mobile device and the second computing environment is the desktop computing system.

3. The method of claim 1, wherein the maintaining display of application graphics for the first application and the second application further comprises receiving, by a console application of the second operating system, a draw notification indicating that the second graphics frame is available at the second memory location.

4. The method of claim 3, wherein the receiving of the draw notification includes receiving a file descriptor to the second memory location.

5. The method of claim 4, wherein the draw notification is received through an inter-process communication channel.

6. The method of claim 5, further comprising receiving a reference to the second memory location by mapping the file descriptor to the process space of the console application, wherein the console application directly reads the first graphics frame in the first memory location, and wherein the console application renders the second application in real-time without copying the first graphics frame and without copying surface information associated with the first graphics frame.

7. The method of claim 1, wherein the second application is incompatible with a graphics server of the second operating system.

8. The method of claim 1, wherein the second application comprises a mobile application and the graphics server of the second operating system comprises a desktop graphics server.

9. The method of claim 1, wherein the second rendering context is established according to characteristics of a display of a second computing environment associated with the second operating system.

19. The method of claim 6, further comprising a graphics server that allocates and tracks displays of the first operating system integrated both within the mobile device and with a remote user interaction space.

10. A mobile computing device, comprising:

a first application running within a first operating system, the first application calculating updates for a first graphics surface using the first operating system;

a second application running within the first operating system, the second application calculating updates for a second graphics surface using the first operating system;

a graphics server of the first operating system, wherein the graphics server establishes a first rendering context of the first operating system, allocates anonymous shared memory for surfaces of the first application and the second application, renders updated surfaces of the first application through the first rendering context to generate a first graphics frame in a first memory location, disestablishes the first rendering context, establishes a second rendering context of the first operating system, renders the updated surfaces of the second application through the second rendering context to generate a second graphics frame in a second memory location, the second memory location being a shared memory location accessible by both the first operating system and a second operating system, and disestablishes the second rendering context; and

an anonymous shared memory comprising named memory blocks accessible by name through a file descriptor by all processes running on the first operating system and the second operating system and managed by a shared kernel, the first operating system and the second operating system running concurrently and independently on the shared kernel, wherein each of the first and second operating systems makes system calls through the shared kernel without virtualized memory, the shared kernel including an interprocess communication (“

IPC”

) driver which allows processes in the first operating system and the second operating system to communicate with one another across process boundaries,

wherein the surfaces of the first application are associated with a mobile device and the surfaces of the second application are associated with a desktop computing system, wherein the first operating system and the second operating system execute on the mobile device, and

wherein device drivers in the single shared kernel implement hardware interfaces.

11. The mobile computing device of claim 10, further comprising a first computing environment associated with the first operating system, the first graphics frame displayed to a display of the first computing environment.

12. A computing environment including the mobile computing device of claim 11, further comprising a second computing environment associated with the second operating system, the second graphics frame displayed to a display of the second computing environment.

13. The mobile computing device of claim 10, further comprising a mobile processor, the first and second operating systems running concurrently on the mobile processor.

14. The mobile computing device of claim 10, further comprising a console application running in the second operating system, wherein the graphics server of the first operating system passes a draw notification to the console application indicating that the second graphics frame is available at the second memory location before disestablishing the second rendering context.

15. The mobile computing device of claim 14, wherein the draw notification includes a file descriptor to the second memory location, wherein the console application directly reads the first graphics frame in the first memory location, and wherein the console application renders the second application in real-time without copying the first graphics frame and without copying surface information associated with the first graphics frame.

16. The mobile computing device of claim 10, wherein a graphics server of the second operating system is a desktop graphics server and wherein the second application uses a graphics library of the first operating system that is incompatible with the desktop graphics server of the second operating system.

20. The mobile computing device of claim 15, wherein the graphics server allocates and tracks displays of the first operating system integrated both within the mobile device and with a remote user interaction space.

17. A computing device including a non-transitory computer-readable medium storing instructions for a physical processor, the instructions, when executed, causing the processor to perform steps comprising:

maintaining display of application graphics for a first application and a second application, both compiled for and in active concurrent execution within a first operating system, by iteratively performing steps comprising;

establishing a first rendering context of the first operating system;

calculating updates to surfaces of the first application using the first operating system;

rendering the updated surfaces of the first application in the first rendering context of the first operating system to generate a first graphics frame in a first memory location of anonymous shared memory, the anonymous shared memory accessible by all processes running on the first operating system and a second operating system and managed by a shared kernel, the first operating system and the second operating system running concurrently and independently on the shared kernel;

disestablishing the first rendering context;

establishing a second rendering context of the first operating system;

calculating updates to surfaces of the second application using the first operating system;

rendering the updated surfaces of the second application in the second rendering context of the first operating system to generate a second graphics frame in a second memory location of the anonymous shared memory comprising named memory blocks, the second memory location being accessible by name through a file descriptor by both the first operating system and a second operating system, wherein each of the first and second operating systems makes system calls through the shared kernel without virtualized memory; and

disestablishing the second rendering context;

wherein the surfaces of the first application are associated with a mobile device and the surfaces of the second application are associated with a desktop computing system, wherein the first operating system and the second operating system execute on the mobile device, wherein a graphics server allocates and tracks displays of the first operating system integrated both within the mobile device and with a remote user interaction space, and

wherein device drivers in the single shared kernel implement hardware interfaces.

18. The computing device of claim 17, wherein the shared kernel includes an IPC driver which allows processes in the first operating system and the second operating system to communicate with one another.