Cross-environment user interface mirroring using remote rendering

• US 9,049,213 B2
• Filed: 09/27/2011
• Issued: 06/02/2015
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Z124 LLC (Flexis)
• Original Assignee: Z124 LLC (Flexis)
• Inventors: Reeves, Brian, Reeves, Paul E., Teltz, Richard, Reeves, David, Sirpal, Sanjiv, Tyghe, Chris, Chincisan, Octavian
• Primary Examiner(s): Zimmerman, Mark
• Assistant Examiner(s): Flora, Nurun N
• Application Number:
  US13/246,133
• Publication Number:
  US 20120081383A1
• Time in Patent Office: 1,344 Days
• Field of Search: 719/319
• US Class Current: 1/1
• CPC Class Codes: G06F 2209/545 Gui
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45537 Provision of facilities of other operating environments, e.g. WINE I/O emulation G06F13/105
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G09G 5/399 using two or more bit-mapped memories, the operations of which are switched in time, e.g. ping-pong buffers
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 69/32 Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

Cross-environment rendering and user interaction support provide a seamless computing experience in a multi-operating system computing environment. The multi-operating system computing environment may include a mobile operating system and a desktop operating system running concurrently and independently on a mobile computing device. The seamless computing experience includes mirroring the active user interaction space of the mobile operating system to a display of a user environment associated with the desktop operating system. The user interface is rendered by the desktop operating system by accessing surface information of the active user interaction space directly from shared memory. The mobile computing device may be a smartphone running the Android mobile operating system and a full desktop Linux distribution on a modified Android kernel.

Claims

1. A method comprising:

calculating, using a first operating system, updates to a set of surfaces of a first application compiled for and in active execution within the first operating system;

storing the updated set of surfaces in a shared memory location of anonymous shared memory comprising named memory blocks, the anonymous shared memory accessible by all processes running on the first operating system and a second operating system and not accessible by name by other processes and managed by a shared kernel, the first operating system and the second operating system running concurrently and independently on the shared kernel, the shared kernel including an interprocess communication (“

IPC”

) driver which allows processes in the first operating system and the second operating system to communicate with one another across process boundaries, wherein each operating system makes system calls through the shared kernel without virtualized memory;

rendering the updated set of surfaces to generate a first graphics frame using the first operating system;

displaying the first graphics frame to a first application display of the first application on a first display of a first computing environment using the first operating system;

passing a file descriptor indicating the shared memory location of the anonymous shared memory to a console application compiled for and in active execution within the second operating system;

rendering the updated set of surfaces from the shared memory location of the anonymous shared memory to generate a second graphics frame that is substantially identical to the first graphics frame via the console application of the second operating system according to the file descriptor; and

displaying the second graphics frame to a second application display of the first application on a second display of a second computing environment via the console application of the second operating system, such that the second application display is substantially identical

to the first application display, wherein the first display is associated with a mobile device and the second display is associated with a desktop computing system, wherein the first operating system and the second operating system execute on the mobile device, wherein the first computing environment is the mobile device and the second computing environment is the desktop computing system, and wherein device drivers in the single shared kernel implement hardware interfaces.

2. The method of claim 1, wherein a graphics server of the second operating system is incompatible with a drawing object instantiated by the first application.

3. The method of claim 1, wherein a graphics server of the first operating system is incompatible with a graphics server of the second operating system.

4. The method of claim 1, wherein the first operating system comprises a mobile operating system and the second operating system comprises a desktop operating system.

5. The method of claim 1, wherein the first operating system comprises a mobile operating system and a graphics server of the second operating system comprises a desktop graphics server.

6. The method of claim 1, further comprising mapping, by the console application, the file descriptor to the process space of the console application to generate a reference to the shared memory location.

7. The method of claim 6, further comprising accessing, by the console application, the graphics frame from the shared memory location via the reference to the shared memory location, wherein the console application directly reads the first graphics frame in the first memory location, and wherein the console application renders the second application in real-time without copying the first graphics frame and without copying surface information associated with the first graphics frame.

8. A mobile computing device, comprising:

a first application compiled for and in active execution within a first operating system;

a set of surfaces updated by the first application stored in a shared memory location of anonymous shared memory comprising named memory blocks, the anonymous shared memory accessible by all processes running on the first operating system and a second operating system and not accessible by name by other processes and managed by a shared kernel, the first operating system and the second operating system running concurrently and independently on the shared kernel, wherein each operating system makes system calls through the shared kernel without virtualized memory, the shared kernel including an interprocess communication (“

IPC”

) driver which allows processes in the first operating system and the second operating system to communicate with one another across process boundaries;

a first graphics frame for the first application rendered by the first operating system; and

a second graphics frame for the first application rendered by a console application of the second operating system by accessing the updated surfaces from the shared memory location of the anonymous shared memory through a file descriptor passed by the first operating system, wherein device drivers in the single shared kernel implement hardware interfaces.

9. The mobile computing device of claim 8, further comprising a first application display of the first application from the first graphics frame displayed on a first display of a first computing environment associated using the first operating system.

10. The mobile computing device of claim 8, wherein the console application displays a second application display of the first application from the second graphics frame on a second display of a second computing environment using the second operating system.

11. The mobile computing device of claim 10, wherein the second application display is substantially identical to the first application display.

12. The mobile computing device of claim 8, wherein the second operating system further comprises a graphics server that is incompatible with the first application.

13. The mobile computing device of claim 8, wherein the first operating system comprises a mobile operating system and the second operating system comprises a desktop operating system.

14. The mobile computing device of claim 8, wherein the file descriptor is passed through an inter-process communication channel.

15. The mobile computing device of claim 8, wherein the console application receives a reference to the shared memory location by mapping the file descriptor to the process space of the console application, wherein the console application directly reads the first graphics frame in the first memory location, and wherein the console application renders the second application in real-time without copying the first graphics frame and without copying surface information associated with the first graphics frame.

16. The mobile computing device of claim 8, wherein a graphics server of the second operating system is a desktop graphics server and wherein the first application uses a graphics library of the first operating system that is incompatible with the desktop graphics server of the second operating system.

18. The mobile computing device of claim 9, wherein the console application displays a second application display of the first application from the second graphics frame on a second display of a second computing environment using the second operating system, wherein the first computing environment is the mobile device and the second computing environment is a desktop computing system, wherein the first operating system and the second operating system execute on the mobile device.

17. A computing device including a non-transitory computer-readable medium storing instructions for a physical processor, the instructions, when executed, causing the processor to perform steps comprising:

calculating, using a first operating system, updates to a set of surfaces of a first application compiled for and in active execution within the first operating system;

storing the updated set of surfaces in a shared memory location of anonymous shared memory comprising named memory blocks, the anonymous shared memory accessible by all processes running on the first operating system and a second operating system and not accessible by name by other processes and managed by a shared kernel, the first operating system and the second operating system running concurrently and independently on the shared kernel, wherein each operating system makes system calls through the shared kernel without virtualized memory;

rendering the updated set of surfaces to generate a first graphics frame using the first operating system;

displaying the first graphics frame to a first application display of the first application on a first display of a first computing environment using the first operating system;

passing a file descriptor indicating the shared memory location of the anonymous shared memory to a console application compiled for and in active execution within the second operating system;

rendering the updated set of surfaces from the shared memory location of the anonymous shared memory to generate a second graphics frame that is substantially identical to the first graphics frame via the console application of the second operating system according to the file descriptor; and

displaying the second graphics frame to a second application display of the first application on a second display of a second computing environment via the console application of the second operating system, such that the second application display is substantially identical to the first application display;

wherein the first display is associated with a mobile device and the second display is associated with a desktop computing system, wherein the first operating system and the second operating system execute on the mobile device, wherein the first computing environment is the mobile device and the second computing environment is the desktop computing system, and

wherein device drivers in the single shared kernel implement hardware interfaces.

19. The computing device of claim 17, wherein the shared kernel includes an interprocess communication (“

IPC”

) driver which allows processes in the first operating system and the second operating system to communicate with one another.

20. The computing device of claim 17, wherein the console application receives a reference to the shared memory location by mapping the file descriptor to the process space of the console application, wherein the console application directly reads the first graphics frame in the first memory location, and wherein the console application renders the second application in real-time without copying the first graphics frame and without copying surface information associated with the first graphics frame.