CROSS-ENVIRONMENT USER INTERFACE MIRRORING USING REMOTE RENDERING

• US 20120081383A1
• Filed: 09/27/2011
• Published: 04/05/2012
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Z124 LLC (Flexis)
• Original Assignee: IMERJ LLC (Flex, Ltd.)
• Inventors: Reeves, Brian, Reeves, Paul E., Teltz, Richard, Sirpal, Sanjiv, Chincisan, Octavian, Reeves, David, Tyghe, Chris
• Granted Patent: US 9,049,213 B2
• Time in Patent Office: Days
• US Class Current: 345/582
• CPC Class Codes: G06F 2209/545 Gui
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45537 Provision of facilities of other operating environments, e.g. WINE I/O emulation G06F13/105
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G09G 5/399 using two or more bit-mapped memories, the operations of which are switched in time, e.g. ping-pong buffers
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 69/32 Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

Cross-environment rendering and user interaction support provide a seamless computing experience in a multi-operating system computing environment. The multi-operating system computing environment may include a mobile operating system and a desktop operating system running concurrently and independently on a mobile computing device. The seamless computing experience includes mirroring the active user interaction space of the mobile operating system to a display of a user environment associated with the desktop operating system. The user interface is rendered by the desktop operating system by accessing surface information of the active user interaction space directly from shared memory. The mobile computing device may be a smartphone running the Android mobile operating system and a full desktop Linux distribution on a modified Android kernel.

Claims

1. A method comprising:

calculating, using a first operating system, updates to a set of surfaces of a first application compiled for and in active execution within the first operating system;

storing the updated set of surfaces in a shared memory location accessible by both the first operating system and a second operating system running concurrently on a shared kernel;

rendering the updated set of surfaces to generate a first graphics frame using the first operating system;

displaying the first graphics frame to a first application display of the first application on a first display of a first computing environment using the first operating system;

passing a file descriptor indicating the shared memory location to a console application compiled for and in active execution within the second operating system;

rendering the updated set of surfaces from the shared memory location to generate a second graphics frame that is substantially identical to the first graphics frame via the console application of the second operating system according to the file descriptor; and

displaying the second graphics frame to a second application display of the first application on a second display of a second computing environment via the console application of the second operating system, such that the second application display is substantially identical to the first application display.

2. The method of claim 1, wherein a graphics server of the second operating system is incompatible with a drawing object instantiated by the first application.

3. The method of claim 1, wherein a graphics server of the first operating system is incompatible with a graphics server of the second operating system.

4. The method of claim 1, wherein the first operating system comprises a mobile operating system and the second operating system comprises a desktop operating system.

5. The method of claim 1, wherein the first operating system comprises an Android mobile operating system and a graphics server of the second operating system comprises an X-windows graphics server.

6. The method of claim 1, further comprising mapping, by the console application, the file descriptor to the process space of the console application to generate a reference to the shared memory location.

7. The method of claim 6, further comprising accessing, by the console application, the graphics frame from the shared memory location via the reference to the shared memory location.

8. A mobile computing device, comprising:

a first application compiled for and in active execution within a first operating system;

a set of surfaces updated by the first application stored in a shared memory location accessible by both the first operating system and a second operating system running concurrently with the first operating system on a shared kernel;

a first graphics frame for the first application rendered by the first operating system;

a second graphics frame for the first application rendered by a console application of the second operating system by accessing the updated surfaces from the shared memory location through a file descriptor passed by the first operating system.

9. The mobile computing device of claim 8, further comprising a first application display of the first application from the first graphics frame displayed on a first display of a first computing environment associated using the first operating system.

10. The mobile computing device of claim 8, wherein the console application displays a second application display of the first application from the second graphics frame on a second display of a second computing environment using the second operating system.

11. The mobile computing device of claim 10, wherein the second application display is substantially identical to the first application display.

12. The mobile computing device of claim 8, wherein the second operating system further comprises a graphics server that is incompatible with the first application.

13. The mobile computing device of claim 8, wherein the first operating system comprises a mobile operating system and the second operating system comprises a desktop operating system.

14. The mobile computing device of claim 8, wherein the file descriptor is passed through an inter-process communication channel.

15. The mobile computing device of claim 8, wherein the console application receives a reference to the shared memory location by mapping the file descriptor to the process space of the console application.

16. The mobile computing device of claim 8, wherein a graphics server of the second operating system is an X-windows type graphics server and wherein the first application uses a graphics library of the first operating system that is incompatible with the X-windows type graphics server of the second operating system.

17. A computing device including a computer-readable medium storing instructions for a physical processor, the instructions, when executed, causing the processor to perform steps comprising:

calculating, using a first operating system, updates to a set of surfaces of a first application compiled for and in active execution within the first operating system;

storing the updated set of surfaces in a shared memory location accessible by both the first operating system and a second operating system running concurrently on a shared kernel;

rendering the updated set of surfaces to generate a first graphics frame using the first operating system;

displaying the first graphics frame to a first application display of the first application on a first display of a first computing environment using the first operating system;

passing a file descriptor indicating the shared memory location to a console application compiled for and in active execution within the second operating system;

rendering the updated set of surfaces from the shared memory location to generate a second graphics frame that is substantially identical to the first graphics frame via the console application of the second operating system according to the file descriptor; and

displaying the second graphics frame to a second application display of the first application on a second display of a second computing environment via the console application of the second operating system, such that the second application display is substantially identical to the first application display.