Cross-environment user interface mirroring

• US 8,957,905 B2
• Filed: 09/28/2011
• Issued: 02/17/2015
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Z124 LLC (Flexis)
• Original Assignee: Z124 LLC (Flexis)
• Inventors: Reeves, Brian, Reeves, Paul E., Teltz, Richard, Reeves, David, Sirpal, Sanjiv, Tyghe, Chris, Chincisan, Octavian
• Primary Examiner(s): Ma, Tize
• Assistant Examiner(s): Chen, Yu
• Application Number:
  US13/247,170
• Publication Number:
  US 20120081380A1
• Time in Patent Office: 1,238 Days
• Field of Search: None
• US Class Current: 345/541
• CPC Class Codes: G06F 2209/545 Gui
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45537 Provision of facilities of other operating environments, e.g. WINE I/O emulation G06F13/105
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G09G 5/399 using two or more bit-mapped memories, the operations of which are switched in time, e.g. ping-pong buffers
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 69/32 Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

A mobile computing device with a mobile operating system and desktop operating system running concurrently and independently on a shared kernel without virtualization. The mobile operating system provides a user experience for the mobile computing device that suits the mobile environment. The desktop operating system provides a full desktop user experience when the mobile computing device is docked to a second user environment. Cross-environment rendering and user interaction support provide a seamless computing experience in a multi-operating system computing environment. The seamless computing experience includes mirroring the active user interaction space of the mobile operating system to a display of a user environment associated with the desktop operating system. The mobile computing device may be a smartphone running the Android mobile operating system and a full desktop Linux distribution on a modified Android kernel.

Claims

1. A method comprising:

running a first operating system and a second operating system concurrently and independently on a shared kernel, wherein each first and second operating system makes system calls through the shared kernel without virtualized memory;

calculating, using the first operating system, updates to a set of surfaces of a first application compiled for and in active execution within the first operating system;

rendering, using the first operating system, the set of surfaces according to the updates to generate a graphics frame;

allocating anonymous shared memory comprising named memory blocks for the set of surfaces by a graphics server of the first operating system, the anonymous shared memory accessible by name through a file descriptor by all processes running on the first operating system and the second operating system and not accessible by name by other processes, the named memory blocks of the anonymous shared memory freed by the shared kernel, the shared kernel including an inter-process communications driver which passes the file descriptor to processes in the first operating system and the second operating system to allow communication between the first operating system and the second operating system to communicate across process boundaries;

storing the graphics frame in the anonymous shared memory location;

displaying the graphics frame to a first application display of the first application on a first display of a first computing environment using the first operating system;

passing a file descriptor indicating the anonymous shared memory location to a console application compiled for and in active execution within the second operating system; and

displaying the graphics frame from the shared memory location to a second application display of the first application on a second display of a second computing environment via the console application of the second operating system according to the file descriptor, such that the second application display is identical to the first application display, wherein the first display is associated with a mobile device and the second display is associated with a desktop computing system, wherein the first operating system and the second operating system execute on the mobile device, wherein the first computing environment is the mobile device and the second computing environment is the desktop computing system, wherein the mobile device and the desktop computing system are distinct computing devices;

wherein the first operating system is a mobile operating system and the second operating system is a desktop operating system;

wherein when the mobile device is connected to the desktop computing system, a user experiences the second application on the active second display of the desktop computing system through a desktop computing experience of the desktop computing system; and

wherein the mobile device is a mobile telephony device.

2. The method of claim 1, wherein a graphics server of the second operating system does not function with a drawing object instantiated by the first application.

3. The method of claim 1, wherein a graphics server of the first operating system is does not function with a graphics server of the second operating system.

4. The method of claim 1, wherein the mobile telephony device is a smartphone.

5. The method of claim 1, wherein a graphics server of the second operating system comprises a desktop graphics server.

6. The method of claim 1, further comprising mapping, by the console application, the file descriptor to the process space of the console application to generate a reference to the shared memory location.

7. The method of claim 6, further comprising accessing, by the console application, the graphics frame from the shared memory location via the reference to the shared memory location.

8. The method of claim 1, wherein the set of surfaces of the first application are rendered through a first context space of an extended rendering context of the first operating system.

9. The method of claim 1, wherein the console application is configured to, upon receipt of a first input event from an input device connected to the mobile computing device, generate a second input event based on the first input event, wherein the second input event from the console application is passed to a virtual input device, wherein the first operating system accesses and passes the second input event from the virtual input device to an application running in the first operating system associated with the virtual display.

10. A mobile computing device comprising:

a first application compiled for and in active execution within a first operating system;

a graphics frame for the first application;

a second operating system and the first operating system running concurrently and independently on a shared kernel, wherein each first and second operating system makes system calls through the shared kernel without virtualized memory;

anonymous shared memory comprising named memory blocks accessible by all processes running on the first operating system and the second operating system, the anonymous shared memory accessible by name through a file descriptor by all processes running on the first operating system and the second operating system and not accessible by name by other processes, the named memory blocks of the anonymous shared memory freed by the shared kernel, the shared kernel including an inter-process communications driver which allows processes in the first operating system and the second operating system to communicate with one another across process boundaries, the graphics frame for the first application rendered in the anonymous shared memory location by the first operating system;

a first application display of the first application displayed on a first display of a first computing environment using the first operating system; and

a console application running within the second operating system that displays the graphics frame to a second application display on a display of a second computing environment associated with the second operating system by accessing the anonymous shared memory location of the graphics frame through a file descriptor to the graphics frame passed by the first operating system, wherein the first display is associated with the mobile device and the second display is associated with a desktop computing system, wherein the first operating system and the second operating system execute on the mobile device, wherein the first computing environment is the mobile device and the second computing environment is the desktop computing system, wherein the mobile device and the desktop computing system are distinct computing devices;

wherein the first operating system is a mobile operating system and the second operating system is a desktop operating system;

wherein when the mobile device is connected to the desktop computing system, a user experiences the second application on the second active display of the desktop computing system through a desktop computing experience of the desktop computing system; and

wherein the mobile device is a mobile telephony device.

11. The mobile computing device of claim 10, wherein a graphics server of the second operating system is a desktop graphics server and wherein the first application uses a graphics library of the first operating system that does not function with the desktop graphics server of the second operating system.

12. The mobile computing device of claim 10, wherein the shared memory location comprises a frame buffer of the first operating system.

13. The device of claim 10, wherein the console application is configured to, upon receipt of a first input event from an input device connected to the mobile computing device, generate a second input event based on the first input event, wherein the second input event from the console application is passed to a virtual input device, wherein the first operating system accesses and passes the second input event from the virtual input device to an application running in the first operating system associated with the virtual display.

14. A computing device including a non-transitory computer-readable medium storing instructions for a physical processor, the instructions, when executed, causing the processor to perform steps comprising:

running a first operating system and a second operating system concurrently and independently on a shared kernel, wherein each first and second operating system makes system calls through the shared kernel without virtualized memory;

calculating, using the first operating system, updates to a set of surfaces of a first application compiled for and in active execution within the first operating system;

rendering, using the first operating system, the set of surfaces according to the updates to generate a graphics frame;

allocating anonymous shared memory comprising named memory blocks for the set of surfaces by a graphics server of the first operating system, the anonymous shared memory accessible by name through a file descriptor by all processes running on the first operating system and the second operating system and not accessible by name by other processes, the named memory blocks of the anonymous shared memory freed by the shared kernel;

storing the graphics frame in the anonymous shared memory location;

displaying the graphics frame to a first application display of the first application on a first display of a first computing environment using the first operating system;

passing a file descriptor indicating the anonymous shared memory location to a console application compiled for and in active execution within the second operating system; and

displaying the graphics frame from the anonymous shared memory location to a second application display of the first application on a second display of a second computing environment via the console application of the second operating system according to the file descriptor, such that the second application display is identical to the first application display, wherein the first display is associated with the mobile device and the second display is associated with a desktop computing system, wherein the first operating system and the second operating system execute on the mobile device, wherein the first computing environment is the mobile device and the second computing environment is the desktop computing system, wherein the mobile device and the desktop computing system are distinct computing devices;

wherein the first operating system is a mobile operating system and the second operating system is a desktop operating system;

wherein when the mobile device is connected to the desktop computing system, a user experiences the second application on the second active display of the desktop computing system through a desktop computing experience of the desktop computing system; and

wherein the mobile device is a mobile telephony device.

15. The computing device of claim 14, wherein a graphics server of the second operating system is does not function with a drawing object instantiated by the first application.

16. The computing device of claim 14, wherein a graphics server of the first operating system is does not function with a graphics server of the second operating system.

17. The computing device of claim 14, wherein the mobile telephony device is a smartphone.

18. The computing device of claim 14, wherein the a graphics server of the second operating system comprises a desktop graphics server.

19. The computing device of claim 14, wherein the instructions, when executed, cause the processor to further perform steps comprising mapping, by the console application, the file descriptor to the process space of the console application to generate a reference to the shared memory location.

20. The computing device of claim 19, wherein the instructions, when executed, cause the processor to further perform steps comprising accessing, by the console application, the graphics frame from the shared memory location via the reference to the shared memory location.

21. The computing device of claim 14, wherein the set of surfaces of the first application are rendered through a first context space of an extended rendering context of the first operating system.

22. The computing device of claim 14, wherein the shared kernel includes an inter-process communications driver which allows processes in the first operating system and the second operating system to communicate with one another.