User interaction across cross-environment applications through an extended graphics context

• US 8,933,949 B2
• Filed: 09/27/2011
• Issued: 01/13/2015
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Z124 LLC (Flexis)
• Original Assignee: Z124 LLC (Flexis)
• Inventors: Reeves, Brian, Reeves, Paul E., Teltz, Richard, Reeves, David, Sirpal, Sanjiv, Tyghe, Chris, Liu, Wuke
• Primary Examiner(s): Thompson, James A
• Assistant Examiner(s): Guo, Xilin
• Application Number:
  US13/246,675
• Publication Number:
  US 20120086716A1
• Time in Patent Office: 1,204 Days
• Field of Search: None
• US Class Current: 345/522
• CPC Class Codes: G06F 2009/45579 I/O management, e.g. providing access to device drivers or storage
• CPC Class Codes: G06F 3/1431 using a single graphics controller
• CPC Class Codes: G06F 9/45558 Hypervisor-specific management and integration aspects
• CPC Class Codes: G09G 5/12 Synchronisation between the display unit and other units, e.g. other display units, video-disc players
• CPC Class Codes: G09G 5/14 Display of multiple viewports

Abstract

Cross-environment rendering and user interaction support provide a seamless computing experience in a multi-operating system computing environment. The multi-operating system computing environment may include a mobile operating system and a desktop operating system running concurrently and independently on a shared kernel of a mobile computing device. User interaction support includes handling input events initially received in the shared kernel by accepting the input events in the desktop operating system and translating, mapping, and/or passing the input events through a virtual input device to the mobile operating system such that applications of the mobile operating system receive the input events as if coming from a user interaction space of the mobile operating system. The mobile computing device may be a smartphone running the Android mobile operating system and a full desktop Linux distribution on a modified Android kernel.

Claims

1. A method, comprising:

receiving a first user input event in a first operating system, a first application and a second application in active concurrent execution within the first operating system, the first application displayed within a first user environment associated with the first operating system and the second application displayed within a second user environment associated with a second operating system, the first operating system maintaining application graphics for the second application by rendering a graphics frame for the second application through a first virtual display of an extended rendering context, the graphics frame in a first memory location of anonymous shared memory comprising named memory blocks, the anonymous shared memory accessible by name through a file descriptor by all processes running on the first operating system and the second operating system and not accessible by name by other processes, the memory blocks freed by a single shared kernel, the first operating system and the second operating system running concurrently on the single shared kernel;

establishing an extended input queue of the first operating system having a first motion space and a second motion space, the second motion space associated with the first virtual display;

receiving the first user input event at a first virtual input device from the first console application of the second operating system;

mapping the first virtual input device to the second motion space of the extended input queue of the first operating system; and

passing the first user input event to the second application from the mapped first virtual input device, wherein the first user environment is a mobile device with a first computing environment and the second user environment is a desktop computing system with a second computing environment, wherein the first operating system and the second operating system execute on the mobile device, wherein the single shared kernel includes an inter-process communications driver which passes the file descriptor to processes in the first operating system and the second operating system to allow communication between the first operating system and the second operating system so as to communicate across process boundaries, wherein the mobile device and the desktop computing system are distinct computing devices.

2. The method of claim 1, wherein a third application is in active concurrent execution with the first and second applications within the first operating system, the first operating system rendering application graphics for the third application to a second virtual display, wherein the first virtual display and the second virtual display are not adjacent to each other in the extended input queue.

3. The method of claim 2, wherein the first virtual display and the second virtual display are offset within the extended input queue by a virtual display offset that is larger than a resolution of virtual displays of the first operating system.

4. The method of claim 2, wherein a second console application of the second operating system displays application graphics for the third application rendered by the first operating system to the second virtual display.

5. The method of claim 4, further comprising:

receiving a second user input command at a second virtual input device from the second console application;

mapping the second virtual device to a third motion space of the extended input queue of the first operating system; and

passing the second user input command to the third application from the mapped second virtual input device.

6. The method of claim 1, wherein the first console application translates the first user input event from a first input event protocol of the window system of the second operating system to a second input event protocol of the first operating system.

7. The method of claim 1, wherein the first operating system comprises a mobile operating system and the second operating system comprises a desktop operating system.

8. A mobile computing device, comprising:

a first application and a second application in active concurrent execution within a first operating system;

a first input device that receives a first input event from an input element of a first user environment, the first user environment associated with the first operating system;

a second input device that receives a second input event from an input element of a second user environment, the second user environment associated with a second operating system;

anonymous shared memory comprising named memory blocks accessible by name through a file descriptor by all processes running on the first operating system and the second operating system and not accessible by name by other processes, the memory blocks freed by a single shared kernel, the first operating system and the second operating system running concurrently on the single shared kernel;

a console application running in the second operating system that receives the second input event and passes a third input event based on the second input event to a virtual input device accessible by the first operating system; and

an extended input queue of the first operating system, the extended input queue including a first motion space associated with a local display of the first operating system and a second motion space associated with a first virtual display of the first operating system, the first virtual display associated with the second application, wherein the first operating system maps the virtual input device to the second motion space, wherein the first user environment is the mobile device and the second user environment is a desktop computing system, wherein the first operating system and the second operating system execute on the mobile device, wherein the single shared kernel includes an inter-process communications driver which passes the file descriptor to processes in the first operating system and the second operating system to allow communication between the first operating system and the second operating system so as to communicate across process boundaries, wherein the mobile device and the desktop computing system are distinct computing devices.

9. The mobile computing device of claim 8, wherein a graphics server of the second operating system is a desktop type graphics server and wherein the second application uses a graphics library of the first operating system that is incompatible with the desktop type graphics server of the second operating system.

10. A computing device including a non-transitory computer-readable medium storing instructions for a physical processor, the instructions, when executed, causing the processor to perform steps comprising:

receiving a first user input event in a first operating system, a first application and a second application in active concurrent execution within the first operating system, the first application displayed within a first user environment associated with the first operating system and the second application displayed within a second user environment associated with a second operating system, the first operating system maintaining application graphics for the second application by rendering a graphics frame for the second application through a first virtual display of an extended rendering context, the graphics frame in a first memory location of anonymous shared memory, the anonymous shared memory comprising named memory blocks accessible by name through a file descriptor by all processes running on the first operating system and the second operating system and not accessible by name by other processes, the memory blocks freed by a single shared kernel, the first operating system and the second operating system running concurrently on the single shared kernel;

establishing an extended input queue of the first operating system having a first motion space and a second motion space, the second motion space associated with the first virtual display;

receiving the first user input event at a first virtual input device from the first console application of the second operating system;

mapping the first virtual input device to the second motion space of the extended input queue of the first operating system; and

passing the first user input event to the second application from the mapped first virtual input device, wherein the first user environment is the mobile device and the second user environment is a desktop computing system, wherein the first operating system and the second operating system execute on the mobile device.

11. The computing device of claim 10, wherein a third application is in active concurrent execution with the first and second applications within the first operating system, the first operating system rendering application graphics for the third application to a second virtual display, wherein the first virtual display and the second virtual display are not adjacent to each other in the extended input queue.

12. The computing device of claim 11, wherein the first virtual display and the second virtual display are offset within the extended input queue by a virtual display offset that is larger than a resolution of virtual displays of the first operating system.

13. The computing device of claim 11, wherein a second console application of the second operating system displays application graphics for the third application rendered by the first operating system to the second virtual display.

14. The computing device of claim 13, further comprising:

receiving a second user input command at a second virtual input device from the second console application;

mapping the second virtual device to a third motion space of the extended input queue of the first operating system; and

passing the second user input command to the third application from the mapped second virtual input device.

15. The computing device of claim 10, wherein the first console application translates the first user input event from a first input event protocol of the window system of the second operating system to a second input event protocol of the first operating system.

16. The computing device of claim 10, wherein the first operating system comprises a mobile operating system and the second operating system comprises a desktop operating system.

17. The computing device of claim 10, wherein single shared kernel includes an inter-process communications driver which passes the file descriptor to processes in the first operating system and the second operating system to allow communication between the first operating system and the second operating system so as to communicate across process boundaries.

18. The computing device of claim 17, wherein the mobile device and the desktop computing system are distinct computing devices.