Extended graphics context with divided compositing

• US 8,963,939 B2
• Filed: 09/27/2011
• Issued: 02/24/2015
• Est. Priority Date: 10/01/2010
• Status: Active Grant

• Current Assignee: Z124 LLC (Flexis)
• Original Assignee: Z124 LLC (Flexis)
• Inventors: Yusupov, Alisher, Reeves, Paul E., Chincisan, Octavian, Liu, Wuke
• Primary Examiner(s): Ma, Tize
• Assistant Examiner(s): Chen, Yu
• Application Number:
  US13/246,671
• Publication Number:
  US 20120081354A1
• Time in Patent Office: 1,246 Days
• Field of Search: None
• US Class Current: 345/541
• CPC Class Codes: G06F 2209/545 Gui
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/45533 Hypervisors; Virtual machine monitors
• CPC Class Codes: G06F 9/45537 Provision of facilities of other operating environments, e.g. WINE I/O emulation G06F13/105
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G06F 9/544 Buffers; Shared memory; Pipes
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: G09G 5/399 using two or more bit-mapped memories, the operations of which are switched in time, e.g. ping-pong buffers
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes
• CPC Class Codes: H04L 67/131 Protocols for games, networked simulations or virtual reality
• CPC Class Codes: H04L 69/32 Architecture of open systems interconnection [OSI] 7-layer type protocol stacks, e.g. the interfaces between the data link level and the physical level
• CPC Class Codes: H04L 69/329 in the application layer [OSI layer 7]

Abstract

A mobile computing device with a mobile operating system and desktop operating system running concurrently and independently on a shared kernel without virtualization. The mobile operating system provides a user experience for the mobile computing device that suits the mobile environment. The desktop operating system provides a full desktop user experience when the mobile computing device is docked to a second user environment. Cross-environment rendering and user interaction support provide a seamless computing experience in a multi-operating system computing environment. Real-time display of applications running in the mobile operating system within an environment of the desktop operating system is provided by rendering the application through an extended graphics context of the mobile operating system. Application graphics for multiple applications are rendered into separate graphics frames. The mobile computing device may be a smartphone running the Android mobile operating system and a full desktop Linux distribution on a modified Android kernel.

Claims

1. A method for maintaining display of application graphics for a first application and a second application, both compiled for and in active concurrent execution within a first operating system, the method comprising:

establishing an extended rendering context of the first operating system having a first context space of the extended rendering context associated with the first application and a second context space of the extended rendering context associated with the second application, the first and second context spaces being non-overlapping, wherein the first context space defines a first active display associated with a mobile device and the second context space defines a second active display associated with a desktop computing system, wherein the mobile device and the desktop computing system are distinct computing devices;

allocating anonymous shared memory comprising named memory blocks for surfaces of the first application and the second application by a graphics server of the first operating system, the anonymous shared memory accessible by name through a file descriptor by all processes running on the first operating system and a second operating system and not accessible by name by other processes, the memory blocks of the anonymous shared memory freed by a shared kernel, the first operating system and the second operating system running concurrently and independently on the shared kernel, the shared kernel including an inter-process communications driver which passes the file descriptor to processes in the first operating system and the second operating system to allow communication between the first operating system and the second operating system to communicate across process boundaries;

calculating updates to surfaces of the first application and the second application using the first operating system, wherein the first operating system and the second operating system execute on a mobile device;

rendering the updated surfaces of the first application according to the first context space using the first operating system to generate a first graphics frame in a frame buffer of the first operating system; and

rendering the updated surfaces of the second application according to the second context space using the first operating system to generate a second graphics frame in an anonymous shared memory location accessible by both the first operating system and the second operating system, wherein the updated surfaces of the second application are displayed on the second active display associated with the desktop computing system;

wherein the first operating system is a mobile operating system and the second operating system is a desktop operating system;

wherein when the mobile device is connected to the desktop computing system, a user experiences the second application on the second active display of the desktop computing system through a desktop computing experience of the desktop computing system;

wherein the mobile device is a mobile telephony device; and

wherein each first and second operating system makes system calls through the shared kernel without virtualized memory.

2. The method of claim 1, further comprising:

displaying the first graphics frame associated with the first context space from the frame buffer to the first active display of the mobile device associated with the first operating system; and

displaying the second graphics frame associated with the second context space from the anonymous shared memory location to the second active display of the desktop computing system associated with the second operating system.

3. The method of claim 2, wherein displaying the second graphics frame from the anonymous shared memory location to the second active display of the desktop computing system associated with the second operating system comprises passing a file descriptor from the first operating system to the second operating system indicating that the second graphics frame is available at the anonymous shared memory location.

4. The method of claim 1, wherein the mobile telephony device is a smartphone.

5. The method of claim 1, wherein the second application does not function with a graphics server of the second operating system.

6. The method of claim 1, wherein the second application comprises an mobile device application and a graphics server of the second operating system comprises an desktop graphics server.

7. The method of claim 1, further comprising receiving, by a console application of the second operating system, a draw notification indicating that the second graphics frame is available at the anonymous shared memory location.

8. The method of claim 7, wherein the draw notification is received through an interprocess communication channel.

9. The method of claim 7, wherein the receiving of the draw notification includes receiving a file descriptor to the anonymous shared memory location.

10. The method of claim 9, further comprising receiving a reference to the anonymous shared memory location by mapping the file descriptor to the process space of the console application.

11. The method of claim 10, further comprising accessing, by the console application, the anonymous shared memory location through the reference.

12. The method of claim 1, further comprising registering, in a graphics server of the first operating system, a console application of the second operating system to receive draw notifications for the second application.

13. The method of claim 1, wherein the rendering of the updated surfaces of the first and second applications using the first operating system uses hardware assisted rendering of the first operating system.

14. A mobile computing device, comprising:

a first application and a second application in active concurrent execution within a first operating system;

a first graphics surface of the first operating system updated by the first application;

a second graphics surface of the first operating system updated by the second application;

an extended rendering context of the first operating system, the extended rendering context including a first context space associated with the first application and a second context space of the extended rendering context associated with the second application, the first and second context spaces being non-overlapping, wherein the first context space defines a first active display associated with a mobile device and the second context space defines a second active display associated with a desktop computing system, wherein the mobile device and the desktop computing system are separate computing devices; and

a graphics server of the first operating system that allocates anonymous shared memory for surfaces of the first application and the second application and renders updated surface information of the first surface to generate a first graphics frame in a frame buffer of the first operating system,

an anonymous shared memory comprising named memory blocks accessible by all processes running on the first operating system and the second operating system, the anonymous shared memory accessible by name through a file descriptor by all processes running on the first operating system and the second operating system and not accessible by name by other processes, the memory blocks of the anonymous shared memory freed by a shared kernel, the first operating system and the second operating system running concurrently and independently on the shared kernel, the shared kernel including an inter-process communications driver which allows processes in the first operating system and the second operating system to communicate with one another across process boundaries;

wherein the graphics server renders updated surface information of the second surface to generate a second graphics frame in an anonymous shared memory location accessible by both the first operating system and the second operating system;

wherein the first operating system is a mobile operating system and the second operating system is a desktop operating system;

wherein when the mobile device is connected to the desktop computing system, a user experiences the second application on the second active display of the desktop computing system through a desktop computing experience of the desktop computing system;

wherein the mobile device is a mobile telephony device; and

wherein each first and second operating system makes system calls through the shared kernel without virtualized memory.

15. The mobile computing device of claim 14, further comprising a console application running in the second operating system that accesses the anonymous shared memory location to display the second graphics frame on a display of a user environment associated with the second operating system.

16. The mobile computing device of claim 14, wherein a graphics server of the second operating system is an desktop type graphics server and wherein the second application uses a graphics library of the first operating system that does not function with the desktop type graphics server of the second operating system.

17. A computing device including a non-transitory computer-readable medium storing instructions for a physical processor, the instructions, when executed, causing the processor to perform steps comprising:

executing a first application and a second application concurrently on a first operating system;

establishing an extended rendering context of the first operating system having a first context space of the extended rendering context associated with the first application and a second context space of the extended rendering context associated with the second application, the first and second context spaces being non-overlapping, wherein the first context space defines a first active display associated with a mobile device and the second context space defines a second active display associated with a desktop computing system, wherein the mobile device and the desktop computing system are distinct computing devices;

allocating anonymous shared memory comprising named memory blocks for surfaces of the first application and the second application by a graphics server of the first operating system, the anonymous shared memory accessible by name through a file descriptor by all processes running on the first operating system and a second operating system and not accessible by name by other processes, the memory blocks of the anonymous shared memory freed by a shared kernel, the first operating system and the second operating system running concurrently and independently on the shared kernel;

calculating updates to surfaces of the first application and the second application using the first operating system;

rendering the updated surfaces of the first application according to the first context space using the first operating system to generate a first graphics frame in a frame buffer of the first operating system; and

rendering the updated surfaces of the second application according to the second context space using the first operating system to generate a second graphics frame in an anonymous shared memory location accessible by both the first operating system and the second operating system, wherein the updated surfaces of the second application are displayed on the second active display associated with the desktop computing system;

wherein the first operating system is a mobile operating system and the second operating system is a desktop operating system;

wherein when the mobile device is connected to the desktop computing system, a user experiences the second application on the second active display of the desktop computing system through a desktop computing experience of the desktop computing system;

wherein the mobile device is a mobile telephony device; and

wherein each first and second operating system makes system calls through the shared kernel without virtualized memory.

18. A computing device as defined in claim 17, wherein the processor further performs steps comprising:

displaying the first graphics frame associated with the first context space from the frame buffer to the first active display of the mobile device associated with the first operating system; and

displaying the second graphics frame associated with the second context space from the anonymous shared memory location to the second active display of the desktop computing system associated with the second operating system.

19. A computing device as defined in claim 17, wherein the mobile telephony device is a smartphone.

20. A computing device as defined in claim 17, wherein the processor further performs steps comprising:

receiving, by a console application of the second operating system, a draw notification indicating that the second graphics frame is available at the anonymous shared memory location.