Graphics display system for multiple remote terminals

• US 8,200,796 B1
• Filed: 06/09/2006
• Issued: 06/12/2012
• Est. Priority Date: 05/05/2005
• Status: Active Grant

• Current Assignee: III Holdings 1, LLC
• Original Assignee: Digital Display Innovations, LLC
• Inventors: Margulis, Neal D.
• Primary Examiner(s): Nguyen, Phuoc
• Assistant Examiner(s): Nguyen, Steven
• Application Number:
  US11/450,100
• Time in Patent Office: 2,195 Days
• Field of Search: 709/217, 709/223, 345/1.14, 345/545
• US Class Current: 709/223
• CPC Class Codes: G06F 2213/0038 System on Chip
• CPC Class Codes: G06F 3/1431 using a single graphics controller
• CPC Class Codes: G06F 3/1438 using more than one graphics controller
• CPC Class Codes: G06F 3/1446 display composed of modules, e.g. video walls
• CPC Class Codes: H04N 21/4122 additional display device, e.g. video projector digital output for controlling a plurality of local displays G06F3/1423
• CPC Class Codes: H04N 21/4316 for displaying supplemental content in a region of the screen, e.g. an advertisement in a separate window
• CPC Class Codes: H04N 21/4402 involving reformatting operations of video signals for household redistribution, storage or real-time display details of conversion of video standards at pixel level H04N7/01; video transcoding H04N19/40; adapting incoming signals to the display format of the display terminal G09G5/005; media handling at the source in data packet switching networks H04L65/764
• CPC Class Codes: H04N 21/4437 Implementing a Virtual Machine [VM] virtual machines in general G06F9/45533
• CPC Class Codes: H04N 21/47 End-user applications
• CPC Class Codes: H04N 7/18 Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast

Abstract

An efficient architecture for a virtual desktop system in a Virtual Machine Monitor (VMM) operating environment. In one embodiment, initially, a virtual desktop computer system utilizing one or more host computer systems provides one or more remote display systems with interactive graphics and video capabilities. The host computer system generally manages windows and frames that correspond to remote display systems and manages updating the remote display systems over a network connection. Preferred embodiments include a multi-display processor that is enhanced for VMM operating environments.

VMM operating environments typically consist of different Virtual Machines (VMs) and a control layer known as the Virtual Machine Monitor (VMM) that manages the different VMs. Each VM is typically designed to be unaware of the existence of other VMs and the VMM is designed to provide the necessary abstraction, isolation and coordination between the different VMs to the physical platform hardware.

Claims

1. A multi-user host computer system for supporting multiple active remote terminals each having their own physical input controls, comprising:

a hardware graphics and display subsystem (GPU) having

display memory which can store display frames or subframes for multiple active remote terminals and which is accessible to the GPU without the need to perform a GPU context switch;

one or more drawing engines for generating display frames which may each correspond to a display frame at one of said remote terminals;

tracking means for tracking modified display subframes and for performing selective updates of subframes from said display memory based on this tracking; and

means for isolating virtual machines from each other;

a host CPU

for running more than one CPU virtual machine each of which

corresponds to a virtual desktop for a different user,

includes a virtual graphics machine having virtual GPU status registers, virtual controls and virtual display memory, and

asynchronously generates virtual GPU operations for execution by the physical GPU hardware as if the virtual machine had an exclusive machine view of the hardware GPU;

for running a Virtual Machine Monitor which tracks context and state information for each of said virtual graphics machines and their asynchronous operations and achieves proper ordering and interleaving of said virtual GPU operations to the physical GPU hardware; and

for controlling access by said remote terminals to said CPU virtual machines;

means for connecting said graphics and display subsystem to said host CPU;

means for connecting said multiple active remote terminals via a network to said host CPU;

means for more than one of said virtual machines to asynchronously access said hardware graphics and display subsystem whereby address translation is performed between the CPU virtual machines and GPU display memory, between the GPU subsystem and the GPU display memory, and between the CPU virtual machines and GPU control registers; and

means for managing the selective updates from said GPU display memory by the host CPU so that only the selective updates of said subframes will be transferred via a network subsystem to corresponding ones of said remote terminals.

2. The system of claim 1 wherein each of said virtual machines is able to directly read and write to controls within registers of the hardware GPU and to physical memory within the GPU subsystem that are translated (virtualized) and tracked by the virtual machine monitor to correspond to the context of said virtual machines, and wherein said virtual machine monitor tracking the context and state information replaces a need for the GPU to be context-switched when the GPU is changed to accessing GPU registers and display memory areas that correspond to different virtual machines and different remote terminals.

3. The system of claim 1 wherein said graphics and display (GPU) subsystem manages multiple physical display contexts and translates asynchronous commands from said virtual machines into commands to operate on a respective area of said physical display memory which corresponds to each display context without having to perform a context switch of the hardware GPU.

4. The system of claim 1 wherein said graphics and display subsystem manages multiple physical display contexts each corresponding to a virtual machine and protects the physical display contexts from being accessed by a virtual machine that does not correspond to that physical display context.

5. The system of claim 1 wherein said graphics and display subsystem includes hardware which, with lossless or lossy compression, encodes said modified display frames or subframes from each virtual display memory before they are transferred via said network subsystem to said remote terminals.

6. The system of claim 1 wherein said graphics and display (GPU) subsystem combines frames or subframes from more than one of said virtual machines to produce a combined frame, and transmits said combined frame with said selective updates to said remote terminals and wherein multiple simultaneous virtual machines for one or more combined displays are controlled by one or more remote users.

7. The system of claim 1 wherein a hypervisor running on said host computer system intercepts some or all graphics drawing commands from CPU virtual machines and, instead of running said graphics drawing commands on said graphics and display GPU, forwards graphics drawing commands, and wherein said remote terminals are equipped with appropriate graphics processing hardware.

8. A graphics and display controller for supporting multiple virtual machines, multiple active remote terminals and multiple user inputs, comprising:

physical display memory which can store display frames or subframes for said multiple active remote terminals and which includes means of isolating the physical display memory between different virtual machines;

one or more hardware drawing engines for concurrently generating display frames or subframes which may each correspond to a display frame or subframe at one of said remote terminals;

one or more compression engines for performing lossy or lossless encoding and compression of said frames or subframes;

tracking means for tracking modified display subframes and for performing selective updates of subframes from said display memory based on this tracking;

means for connecting said controller to a host CPU that

runs more than one CPU virtual machine each of which

corresponds to a virtual desktop for a different user,

includes a virtual graphics machine having virtual GPU status registers, virtual controls and virtual display memory, and

asynchronously generates virtual GPU operations for execution by the physical GPU hardware as if the virtual machine had an exclusive machine view of the hardware GPU;

runs a Virtual Machine Monitor which tracks context and state information for each of said virtual graphics machines and their asynchronous operations and achieves proper ordering and interleaving of said virtual GPU operations to the physical GPU hardware; and

controls access by said remote terminals to said CPU virtual machines; and

for translating memory and control accesses from the CPU into context specific operations for virtual machine displays without requiring said graphics and display controller to perform a context switch;

virtual machine monitor means for allowing more than one of said virtual machines to asynchronously access said graphics and display controller without said virtual machines performing software address or control translations to support said graphics and display controller and without the controller having to switch contexts between accesses by said multiple virtual machines; and

means for managing the selective updates from said display memory so that only the selective updates will be transferred via a network subsystem to corresponding ones of said remote terminals.

9. The controller of claim 8 wherein each of said virtual machines is able to read and write to an area of said physical display memory based on a virtual display memory address that at run time is translated in hardware to a corresponding physical display memory address, and wherein the host CPU has a main memory that is physically separate from said physical display memory and the main and display memories are controlled by different memory controllers.

10. The controller of claim 8 wherein said controller manages multiple physical display contexts and translates asynchronous commands from said virtual machines into commands to operate on respective areas of said physical display memory which correspond to each said display context without the need for the GPU to perform a full context switch or change its context registers between operations from different virtual machines.

11. The controller of claim 8 wherein said controller manages multiple physical display memory regions each corresponding to a virtual machine and performs synchronization to maintain proper ordering of execution and memory accesses for each virtual machine.

12. The controller of claim 8 comprising hardware which, with lossy or lossless compression techniques, encodes said modified subframes before they are transferred via said network subsystem.

13. The controller of claim 8 wherein said controller combines frames or subframes from more than one of said virtual machines, said more than one virtual machines being controlled by a remote user with a dedicated physical input device to produce a combined frame and transmits said combined frame with said selective updates via said network subsystem to corresponding ones of said remote terminals.

14. The controller of claim 8 wherein said controller stores a virtual display, encoded in a lossy or lossless compressed format, in said physical display memory and manages said virtual display in said physical memory.

15. The controller of claim 8 wherein multiple virtual displays are encoded and stored in corresponding areas of physical display memory, said controller performs drawing operations on said physical display memory, and frame or subframe data that is encoded in either a lossless or lossy compressed format is transferred via a network to a remote terminal which needs to decode said encoded virtual display prior to display.

16. A method for operating a multi-user host system having multiple virtual machines, networked multiple remote terminals and multiple input controls, comprising the steps of:

using a host computer that

includes software, a main CPU and a hardware graphics and display controller,

runs more than one CPU virtual machine each of which

corresponds to a virtual desktop for a different user,

includes a virtual graphics machine having virtual GPU status registers, virtual controls and virtual display memory, and

asynchronously generates virtual GPU operations for execution by the physical GPU hardware as if the virtual machine had an exclusive machine view of the hardware GPU;

runs a Virtual Machine Monitor which tracks context and state information for each of said virtual graphics machines and their asynchronous operations and achieves proper ordering and interleaving of said virtual GPU operations to the physical GPU hardware; and

controls access by said remote terminals to said CPU virtual machines;

processing said virtual machines independently so that each has access to I/O devices including the graphics and display controller and associated memory subsystems;

processing graphics and display controller commands asynchronously and directly from said virtual machines and translating said commands into context specific commands, without performing a context switch of the graphics and display controller, for said graphics and display controller including managing display frames and subframes for each virtual machine;

determining which of said networked multiple remote terminals is the destination for different ones of said display frames and subframes; and

propagating said display frames and subframes utilizing said main CPU according to network protocol techniques through a network interface to one or more of said networked multiple remote terminals.

17. The method of claim 16 further comprising the step of tracking said frames and subframes such that only changed frames and subframes are propagated through said network interface.

18. The method of claim 16 further comprising the step of encoding said frames and subframes with lossless or lossy compression prior to propagating them through said network interface.

19. The method of claim 16 wherein a virtual machine monitor operates as a hypervisor level on said main CPU to support said multiple virtual machines and manages address translation from each virtual machine into context specific addresses for the physical graphics and display controller.

20. The method of claim 19 wherein said virtual machine monitor sets up virtual I/O devices including the graphics and display controller such that each virtual machine runs an unmodified version of driver software as if it were running on a non-virtualized graphics and display controller, and wherein each virtual machine asynchronously accesses the graphics and display controller.