User interface virtualization techniques

• US 10,254,929 B2
• Filed: 04/04/2016
• Issued: 04/09/2019
• Est. Priority Date: 08/25/2011
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Beveridge, Daniel James
• Primary Examiner(s): Phantana-angkool, David
• Application Number:
  US15/090,298
• Publication Number:
  US 20160216860A1
• Time in Patent Office: 1,100 Days
• Field of Search: 715740
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: H04L 67/01 Protocols

Abstract

User interface virtualization describes a technique for providing a user with access to one computing device from another computing device, while translating the ergonomics of one computer'"'"'s user interface style into the ergonomics of the other'"'"'s. An agent running on a remote desktop exchanges metadata and input events with a corresponding client running on a client device that accepts a “touch and swipe” style input. The agent running on the remote desktop may use an interface interaction API or software framework to programmatically manipulate the user interface of the remote desktop and act as a proxy for the corresponding client. The client constructs and displays native graphical user interface elements at the client device that are virtual representations of a corresponding user interface element at the remote desktop, including file dialogs, clipboards, start menus, and task switchers.

Claims

1. A method for displaying a graphical user interface (GUI) on a client device connected to a server, the method comprising:

receiving, at the client device from the server, user interface (UI) metadata that describes elements of a GUI on a remote desktop of the server, wherein the UI metadata is received at the client device from a virtual machine (VM) running on the server;

receiving, at the client device from the server, a base image corresponding to the GUI on the remote desktop;

converting, at the client device, the UI metadata into at least one native GUI element that is native to the GUI of the client device;

displaying, at the client device, the base image and the native GUI element on the client device; and

responsive to detecting a manipulation of an element of the GUI of the client device, transmitting, by the client device to the server, an indication of a manipulation of a GUI element of the GUI of the remote desktop, wherein the indication corresponds to the manipulated element of the GUI of the client device.

2. The method of claim 1, wherein:

the manipulation comprises a touch operation on a selection region that includes copy content; and

transmitting the indication of the manipulation includes transmitting a clipboard operation based on the selection region and the copy content.

3. The method of claim 1, wherein the manipulation comprises:

manipulation of a native GUI element of the client device corresponding to a file dialog on the remote desktop.

4. The method of claim 3, further comprising:

receiving UI metadata from the server corresponding to the file dialog; and

converting the UI metadata corresponding to the file dialog into a native file dialog.

5. The method of claim 1, wherein the manipulation comprises:

manipulation of a native GUI element corresponding to a start menu UI element of the remote desktop.

6. The method of claim 1, wherein the manipulation comprises:

manipulation of a native GUI element corresponding to a task switcher UI element of the remote desktop.

7. The method of claim 6, further comprising:

receiving GUI metadata from the server corresponding to the task switcher; and

converting the GUI metadata corresponding to the task switcher to a native GUI element.

8. A method for providing a client device with access to a remote desktop on a server, the method comprising:

transmitting, from the server to the client device, user interface (UI) metadata that describes elements of a GUI on a remote desktop of the server, and that is to be converted at the client device into at least one native GUI element that is native to the GUI of the client device, wherein the UI metadata is transmitted to the client device from a virtual machine (VM) running on the server;

transmitting, from the server to the client device, a base image corresponding to the GUI on the remote desktop;

receiving, at the server from the client device, an indication of a manipulation of a GUI element of the GUI of the remote desktop, wherein the indication corresponds to the manipulated element of the GUI of the client device; and

interacting with the GUI on the remote desktop based on the manipulation.

9. The method of claim 8, wherein:

the manipulation comprises a touch operation on a selection region that includes copy content; and

interacting with the GUI on the remote desktop includes performing a clipboard operation associated with the copy content.

10. The method of claim 8, wherein the manipulation comprises:

the manipulation of manipulation of a native GUI element of the client device corresponding to a file dialog on the remote desktop.

11. The method of claim 10, further comprising:

transmitting UI metadata corresponding to the file dialog to the client device.

12. The method of claim 8, wherein the manipulation comprises:

manipulation of a native GUI element corresponding to a start menu UI element of the remote desktop.

13. The method of claim 8, wherein the manipulation comprises:

manipulation of a native GUI element corresponding to a task switcher UI element of the remote desktop.

14. The method of claim 13, further comprising:

transmitting GUI metadata corresponding to the task switcher to the client device.