User interface virtualization techniques

• US 9,304,662 B2
• Filed: 10/23/2012
• Issued: 04/05/2016
• Est. Priority Date: 08/25/2011
• Status: Active Grant

• Current Assignee: Omnissa, LLC
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Beveridge, Daniel James
• Primary Examiner(s): PHANTANA ANGKOOL, DAVID
• Application Number:
  US13/658,291
• Publication Number:
  US 20130290857A1
• Time in Patent Office: 1,260 Days
• Field of Search: 715/740
• US Class Current: 1/1
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: H04L 67/01 Protocols

Abstract

User interface virtualization describes a technique for providing a user with access to one computing device from another computing device, while translating the ergonomics of one computer'"'"'s user interface style into the ergonomics of the other'"'"'s. An agent running on a remote desktop exchanges metadata and input events with a corresponding client running on a client device that accepts a “touch and swipe” style input. The agent running on the remote desktop may use an interface interaction API or software framework to programmatically manipulate the user interface of the remote desktop and act as a proxy for the corresponding client. The client constructs and displays native graphical user interface elements at the client device that are virtual representations of a corresponding user interface element at the remote desktop, including file dialogs, clipboards, start menus, and task switchers.

Claims

1. A method of generating a local graphical user interface (GUI) on a touch screen of a client device that is connected to a server device having a remote desktop, the method comprising:

receiving, from the server device, user interface (UI) metadata that specifies content of a GUI element in the remote desktop, and a base image comprising a graphical representation of the remote desktop;

generating, at the client device, a native GUI element based on received UI metadata, wherein the native GUI element comprises a virtual representation of a corresponding GUI element of the remote desktop, wherein the native GUI element includes a copy of content shown in an application within the remote desktop, wherein the copy of continent is provided by the received UI metadata;

generating a local GUI to be displayed on the touch screen of the client device, the local GUI comprising the base image received from the server device and the native GUI element rendered on top of the base image; and

responsive to detecting that the native GUI element has been manipulated through a touch input, modifying a selection region of the native GUI element that indicates a selected portion of the copy of content by using touch input to resize the selection region, and transmitting, to the server device, an input event indicating a manipulation of the corresponding GUI element in the remote GUI by the client device, the input event indicating a selection of a corresponding portion of the content shown in the application within the remote desktop, the input event also indicating a clipboard operation at the remote desktop based on a local selection of the copy of content, as specified by the selection region.

2. The method of claim 1, wherein the generated native GUI element includes a listing of available files based on the received UI metadata that specify a plurality of file objects available at the remote desktop;

wherein the step of transmitting the input event further comprises;

responsive to detecting that the native GUI element has been manipulated through touch input to select one from the listing of available files, transmitting a selection of a corresponding one of the plurality of file objects available at the remote desktop.

3. The method of claim 1, wherein the generated native GUI element comprises a launcher GUI element having contents of a start menu of the remote desktop as provided by the received UI metadata;

wherein the step of transmitting the input event further comprises;

responsive to detecting that the native GUI element has been manipulated through touch input, transmitting the input event indicating selection from the contents of the start menu of the remote desktop.

4. The method of claim 1, wherein the generated native GUI element is configured to display a plurality of images representing a plurality of windows listed in a task switcher GUI element within the remote desktop;

wherein the step of transmitting the input event further comprises;

responsive to detecting that the native GUI element has been manipulated through touch input to select one of the plurality of images, transmitting the input event indicating a selection of one of the plurality of windows in task switcher GUI element of the remote desktop.

5. A method of providing access to a remote desktop of a server device having a guest operating system executing therein, the method comprising:

generating user interface (UI) metadata specifying a graphical user interface (GUI) element in the remote desktop of the server device using an application programming interface of the guest operating system configured to programmatically manipulate the remote desktop of the server device;

transmitting a base image of the GUI and the generated UI metadata to a touch input client device connected to the server device;

receiving, from the touch input client device, an input event indicating manipulation of the GUI element in the remote desktop of the server device; and

invoking the application programming interface to programmatically manipulate the remote desktop of the server device according to the received input event;

detecting invocation, by an application running in the remote desktop, of the GUI element comprising a file dialog configured to display a plurality of files in a filesystem of the guest operating system and enable selection of one of the plurality of files;

in response to detecting the invocation of the file dialog, generating the UI metadata specifying the plurality of files displayed in the file dialog,

wherein the step of generating the UI metadata specifying the plurality of files displayed in the file dialog further comprises;

generating the UI metadata specifying a subset of the plurality of files determined to satisfy a file filter specified by the file dialog.

6. A method of providing access to a remote desktop of a server device having a guest operating system executing therein, the method comprising:

generating user interface (UI) metadata specifying a graphical user interface (GUI) element in the remote desktop of the server device using an application programming interface of the guest operating system configured to programmatically manipulate the remote desktop of the server device;

transmitting a base image of the GUI and the generated UI metadata to a touch input client device connected to the server device;

receiving, from the touch input client device, an input event indicating manipulation of the GUI element in the remote desktop of the server device;

detecting invocation, by an application running in the remote desktop, of the GUI element comprising a file dialog configured to display a plurality of files in a filesystem of the guest operating system and enable selection of one of the plurality of files;

in response to detecting the invocation of the file dialog, generating the UI metadata specifying the plurality of files displayed in the file dialog; and

invoking the application programming interface to programmatically manipulate the remote desktop of the server device according to the received input event, the invoking the application programming interface further comprising invoking the application programming interface to insert a filepath provided by the received input event into a field of the file dialog.

7. A method of providing access to a remote desktop of a server device having a guest operating system executing therein, the method comprising:

generating user interface (UI) metadata specifying a graphical user interface (GUI) element in the remote desktop of the server device using an application programming interface of the guest operating system configured to programmatically manipulate the remote desktop of the server device, wherein the generated UI metadata comprises content visible within bounds of a viewport GUI element of an application on the remote desktop and retrieved using the application programming interface;

transmitting a base image of the GUI and the generated UI metadata to a touch input client device connected to the server device;

receiving, from the touch input client device, an input event indicating manipulation of the GUI element in the remote desktop of the server device;

invoking the application programming interface to programmatically manipulate the remote desktop of the server device according to the received input event, wherein the step of invoking the application programming interface further comprises invoking the application programming interface to select a portion of the content visible within bounds of the viewport GUI element according to the received input event.

8. The method of claim 7, further comprising:

invoking the application programming interface to execute an operation on a clipboard of the remote desktop based on the received input event that specifies the operation to be performed and selected content on which the operation is to be performed.

9. A method of providing access to a remote desktop of a server device having a guest operating system executing therein, the method comprising:

retrieving contents of a start menu UI element on the remote desktop based on usage data that defines a historical pattern of interactions from a touch input client device;

generating user interface (UI) metadata specifying a graphical user interface (GUI) element in the remote desktop of the server device using an application programming interface of the guest operating system configured to programmatically manipulate the remote desktop of the server device;

transmitting a base image of the GUI and the generated UI metadata to the touch input client device connected to the server device;

receiving, from the touch input client device, an input event indicating manipulation of the GUI element in the remote desktop of the server device; and

invoking the application programming interface to programmatically manipulate the remote desktop of the server device according to the received input event, wherein the step of invoking the application programming interface further comprises invoking the application programming interface to launch an item in the start menu UI element on the remote desktop based on a selection specified in the received input event; and

modifying the usage data stored at the server device to include the selection specified in the received input event.

10. A method of providing access to a remote desktop of a server device having a guest operating system executing therein, the method comprising:

generating user interface (UI) metadata specifying a graphical user interface (GUI) element in the remote desktop of the server device using an application programming interface of the guest operating system configured to programmatically manipulate the remote desktop of the server device;

transmitting a base image of the GUI and the generated UI metadata to a touch input client device connected to the server device;

receiving, from the touch input client device, an input event indicating manipulation of the GUI element in the remote desktop of the server device;

invoking the application programming interface to programmatically manipulate the remote desktop of the server device according to the received input event, the invoking the application programming interface including invoking the application programming interface to switch to a top-level window specified by the received input event;

detecting invocation of the GUI element in the remote desktop comprising a task switcher that lists a plurality of top-level windows open on the remote desktop and configured to allow a user to switch to one of the plurality of top-level windows; and

in response to detecting the invocation of the task switcher, generating UI metadata describing the plurality of top-level windows, including graphical image that represents contents of the plurality of top-level windows.