User Interface Virtualization for Remote Devices

• US 20130290856A1
• Filed: 10/08/2012
• Published: 10/31/2013
• Est. Priority Date: 08/25/2011
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: BEVERIDGE, Daniel James, HIRSTIUS, Charles Andrew, LEIBOVICI, Andre
• Granted Patent: US 9,250,854 B2
• Time in Patent Office: Days
• US Class Current: 715/740
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/455 Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines

Abstract

User interface virtualization describes a technique for providing a user with access to one computing device from another computing device, while translating the ergonomics of one computer'"'"'s user interface style into the ergonomics of the other'"'"'s. An agent running on a remote desktop collaborates with a corresponding client running on a client machine that accepts a “touch and swipe” style input. The agent and client exchange user interface metadata and user interface input events and translate the exchanged information to provide native graphical user interface elements (at the client machine) and simulated user actions (at the remote desktop). The agent running on the remote desktop may use an interface interaction API or library to programmatically manipulate the user interface of the remote desktop responsive and act as a proxy for the corresponding client.

Claims

1. A method of generating a local graphical user interface (GUI) on a touch screen of a client device that is connected to a server device having a remote GUI, the method comprising:

receiving, from the server device, a base image of the remote GUI and user interface (UI) metadata describing a GUI element in the remote GUI;

generating, at the client device, a native GUI element to be displayed on the touch screen according to the received UI metadata, the native GUI element corresponding to the GUI element in the remote GUI; and

generating the local GUI to be displayed on the touch screen of the client device, the local GUI including the base image received from the server device and the native GUI element.

2. The method of claim 1, wherein the UI metadata comprises semantic information related to contents of the GUI element in the remote GUI, wherein the UI metadata is organized in a tree-like structure.

3. The method of claim 1, wherein the generated native GUI element is different than the corresponding GUI element in the remote GUI in at least one of size, shape, color, style, manner of interaction, animation, and interactive behavior.

4. The method of claim 1, wherein the step of generating the native GUI element further comprises:

generating a system-provided control element based on the received UI metadata using a user interface framework local to the client device.

5. The method of claim 1, wherein the local GUI element comprises the native GUI element arranged on top of the base image of the remote GUI received from the server device.

6. The method of claim 1, further comprising:

receiving, at the client device, a touch input through the local GUI; and

responsive to determining that the native GUI element has been manipulated through the received input, transmitting, to the server device, information indicating that the corresponding GUI element in the remote GUI has been manipulated at the client device.

7. The method of claim 6, wherein the information transmitted to the server device comprises information of a change in application state of a portion of the remote GUI.

8. The method of claim 6, wherein the information transmitted to the server device comprises event information configured to trigger an accessibility assistive application programming interface of the server device, wherein the accessibility assistive application programming interface is configured to programmatically manipulate the remote GUI of the server device.

9. The method of claim 6, wherein the information transmitted to the server device comprises information indicating menu navigation of a menu element of the remote GUI.

10. A method of providing access to a graphical user interface (GUI) of a server device having a guest operating system executing therein, the method comprising:

generating user interface (UI) metadata specifying a GUI element in the GUI of the server device using an application programming interface of the guest operating system configured to programmatically manipulate the GUI of the server device;

transmitting abuse image of the GUI and the generated UI metadata to a touch input client device connected to the server device;

receiving, from the touch input client device, information indicating user input linked to the GUI element in the GUI of the server device; and

invoking the application programming interface to programmatically manipulate the GUI of the server device according to the received information.

11. The method of claim 10, wherein the step of invoking the application programming interface comprises generating an event indicating a change in property value in the GUI element specified by the metadata.

12. The method of claim 10, further comprising:

upon determining a change in state of a second GUI element in the GUI of the server device, generating updated UI metadata specifying the change in state of the second GUI element using the application programming interface of the guest operating system configured to programmatically manipulate the GUI of the server device.

13. The method of claim 10, further comprising:

upon determining a change in application state of an application executing on the guest operating system, generating updated UI metadata specifying the change in application state using the application programming interface of the guest operating system configured to programmatically manipulate the GUI of the server device.

14. The method of claim 10, wherein the UI metadata comprises semantic information related to contents of the GUI elements and organized in a tree-like structure.

15. The method of claim 10, wherein the application programming interface comprises an accessibility assistive application programming interface of the guest operating system configured to expose functionality of GUI to application-generated events.

16. The method of claim 10, wherein the GUI element on the GUI of the server device comprises a menu bar of an application window;

and wherein the received information indicating user input linked to the GUI element in the GUI of the server device comprises information indicating menu navigation of the menu bar of the GUI of the server device.

17. A method of generating a local graphical user interface (GUI) on a client device that is connected to a server device having a remote GUI, the method comprising:

receiving, at the client device, a voice input through an input device of the client device;

processing the voice input to determine a command input to manipulate the remote GUI of the server device; and

transmitting, to the server device, information indicating that the command input configured to manipulate the remote GUI.

18. The method of claim 17, wherein the step of processing the voice input further comprises:

generating the command input based on a match of a plurality of rules associating a plurality of pre-determined commands with a corresponding input event to be executed at the server device.

19. The method of claim 17, further comprising:

receiving, from the server device, a base image of the remote GUI and UI metadata describing a GUI element in the remote GUI;

generating the local GUI to be displayed on the touch screen of the client device, the local GUI including the base image received from the server device and a native GUI element generated according to the received UI metadata, the native GUI element corresponding to the GUI element in the remote GUI;

wherein the processing the voice input further comprises processing the voice input to determine a command input to manipulate the remote GUI the native GUI element, wherein the processing is performed based on information from the received UI metadata; and

transmitting, to the server device, information indicating that the corresponding GUI element in the remote GUI has been manipulated at the client device.

20. The method of claim 19, wherein the step of processing the voice input further comprises:

determining the command input from a plurality of candidate texts based on the command input matching information from the UI metadata that specifies the GUI element of the remote GUI.