User interface virtualization of context menus

• US 9,542,080 B2
• Filed: 08/23/2013
• Issued: 01/10/2017
• Est. Priority Date: 04/25/2012
• Status: Active Grant

• Current Assignee: Vmware LLC (Broadcom, Inc.)
• Original Assignee: VMware, Inc. (Broadcom, Inc.)
• Inventors: Beveridge, Daniel James, Zhao, Ming
• Primary Examiner(s): PHANTANA ANGKOOL, DAVID
• Application Number:
  US13/975,093
• Publication Number:
  US 20140013234A1
• Time in Patent Office: 1,236 Days
• Field of Search: 715/740
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/14 Digital output to display device ; Cooperation and interconnection of the display device with other functional units
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/455 Emulation; Interpretation; Software simulation, e.g. virtualisation or emulation of application or operating system execution engines
• CPC Class Codes: G09G 2354/00 Aspects of interface with display user
• CPC Class Codes: G09G 2370/022 Centralised management of display operation, e.g. in a server instead of locally
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 2380/14 Electronic books and readers

Abstract

User interface virtualization describes a technique for providing a user with access to one computing device from another computing device, while translating the ergonomics of one computer'"'"'s user interface style into the ergonomics of the other'"'"'s. An agent running on a remote desktop may use an interface interaction API or software framework to programmatically open and manipulate a context menu on the remote desktop, and act as a proxy for a corresponding client running on a client device that accepts a “touch and swipe” style input. The client constructs and displays a native graphical user interface element at the client device that is a virtual representation of the context menu at the remote desktop.

Claims

1. A method of generating a local graphical user interface (GUI) on a touch screen of a client device that is connected to a server device having a remote desktop, the method comprising:

receiving, from the server device, user interface (UI) metadata that specifies content of a context menu in the remote desktop, and a base image comprising a graphical representation of the remote desktop;

generating, at the client device, a native context menu GUI element based on received UI metadata, wherein the native context menu GUI element comprises a virtual representation of the context menu of the remote desktop and operates in accordance with touch and gesture input functions native to the client device;

generating a local GUI to be displayed on the touch screen of the client device, the local GUI comprising the base image received from the server device and the native context menu GUI element; and

responsive to detecting that the native context menu GUI element has been manipulated through a touch or gesture input, transmitting, to the server device, an input event indicating a corresponding manipulation of the context menu in the remote desktop.

2. The method of claim 1, further comprising:

responsive to detecting a gesture input associated with context menus at a location within the local GUI,

transmitting, to the server device, an indication to open the context menu at a specified location in the remote desktop corresponding to the location in the local GUI.

3. The method of claim 1,

wherein the UI metadata comprises semantic information related to contents of the context menu in the remote desktop, and

wherein the UI metadata is organized in a tree-like data structure.

4. The method of claim 1, wherein the generated native context menu GUI element is different than the corresponding context menu in the remote desktop in at least one of size, shape, color, style, manner of interaction, animation, and interactive behavior.

5. The method of claim 1, wherein the local GUI comprises at least one of the native context menu GUI element arranged on top of the base image received from the server device, and the native context menu GUI element arranged in a side-panel alongside the base image received from the server device.

6. The method of claim 1, wherein the input event transmitted to the server device comprises event information configured to trigger an application programming interface of the server device to programmatically manipulate the context menu of the remote desktop.

7. The method of claim 1, wherein the context menu in the remote desktop is activated in response to a point-and-click type input and the native context menu GUI element is activated through the touch or gesture input.

8. The method of claim 7, wherein the context menu is a pop-up menu.

9. A method of providing access to a remote desktop of a server device having a guest operating system executing therein, the method comprising:

generating user interface (UI) metadata specifying content of a context menu in a graphical user interface (GUI) of a remote desktop using an application programming interface of the guest operating system configured to programmatically manipulate the GUI of the server device;

transmitting a base image of the GUI and the generated UI metadata to a touch input client device connected to the server device, the touch input client device having a native context menu GUI element that is a virtual representation of the context menu of the remote desktop and operates in accordance with touch and gesture input functions native to the client device;

receiving, from the touch input client device, an input event relating to a touch or gesture navigation of the native context menu and indicating a corresponding menu navigation of the context menu in the GUI of the remote desktop; and

programmatically manipulating the context menu in the GUI of the remote desktop according to the received input event using the application programming interface of the guest operating system.

10. The method of claim 9,

wherein the context menu of the remote desktop comprises a plurality of items, and

wherein programmatically manipulating the context menu further comprises:

invoking at least one of the plurality of items specified by the received input event using the application programming interface.

11. The method of claim 9, further comprising:

upon determining a change in state of the context menu in the GUI of the remote desktop, generating updated UI metadata specifying the change in state of the context menu using the application programming interface of the guest operating system configured to programmatically manipulate the GUI of the server device.

12. The method of claim 9, wherein the UI metadata comprises semantic information related to contents of the context menu and organized in a tree-like data structure.

13. The method of claim 9, wherein the application programming interface comprises an accessibility assistive application programming interface of the guest operating system configured to expose functionality of the GUI.

14. A non-transitory computer readable medium containing program instructions for generating a local graphical user interface (GUI) on a touch screen of a client device that is connected to a server device having a remote desktop, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to carry out the steps of:

receiving, from the server device, user interface (UI) metadata that specifies content of a context menu in the remote desktop, and a base image comprising a graphical representation of the remote desktop;

generating, at the client device, a native context menu GUI element based on received UI metadata, wherein the native context menu GUI element comprises a virtual representation of the context menu of the remote desktop that operates in accordance with touch and gesture input functions native to the client device;

generating a local GUI to be displayed on the touch screen of the client device, the local GUI comprising the base image received from the server device and the native context menu GUI element; and

responsive to detecting that the native context menu GUI element has been manipulated through a touch input,

transmitting, to the server device, an input event indicating a corresponding manipulation of the context menu in the remote desktop.

15. The non-transitory computer readable medium of claim 14, wherein the execution of the program instructions further causes the one or more processors to carry out the steps of:

responsive to detecting a gesture input associated with context menus at a location within the local GUI, transmitting, to the server device, an indication to open the context menu at a specified location in the remote desktop corresponding to the location in the local GUI.

16. The non-transitory computer readable medium of claim 14, wherein the UI metadata comprises semantic information related to contents of the context menu in the remote desktop, wherein the UI metadata is organized in a tree-like data structure.

17. A non-transitory computer readable medium containing program instructions for providing access to a remote desktop of a server device having a guest operating system executing therein, wherein execution of the program instructions by one or more processors of a computer system causes the one or more processors to carry out the steps of:

generating user interface (UI) metadata specifying content of a context menu in a graphical user interface (GUI) of a remote desktop using an application programming interface of the guest operating system configured to programmatically manipulate the GUI of the server device;

transmitting a base image of the GUI and the generated UI metadata to a touch input client device connected to the server device, the touch input client device having a native context menu GUI element that is a virtual representation of the context menu of the remote desktop and operates in accordance with touch and gesture input functions native to the client device;

receiving, from the touch input client device, an input event relating to a touch or gesture navigation of the native context menu and indicating a corresponding menu navigation of the context menu in the GUI of the remote desktop; and

programmatically manipulating the context menu in the GUI of the remote desktop according to the received input event using the application programming interface of the guest operating system.

18. The non-transitory computer readable medium of claim 17, wherein the context menu of the remote desktop comprises a plurality of items, and wherein programmatically manipulating the context menu further comprises:

invoking at least one of the plurality of items specified by the received input event using the application programming interface.

19. The non-transitory computer readable medium of claim 17, wherein the execution of the program instructions further causes the one or more processors to carry out the steps of:

upon determining a change in state of the context menu in the GUI of the remote desktop, generating updated UI metadata specifying the change in state of the context menu using the application programming interface of the guest operating system configured to programmatically manipulate the GUI of the server device.

20. The non-transitory computer readable medium of claim 17, wherein the UI metadata comprises semantic information related to contents of the context menu and organized in a tree-like data structure.