Display management for native user experiences

• US 9,400,585 B2
• Filed: 10/05/2011
• Issued: 07/26/2016
• Est. Priority Date: 10/05/2010
• Status: Active Grant

• Current Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Original Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Inventors: Momchilov, Georgy, Valicherla, Chakravarthi, Ingale, Mukund, Valicherla, Kireeti
• Primary Examiner(s): Hong, Stephen
• Assistant Examiner(s): ROBINSON, MARSHON L
• Application Number:
  US13/253,414
• Publication Number:
  US 20120084663A1
• Time in Patent Office: 1,756 Days
• Field of Search: 715/744, 715/762, 725/37
• US Class Current: 1/1
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/1423 controlling a plurality of local displays, e.g. CRT and flat panel display
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G09G 2340/0464 Positioning
• CPC Class Codes: G09G 2340/14 Solving problems related to the presentation of information to be displayed
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 5/14 Display of multiple viewports
• CPC Class Codes: H04L 67/025 for remote control or remote monitoring of applications

Abstract

In some remote application execution environments, a client device (e.g., to which application output is sent and where user interaction is performed) may include native control elements for touch input such as virtual keyboards, virtual pickers and the like. When the server device detects an editable/control field of a remoted application receiving focus, the server may instruct the client to activate a native control element for user interaction. The client may determine the manner in which to modify an application display to resolve potential display conflicts. For example, a native control element may obscure or hide portions of the remoted application on the client display. Modifying the application display may include panning the display, zooming into or out of portions of the display, adapting the display for scrolling and the like. The remote computing device may also transmit values for populating a native control element.

Claims

1. A method comprising:

generating, by a client device, an application display for an application executing at a remote computing device;

receiving, by the client device, an instruction from the remote computing device requesting activation of a native graphical user input element native to the client device;

receiving, by the client device, from the remote computing device, one or more control values for an application control element on the remote computing device; and

in response to receiving the instruction from the remote computing device;

determining at least one characteristic of the native graphical user input element;

activating, by the client device, separately from the application display for the application executing at the remote computing device, the native graphical user input element native to the client device;

populating, by the client device, the native graphical user input element with the one or more control values received from the remote computing device;

generating, by the client device, a display simultaneously comprising a display of the native graphical user input element native to the client device, the application display for the application executing at the remote computing device, and the application control element on the remote computing device, the display of the native graphical user input element being different from the application display; and

modifying the application display for the application executing at the remote computing device based on the at least one characteristic of the native graphical user input element native to the client device.

2. The method of claim 1, wherein the instruction from the remote computing device specifies a type of control element to be activated on the client device and wherein the method further comprises determining the native graphical user input element native to the client device to activate based on the type of control element.

3. The method of claim 1, further comprising:

determining whether the display of the native graphical user input element will overlap one or more portions of the application display when the native graphical user input element is activated.

4. The method of claim 3, wherein determining whether the display of the native graphical user input element will overlap the one or more portions of the application display is performed based on at least one of:

a size and a position of the native graphical user input element.

5. The method of claim 1, further comprising:

detecting user interaction with the native graphical user input element, wherein the detected user interaction is made separately from the application; and

transmitting a notification message to the remote computing device, the notification message reporting the user interaction.

6. The method of claim 5, wherein the native graphical user input element comprises a menu with one or more user-selectable items corresponding to the one or more control values received from the remote computing device, wherein the user interaction comprises a selection of one of the one or more user-selectable items and wherein the notification message comprises an identification of the selected one of the one or more user-selectable items.

7. The method of claim 1, further comprising:

receiving an instruction to deactivate the native graphical user input element;

deactivating the native graphical user input element; and

returning the application display to a state of the application display existing before the modifying.

8. The method of claim 1, wherein the modifying the application display based on the at least one characteristic of the native graphical user input element comprises scrolling the application display.

9. The method of claim 1, wherein the modifying the application display based on the at least one characteristic of the native graphical user input element comprises panning the application display in a specified direction.

10. The method of claim 1, wherein the modifying the application display based on the at least one characteristic of the native graphical user input element comprises zooming the application display in or zooming the application display out.

24. The method of claim 3, wherein modifying the application display comprises adjusting the application display so that at least a portion of the application display is not overlapped by the display of the native graphical user input element.

25. The method of claim 24, wherein the portion of the application display that is not overlapped by the display of the native graphical user input element comprises the application control element.

26. The method of claim 1, wherein the application control element on the remote computing device is a menu, and the one or more control values comprise values of the menu.

27. The method of claim 26, wherein the native graphical user input element native to the client device comprises a virtual picker, and the virtual picker comprises the one or more control values of the menu of the remote computing device received from the remote computing device.

28. The method of claim 27, wherein generating, by the client device, the display simultaneously comprising the display of the native graphical user input element native to the client device, the application display, and the application control element on the remote computing device comprises generating, by the client device, a display simultaneously comprising the menu of the remote computing device and the virtual picker of the client device.

11. A method comprising:

receiving, by a first computing device, a first notification indicating an application control element of a display of a remoted application receiving focus, wherein a display of the remoted application is transmitted to a second computing device remote from the first computing device;

determining a type of the application control element of the remoted application receiving focus; and

transmitting an instruction to the second computing device, the instruction comprising a request for the second computing device to activate, separately from the remoted application, a graphical user input element native to the second computing device and corresponding to the type of the application control element, wherein the graphical user input element native to the second computing device is different from the display of the remoted application; and

transmitting, to the second computing device, one or more control values for the application control element for populating the graphical user input element native to the second computing device,

wherein the instruction to the second computing device further comprises a request for the second computing device to generate a display simultaneously comprising the graphical user input element native to the second computing device, the display of the remoted application, and the application control element of the remoted application receiving focus.

12. The method of claim 11, further comprising:

receiving a second notification of focus leaving the application control element; and

in response to receiving the second notification, transmitting a different instruction to the second computing device to further modify the display of the remoted application.

13. The method of claim 12, wherein the remoted application is displayed in a first state before receiving the first notification and is returned to the first state after receiving the second notification.

14. The method of claim 12, wherein the different instruction comprises a request to deactivate the graphical user input element native to the second computing device.

15. The method of claim 11, wherein transmitting the instruction to the second computing device comprises transmitting at least one of:

a size and a position of the application control element of the remoted application receiving focus.

16. The method of claim 11, further comprising:

receiving a second notification message comprising information regarding a user interaction with the graphical user input element native to the second computing device.

17. The method of claim 16, wherein the user interaction with the graphical user input element native to the second computing device comprises receiving user input via the graphical user input element native to the second computing device.

18. The method of claim 16, wherein the user interaction with the graphical user input element native to the second computing device comprises receiving a command to close the graphical user input element native to the second computing device.

19. One or more non-transitory computer-readable media storing computer-readable instructions that, when executed, cause a system to:

receive a first notification indicating at least an application control element of a remoted application receiving focus, wherein a display of the remoted application is transmitted to a computing device remote from the system;

determine a type of the application control element of the remoted application receiving focus; and

transmit an instruction to the computing device, the instruction comprising a request for the computing device to activate, separately from the remoted application, a graphical user input element native to the computing device and corresponding to the type of the application control element, wherein the graphical user input element native to the computing device is different from the display of the remoted application; and

transmit, to the computing device, one or more control values for the application control element for populating the graphical user input element native to the computing device,

wherein the instruction to the computing device further comprises a request for the computing device to generate a display simultaneously comprising the graphical user input element native to the computing device, the display of the remoted application, and the application control element of the remoted application receiving focus.

20. The one or more non-transitory computer-readable media of claim 19, wherein the computer-readable instructions, when executed, further cause the system to:

receive a second notification of focus leaving the application control element; and

in response to receiving the second notification, transmit another instruction to the computing device to further modify the display of the remoted application.

21. The one or more non-transitory computer-readable media of claim 19, wherein the remoted application is displayed in a first state before receiving the first notification and is returned to the first state after receiving a second notification of focus leaving the application control element.

22. The one or more non-transitory computer-readable media of claim 19, wherein the graphical user element native to the computing device does not obscure the display of the remoted application.

23. One or more non-transitory computer-readable media storing computer-readable instructions that, when executed by a processor, cause a system to:

generate a display for an application executing at a remote computing device, the display comprising an application control element of the application;

receive an instruction from the remote computing device requesting activation of a graphical user input element native to the system, the graphical user input element corresponding to the application control element of the application;

receive one or more control values for the application control element of the application;

determine at least one characteristic of the graphical user input element native to the system;

activate, separately from the application, the graphical user input element native to the system;

populate the graphical user input element native to the system with the one or more control values received from the remote computing device;

generate a display simultaneously comprising the graphical user input element native to the system and the application control element of the application; and

modify the display for the application based on the at least one characteristic of the graphical user input element native to the system.

29. The one or more non-transitory computer-readable media of claim 23, wherein the computer-readable instructions, when executed by the processor, cause the system to:

determine whether the application control element of the application is obscured by the graphical user input element native to the system; and

modify the display for the application such that at least the application control element of the application is not obscured by the graphical user input element.

30. The one or more non-transitory computer-readable media of claim 23, wherein the computer-readable instructions, when executed by the processor, cause the system to:

transmit, to the remote computing device, a confirmation of the graphical user input element native to the system.