Touch Support for Remoted Applications

• US 20120092277A1
• Filed: 10/05/2011
• Published: 04/19/2012
• Est. Priority Date: 10/05/2010
• Status: Active Grant

• Current Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Original Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Inventors: Momchilov, Georgy
• Granted Patent: US 9,110,581 B2
• Time in Patent Office: Days
• US Class Current: 345/173
• CPC Class Codes: G06F 3/038 Control and interface arrangements therefor, e.g. drivers or device-embedded control circuitry
• CPC Class Codes: G06F 3/041 Digitisers, e.g. for touch screens or touch pads, characterised by the transducing means
• CPC Class Codes: G06F 3/0488 using a touch-screen or digitiser, e.g. input of commands through traced gestures
• CPC Class Codes: G06F 3/04886 by partitioning the display area of the touch-screen or the surface of the digitising tablet into independently controllable areas, e.g. virtual keyboards or menus
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G09G 2370/02 Networking aspects
• CPC Class Codes: G09G 5/003 Details of a display terminal, the details relating to the control arrangement of the display terminal and to the interfaces thereto specific for a CRT G09G1/165; for a flat panel G09G3/2092
• CPC Class Codes: G09G 5/18 Timing circuits for raster scan displays specially adapted for television H04N ; synchronisation between the display unit and other display units, videodisc player G09G5/12

Abstract

Touch, multi-touch, gesture, flick and stylus pen input may be supported for remoted applications. For example, a touch capable client device may receive touch input for a remoted application executing on a server. In such an instance, the touch input may be transmitted to the server for processing. The server may subsequently modify the application display or the application functionality and provide an output to the client device. In some arrangements, the output may correspond to instructions for modifying a display of the application while in other examples, the output may correspond to an image of the changed application display. Additionally or alternatively, determining a functionality associated with touch input may be performed based on user definitions, user preferences, server definitions (e.g., operating system on the server), client definitions (e.g., operating system on the client) and the like and/or combinations thereof. Aspects may also include resolving latency and enhancing user experience using various features.

Claims

1. A method comprising:

receiving, by a first computing device, an indication of touch input from a second computing device different from the first computing device, wherein the touch input is directed to an application executing on the first computing device;

generating, by the first computing device, a touch input notification message in a first message format configured to trigger a first type of function call by the application for processing the touch input; and

forwarding, by the first computing device, the touch input notification message to the application.

2. The method of claim 1, further comprising:

receiving, by the first computing device, device capability information of the second device, wherein the device capability information includes input capabilities of the second device.

3. The method of claim 1, further comprising generating a touch input event at the first computing device using a driver configured to generate touch input events in an underlying operating system of the first computing device, wherein the driver executes as part of the operating system.

4. The method of claim 1, further comprising executing a function provided by an underlying operating system of the first computing device to generate a touch input event corresponding to the indication of touch input.

5. The method of claim 1, further comprising:

detecting initiation of the first type of function call by the application to process the touch input; and

in response to detecting initiation of the first type of function call, replacing the first type of function call with a second type of function call different from the first type of function call

6. The method of claim 5, further comprising:

processing the touch input using the second type of function call, wherein the second type of function call is configured to provide data for the touch input.

7. The method of claim 5, wherein the first type of function call is provided by an operating system.

8. The method of claim 5, wherein executing the second type of function call includes:

determining a type of application corresponding to the application; and

generating the one or more touch input commands based on the determined type of application.

9. The method of claim 5, wherein executing the second type of function call includes:

determining a type of operating system executing on at least one of;

the first computing device and second computing device; and

generating the one or more touch input commands based on the determined type of operating system executing on the at least one of;

the first computing device and second computing device.

10. The method of claim 5, wherein the touch input data includes coordinates and wherein the method further includes:

modifying the coordinates based on specified translation values.

11. The method of claim 5, wherein the first type of function call is configured to use kernel memory and the second type of function call is configured to use non-kernel memory.

12. The method of claim 1, further comprising:

retrieving raw input data corresponding to the touch input;

determining a type of touch input information to provide the application, wherein the type of touch input information includes one of;

the raw input data and one or more commands; and

generating the one or more commands based on the raw input data upon determining that the type of touch input information includes the one or more commands, otherwise,

providing the raw input data to the application.

13. The method of claim 1, further comprising:

retrieving data for the touch input;

processing the touch input data into one or more touch input commands; and

executing the one or more touch input commands.

14. The method of claim 13, wherein executing the one or more touch input commands results in a modification of an appearance of a display of the application and wherein the method further comprises:

transmitting the modification of the application display appearance to the second computing device.

15. The method of claim 1, wherein the touch input corresponds to a flick input.

16. The method of claim 1, wherein the touch input corresponds to stylus pen input.

17. The method of claim 1, wherein the touch input corresponds to gesture input.

18. A method comprising:

detecting, by a first computing device, a first touch input event directed to an application executing on a second computing device, wherein an output of the application is received at the first computing device;

transmitting information relating to the first touch input event to the second computing device;

receiving instructions for modifying a display of the application based on the first touch input event; and

modifying the application display in accordance with the received instructions.

19. The method of claim 18, wherein the first touch input event includes a multi-touch input event.

20. The method of claim 18, wherein transmitting the information relating to the first touch input event is performed by a remote application client executing on the first computing device, wherein the remote application client is configured to coordinate communications between the first computing device and the second computing device for interacting with the application.

21. The method of claim 18, wherein the information relating to the first touch input event includes raw input data detected by one or more touch-sensitive hardware elements of the first computing device.

22. The method of claim 18, further comprising:

detecting a second touch input event; and

determining whether to process the second touch input event locally at the first computing device based on at least one of;

an application to which the second touch input event is directed and a type of the touch input event.

23. The method of claim 18, wherein the touch input data includes coordinates and wherein the method further includes:

modifying the coordinates based on specified translation values

24. The method of claim 18, further comprising:

detecting a second touch input event; and

determining whether to process the second touch input event locally at the first computing device based on at least one of an amount of available network bandwidth between the first computing device and the second computing device and a processing speed of the application.

25. The method of claim 24, further comprising processing the second touch input event locally at the first computing device in response to at least one of:

the amount of available network bandwidth is below a specified bandwidth threshold; and

the processing speed of the application being below a speed threshold.

26. The method of claim 24, further comprising:

storing the second touch input event into a queue;

determining an age of the second touch input event; and

discarding the second touch input event upon the age of the second touch input reaching a specified age.

27. The method of claim 26, wherein discarding the second touch input event includes discarding a touch input event having an oldest creation date.

28. The method of claim 26, wherein discarding the second touch input event includes discarding all touch input events existing for a specified amount of time.

29. The method of claim 18, further comprising:

detecting a second touch input event directed to the application;

determining whether to process the second touch input event locally based on one or more rules; and

processing the second touch input event locally by the first computing device in response to determining that the second touch input event is to be processed locally, wherein processing the second touch input event locally includes modifying the application display locally.

30. The method of claim 29, wherein the one or more rules include one or more types of touch inputs that are to be processed locally.

31. A method comprising:

detecting, by a first computing device, a touch input event directed to an application executing on a second computing device, wherein an output of the application is received at the first computing device;

determining, by the first computing device, an amount of network latency between the first computing device and the second computing device;

determining whether the amount of network latency is above a specified threshold; and

in response to determining that the amount of network latency is above the specified threshold, processing the touch input event locally at the first computing device, including modifying a display of the application.

32. The method of claim 31, further comprising, when the touch input event is processed locally, transmitting information relating to the touch input event to the second computing device.

33. The method of claim 32, further comprising receiving an application display update from the second computing device in response to transmitting the information relating to the touch input event to the second computing device.

34. One or more non-transitory computer readable media storing computer readable instructions that, when executed, cause an apparatus to:

receive an indication of touch input from a computing device different from the apparatus, wherein the touch input is directed to an application executing on the apparatus;

generate a touch input notification message in a first message format configured to trigger a first type of function call by the application for processing the touch input; and

forward the touch input notification message to the application.

35. The one or more non-transitory computer readable media of claim 34, wherein the computer readable instructions, when executed, further cause the apparatus to:

receive device capability information of the computing device, wherein the device capability information includes input capabilities of the computing device.

36. The one or more non-transitory computer readable media of claim 34, wherein the computer readable instructions, when executed, further cause the apparatus to generate a touch input event at the apparatus using a driver configured to generate touch input events in an underlying operating system of the apparatus, wherein the driver executes as part of the operating system.

37. An apparatus comprising:

a processor; and

memory storing computer readable instructions that, when executed, cause the apparatus to;

receive an indication of touch input from a computing device different from the apparatus, wherein the touch input is directed to an application executing on the apparatus;

generate a touch input notification message in a first message format configured to trigger a first type of function call by the application for processing the touch input; and

forward the touch input notification message to the application.

38. The apparatus of claim 37, wherein the computer readable instructions, when executed, further cause the apparatus to:

retrieve raw input data corresponding to the touch input;

determine a type of touch input information to provide the application, wherein the type of touch input information includes one of;

the raw input data and one or more commands; and

generate the one or more commands based on the raw input data upon determining that the type of touch input information includes the one or more commands, otherwise,

provide the raw input data to the application.

39. The apparatus of claim 37, wherein the computer readable instructions, when executed, further cause the apparatus to:

receive data for the touch input;

process the touch input data into one or more touch input commands; and

execute the one or more touch input commands.