Responsive Scroller Controls in Server-Hosted Applications

• US 20120266068A1
• Filed: 04/12/2012
• Published: 10/18/2012
• Est. Priority Date: 04/12/2011
• Status: Active Grant

• Current Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Original Assignee: Citrix Systems, Inc. (Cloud Software Group)
• Inventors: Ryman, Paul Adam, Hackett, Donovan Ross
• Granted Patent: US 9,558,013 B2
• Time in Patent Office: Days
• US Class Current: 715/719
• CPC Class Codes: G06F 1/163 Wearable computers, e.g. on a belt
• CPC Class Codes: G06F 2209/544 Remote
• CPC Class Codes: G06F 2209/545 Gui
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0485 Scrolling or panning
• CPC Class Codes: G06F 3/04855 Interaction with scrollbars
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/542 Event management; Broadcasting; Multicasting; Notifications
• CPC Class Codes: G11B 27/34 Indicating arrangements indicating means incorporated in magazine or cassette G11B23/046 and G11B23/0875; indicating measured values in general G01D
• CPC Class Codes: H04N 21/47 End-user applications

Abstract

Environments are disclosed in which a server executes an application on behalf of a client device and, in particular, to providing responsive scrolling in server-hosted applications. In one example, a remote computing device may be a mobile (or stationary) computing device with (or without) a touch screen input surface. Some or all of the client-server communication between a client-side scrollable UI control module of the remote computing device and a server-side scrollable UI control module of the server device may occur in an asynchronous manner.

Claims

1. An apparatus, comprising:

at least one processor configured to execute computer-executable instructions and a hosted application; and

at least one memory storing computer-executable instructions, which when executed by the processor, cause the apparatus to;

create, in the memory, a session of the hosted application corresponding to a remote computing device;

transmit, to the remote computing device, a graphical content of the hosted application;

determine that the graphical content includes a scrollable UI control;

retrieve metadata about the scrollable UI control;

transmit, to the remote computing device, metadata about the scrollable UI control;

transmit, to cache memory at the remote computing device, tile content associated with the scrollable UI control, wherein the tile content corresponds to scrollable content outside of a viewport of the scrollable UI control; and

receive, from the remote computing device, information about scrolling input action.

2. The apparatus of claim 1, wherein the graphical content of the hosted application includes a snapshot of a display of the hosted application.

3. The apparatus of claim 1, wherein the scrolling input actions comprise actions inputted on a touch screen input surface of the remote computing device, and wherein the memory further stores computer-executable instructions, which when executed by the processor, cause the apparatus to:

adjust position of the viewport with respect to the scrollable content in response to at least one of;

the received information about scrolling input actions.

5. The apparatus of claim 1, wherein the memory further stores computer-executable instructions, which when executed by the processor, cause the apparatus to:

identify the tile content to pre-fetch using a rate of deceleration and the information about scrolling input actions, which comprises at least direction of scrolling and velocity of scrolling.

6. The apparatus of claim 1, wherein the memory further stores computer-executable instructions, which when executed by the processor, cause the apparatus to:

adjust dimensions of the tile content based on at least one of;

measured network latency, network bandwidth, network queue lengths, and network bandwidth costs;

wherein adjusting the dimensions of tile content comprises changing length and width of the tile content.

7. The apparatus of claim 1, wherein the memory further stores computer-executable instructions, which when executed by the processor, cause the apparatus to:

adjust dimensions of the tile content based on at least one of;

download size of encoded data, memory constraints, image encoding method, and proximity to an edge of the scrollable content.

8. The apparatus of claim 1, wherein the metadata about the scrollable UI control comprises at least one of:

position and size;

extent of scrollable content;

viewport position onto this extent;

whether the control is visible;

relative z-order to other scrollable UI controls in the session of the hosted application;

background color; and

scrollbar visibility.

10. The apparatus of claim 1, wherein the hosted application is associated with a mobile application runtime component that comprises computer-executable instructions configured to discover, enumerate, and control a scrollable UI control contained in the hosted application;

and wherein the mobile application runtime component performs the determining that the graphical content includes a scrollable UI control.

13. The apparatus of claim 1, wherein the graphical content includes data associated with command remoting of a display of the hosted application.

4. (canceled)

9. (canceled)

11. (canceled)

12. (canceled)

14. (canceled)

15. (canceled)

16. (canceled)

17. (canceled)

18. (canceled)

19. (canceled)

20. (canceled)

21. A computer-assisted method, comprising:

creating, in computer memory by a computer processor, a session of a hosted application, wherein the session is associated with a remote computing device with a touch screen input surface configured to receive scrolling input actions;

determining, by a mobile application runtime component, that a graphical content of the hosted application includes a scrollable UI control;

retrieving, by the mobile application runtime component, metadata about the scrollable UI control, wherein the metadata comprises at least one of;

position and size;

extent of scrollable content;

viewport position onto this extent;

whether the control is visible;

relative z-order to other scroller controls in the same application and session;

background color; and

scrollbar visibility;

identifying, by the processor, based on direction of scrolling, velocity of scrolling, and measured latency of a network, scrollable content outside of a viewport of the scrollable UI control;

pre-fetching, by the processor, the identified scrollable content as tile content configured for storage in a memory at the remote computing device, wherein the tile content has dimensions based on predetermined factors;

transmitting, by the processor to the remote computing device, the graphical content of the hosted application;

transmitting, by the processor to the remote computing device, the metadata about the scrollable UI control; and

transmitting, by the processor to the remote computing device, the pre-fetched tile content.

22. The method of claim 21, comprising:

creating, in the computer memory by the processor, a remote presentation layer protocol stack corresponding to the remote computing device storing a client-side scrollable UI control module; and

receiving, from the client-side scrollable UI control module, information about scrolling input actions.

23. The method of claim 22, comprising:

adjusting position of the viewport with respect to the scrollable content in response to the received information about scrolling input actions from the client-side scrollable UI control module.

24. The method of claim 21, wherein the predetermined factors affecting the dimensions of the tile content include at least one of:

measured network latency, network bandwidth, network queue lengths, network bandwidth costs, download size of encoded data, memory constraints, image encoding method, and proximity to edge of the scrollable content.

25. (canceled)

26. (canceled)

27. (canceled)

28. (canceled)

29. (canceled)

30. (canceled)

31. (canceled)

32. (canceled)

33. (canceled)

34. (canceled)

35. (canceled)

36. (canceled)

37. (canceled)

38. (canceled)

39. (canceled)

40. (canceled)

41. (canceled)

42. (canceled)

43. (canceled)

44. (canceled)

45. (canceled)

46. (canceled)

47. (canceled)

48. (canceled)

49. A computer-assisted method, comprising:

receiving, from a remote server device, a graphical content of a hosted application executing on the remote server device;

receiving, from the remote server device by a client-side scrollable UI control module, metadata about a scrollable user interface (UI) control in the graphical content;

storing, in a local video buffer by a processor, the graphical content;

displaying, on a display device by the processor, the graphical content;

rendering, on the display device by the client-side scrollable UI control module, a viewport of the scrollable UI control based on the received metadata;

detecting, by the client-side scrollable UI control module, a scrolling input action from a touch screen input surface;

receiving, from the remote server device by the client-side scrollable UI control module, tile content corresponding to scrollable content outside of the viewport of the scrollable UI control; and

storing, in the cache memory by the client-side scrollable UI control module, the tile content;

50. The method of claim 49, comprising:

adjusting a position of the viewport with respect to the scrollable content based on the detected scrolling input action;

setting the viewport to opaque such that tile content from the cache memory covers the graphical content in the local video buffer underneath the viewport; and

displaying, in the viewport on the display device, tile content from the cache memory over the graphical content in the local video buffer underneath the viewport.

51. The method of claim 50, comprising:

generating information about the scrolling input action, wherein the information about the scrolling input action comprises at least direction of scrolling and velocity of scrolling; and

transmitting the information about the scrolling input action to the remote server device.

52. The method of claim 51, comprising:

receiving, from the remote server device, an updated graphical content of a hosted application executing on the remote server device, after the transmitting of the generated information about the scrolling input action;

storing, in the local video buffer by the processor, the updated graphical content;

setting, by the client-side scrollable UI control module, the viewport to transparent such that the updated graphical content is capable of being displayed on the display device underneath the viewport after the position of the viewport with respect to the scrollable content is stationary; and

displaying, on the display device, the updated graphical content stored in the local video buffer.

54. The method of claim 49, wherein the scrolling input action is a touch release action subsequent to a touch and move action.

55. The method of claim 49, wherein the received metadata about the scrollable UI control comprises at least one of:

position and size;

extent of scrollable content;

viewport position onto this extent;

whether the control is visible;

relative z-order to other scrollable UI controls in the session of the hosted application;

background color; and

scrollbar visibility.

56. The method of claim 49, wherein dimensions of the received tile content is based on at least one of:

measured network latency, network bandwidth, network queue lengths, and network bandwidth costs.

53. (canceled)