Systems and methods for graphical user interface interaction with cloud-based applications

• US 9,965,151 B2
• Filed: 04/25/2013
• Issued: 05/08/2018
• Est. Priority Date: 09/09/2011
• Status: Active Grant

• Current Assignee: Apptou Technologies Limited, CloudOn Ltd. (Dropbox, Inc.)
• Original Assignee: CloudOn Ltd. (Dropbox, Inc.)
• Inventors: Zaveri, Jay, Yaacoby, Doron, Woodman, Alyssa, Domurat, Jerome
• Primary Examiner(s): Gaffin, Jeffrey A
• Assistant Examiner(s): Brahmachari, Mandrita
• Application Number:
  US13/870,936
• Publication Number:
  US 20130246932A1
• Time in Patent Office: 1,839 Days
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/04815 Interaction with a metaphor-based environment or interaction object displayed as three-dimensional, e.g. changing the user viewpoint with respect to the environment or object
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04883 for inputting data by handwriting, e.g. gesture or text
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131

Abstract

Systems and methods may: execute a virtualized application instance in a virtual computing environment; receive from a remote client device first object data relating to a first set of user interactions received with respect to a client-side graphical use interface (GUI) object at the remote client device, the client-side GUI object representing a GUI object of the virtualized application instance; translating the first object data to second object data relating to the GUI object of the virtualized application instance, the second object data executing a command when applied to the GUI object of the virtualized application instance; and synchronizing the GUI object of the virtualized application instance with the client-side GUI object by applying the second object data to the GUI object of the virtualized application instance.

Claims

1. A system comprising:

one or more processors; and

a memory accessible to the one or more processors, the memory storing instructions executable by the one or more processors to;

execute a virtualized application instance in a virtual computing environment, the virtualized application instance comprising a server-side graphical user interface (GUI) object;

receive from a remote client device first object data relating to a first set of user interactions received at a client-side GUI object at the remote client device, the client-side GUI object corresponding to the server-side GUI object of the virtualized application instance, and the first set of user interactions comprising a gesture input compatible with the client-side GUI object, wherein the gesture input is not natively compatible with the virtualized application instance;

convert the gesture input to a set of input events that are compatible with the virtualized application instance;

provide second object data based on the first object data and the set of input events, the second object data relating to the server-side GUI object, and the second object data executing a command when applied to the server-side GUI object of the virtualized application instance; and

synchronize the server-side GUI object with the client-side GUI object by applying the second object data to the server-side GUI object of the virtualized application instance.

2. The system of claim 1, the second object data comprising a second set of user interactions intended for the server-side GUI object of the virtualized application instance.

3. The system of claim 2, the application of the second object data to the server-side GUI object of the virtualized application instance comprising performing the second set of user interactions with respect to the client-side GUI object.

4. The system of claim 1, wherein the memory further stores instructions executable by the one or more processors to:

capture context data for the server-side GUI object of the virtualized application instance, and to transmit the context data to the remote client device as third object data.

5. The system of claim 4, the context data comprising information regarding a current graphical representation of the server-side GUI object of the virtualized application instance.

6. The system of claim 1, the client-side GUI object or the server-side GUI object of the virtualized application instance being a text box, a dialog box, a button, a drop down box, a toolbar, a toolbar item, a menu, a menu item, a radio option, or a check box.

7. A system comprising:

one or more processors; and

a memory accessible to the one or more processors, the memory storing instructions executable by the one or more processors to;

control a virtualized application instance operating in a virtual computing environment on a remote server;

receive from a remote client device a set of user interactions with respect to a client-side graphical user interface (GUI) object, the set of user interactions comprising a gesture input compatible with the client-side GUI object, and the client-side GUI object corresponding to a server-side GUI object of the virtualized application instance, wherein the gesture input is not natively compatible with the virtualized application instance;

convert the gesture input to a set of input events that are compatible with the virtualized application instance;

provide first object data relating to the set of user interactions received with respect to the client-side GUI object;

synchronize the client-side GUI object with the server-side GUI object of the virtualized application instance based on second object data relating to the server-side GUI object of the virtualized application instance, and the second object data being provided based on the first object data and the set of input events;

transmit to the remote server the first object data; and

receive from the remote server the second object data.

8. The system of claim 7, the second object data comprising context data relating to the server-side GUI object of the virtualized application instance.

9. The system of claim 7, the synchronization of the client-side GUI object with the server-side GUI object of the virtualized application instance comprising:

updating a current graphical representation of the client-side GUI object according to a current graphical representation of the server-side GUI object of the virtualized application instance based on the second object data.

10. The system of claim 7, the client-side GUI object or the server-side GUI object of the virtualized application instance being a text box, a dialog box, a button, a drop down box, a toolbar, a toolbar item, a menu, a menu item, a radio option, or a check box.

11. A method comprising:

executing a virtualized application instance based on a version of the application in a virtual computing environment;

receiving from a remote client device first object data relating to a first set of user interactions received at the remote client device with respect to a client-side graphical use interface (GUI) object at the remote client device, the first set of user interactions comprising a gesture input compatible with the client-side GUI object, and the client-side GUI object representing a server-side GUI object of the virtualized application instance, wherein the gesture input is not natively compatible with the virtualized application instance;

converting the gesture input to a set of input events that are compatible with the virtualized application instance;

translating the first object data to second object data relating to the server-side GUI object of the virtualized application instance, the second object data executing a command when applied to the server-side GUI object of the virtualized application instance based on the set of input events; and

synchronizing the server-side GUI object of the virtualized application instance with the client-side GUI object by applying the second object data to the server-side GUI object of the virtualized application instance.

12. The method of claim 11, the second object data comprising a second set of user interactions with respect to the server-side GUI object.

13. The method of claim 12, the application of the second object data to the server-side GUI object of the virtualized application instance comprising:

performing the second set of user interactions with respect to the set of server-side GUI elements.

14. The method of claim 11, further comprising:

capturing context data for the server-side GUI object of the virtualized application instance; and

transmitting the context data to the remote client device as third object data.

15. The method of claim 14, the context data comprising information regarding a current graphical representation of the server-side GUI object of the virtualized application instance.

16. The method of claim 11, the client-side GUI object being a text box, a dialog box, a button, a drop down box, a toolbar, a toolbar item, a menu, a menu item, a radio option, or a check box.

17. A system comprising:

one or more processors; and

a memory accessible to the one or more processors, the memory storing instructions executable by the one or more processors to;

execute a virtualized application instance based on a version of the application in a virtual computing environment;

receive from the remote client device first object data relating to a first set of user interactions received at the remote client device with respect to a client-side graphical use interface (GUI) object at the remote client device, the first set of user interactions comprising a gesture input compatible with the client-side GUI object, and the client-side GUI object representing a server-side GUI object of the virtualized application instance, wherein the gesture input is not natively compatible with the virtualized application instance;

convert the gesture input to a set of input events that are compatible with the virtualized application instance;

provide second object data based on the first object data and the set of input events, the second object data relating to the server-side GUI object of the virtualized application instance, and the second object data executing a command when applied to the server-side GUI object of the virtualized application instance; and

synchronize the server-side GUI object of the virtualized application instance with the client-side GUI object by applying the second object data to the server-side GUI object of the virtualized application instance.

18. A method comprising:

presenting a virtualized application instance operating in a virtual computing environment on a remote server;

receiving from a remote client device a set of user interactions with respect to a client-side graphical user interface (GUI) object, the set of user interactions comprising a gesture input compatible with the client-side GUI object, and the client-side GUI object representing a server-side GUI object of the virtualized application instance, wherein the gesture input is not natively compatible with the virtualized application instance;

converting the gesture input to a set of input events that are compatible with the virtualized application instance;

providing first object data relating to the set of user interactions received with respect to the client-side GUI object;

transmitting to the remote server the first object data;

receiving from the remote server the second object data; and

synchronizing the client-side GUI object with the server-side GUI object of the virtualized application instance based on second object data and the set of input events relating to the server-side GUI object of the virtualized application instance.

19. The method of claim 18, the second object data comprising context data relating to the server-side GUI object of the virtualized application instance.

20. The method of claim 18, the synchronization of the client-side GUI object with the server-side GUI object of the virtualized application instance comprising updating a current graphical representation of the client-side GUI object according to a current graphical representation of the server-side GUI object of the virtualized application instance based on the second object data.

21. The method of claim 18, the client-side GUI object being a text box, a dialog box, a button, a drop down box, a toolbar, a toolbar item, a menu, a menu item, a radio option, or a check box.

22. A system comprising:

one or more processors; and

a memory accessible to the one or more processors, the memory storing instructions executable by the one or more processors to;

present a virtualized application instance operating in a virtual computing environment on a remote server;

receive from a remote client device a set of user interactions with respect to a client-side graphical user interface (GUI) object, the set of user interaction comprising a gesture input compatible with the client-side GUI object, and the client-side GUI object representing a server-side GUI object of the virtualized application instance, wherein the gesture input is not natively compatible with the virtualized application instance;

convert the gesture input to a set of input events that are compatible with the virtualized application instance;

provide first object data relating to the set of user interactions received with respect to the client-side GUI object;

transmit to the remote server the first object data;

receive from the remote server the second object data; and

synchronize the client-side GUI object with the server-side GUI object of the virtualized application instance based on second object data and the set of input events relating to the server-side GUI object of the virtualized application instance.