Uncoupled application extensions including interactive digital surface layer for collaborative remote application sharing and annotating

• US 9,596,320 B2
• Filed: 10/01/2012
• Issued: 03/14/2017
• Est. Priority Date: 09/30/2011
• Status: Active Grant

• Current Assignee: Calgary Scientific Inc.
• Original Assignee: Calgary Scientific Inc.
• Inventors: Thomas, Monroe Milas, McFadzean, David Bruce, Stephure, Matthew James
• Primary Examiner(s): Christensen, Scott B.
• Assistant Examiner(s): Concannon, Sean
• Application Number:
  US13/632,245
• Publication Number:
  US 20130086155A1
• Time in Patent Office: 1,625 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 9/44526 Plug-ins; Add-ons
• CPC Class Codes: G06F 9/452 Remote windowing, e.g. X-Window System, desktop virtualisation protocols for virtual reality H04L67/131
• CPC Class Codes: G06F 9/541 via adapters, e.g. between incompatible applications
• CPC Class Codes: H04L 67/01 Protocols
• CPC Class Codes: H04L 67/148 Migration or transfer of sessions
• CPC Class Codes: H04L 67/561 Adding application-functional data or data for application control, e.g. adding metadata
• CPC Class Codes: H04L 67/75 Indicating network or usage conditions on the user display

Abstract

Systems and method for providing for uncoupled application extensions to a remote access application. The application extensions are provided as part the remote access application to provide features and functionalities that otherwise are not provided by underlying the application programs that are made available by the remote access program. These features and functionalities may be provided without a need to modify the underlying application program, as they are integral with the remote access application.

Claims

1. A method of providing an uncoupled application extension to a remotely-accessed application, comprising:

providing a tiered remote access framework comprising an application tier, a server tier and a client tier, the tiered remote access framework communicating first information regarding the remotely-accessed application in a state model between client computing devices in the client tier that are accessing the remotely-accessed application, wherein the state model enables the client computing devices to natively display logical elements of the remotely-accessed application;

updating the state model by a server executing the remote-accessed application in accordance with the screen or application data of the remote-accessed application, and sending the updates of the state model to the client tier;

updating by a client access application program in the client tier the state model in accordance with input data received from a client tier user interface program and sending the updates of the state model to the server tier;

providing the uncoupled application extension at each client computing device, the uncoupled application extension enhances the remotely-accessed application by providing a functionality at the client computing device that is independent of the remotely-accessed application;

updating by a client access application in the client tier an extension state model to include second information regarding the uncoupled application extension such that the functionality associated with the uncoupled application extension is enabled in at least one of the client computing devices;

communicating the state model and the extension state model to synchronize a state of the application and the uncoupled application extension, respectively, during a session that includes the client computing devices;

receiving, by at least one of the client computing devices or a remote application server, the state model updates and the extension state model updates; and

composing an updated state model based on the state model updates and the extension state model updates, wherein the updated state model enables the client computing device to natively display modified logical elements of the remotely-accessed application.

2. The method of claim 1, wherein the state model is used to display the remotely-accessed application at the client computing devices.

3. The method of claim 1, further comprising providing the extension state model as a markup language document having a plurality of tags that are associated with the uncoupled application extension.

4. The method of claim 1, further comprising:

providing the uncoupled application extension at the application tier; and

communicating the extension state model from application tier to the client tier.

5. The method of claim 4, wherein the extension state model is modified at the application tier by the remotely-access application.

6. The method of claim 1, further comprising providing an interactive digital surface layer as the uncoupled application extension.

7. The method of claim 6, further comprising:

receiving inputs within the interactive digital surface layer; and

representing the inputs in the second information in the extension state model.

8. The method of claim 7, wherein the inputs are associated with users at each of the client computing devices.

9. The method of claim 7, further comprising representing the inputs as x-y coordinates within a display of the client computing devices.

10. The method claim 7, further comprising separating mouse cursor position data from mouse input events to provide an indication of a location to users at each of the client computing devices.

11. The method of claim 7, further comprising providing a user control to disable interaction between the inputs and the remotely-accessed application.

12. The method of claim 6, further comprising providing notifications in the interactive digital surface layer in accordance with information contained in the extension state model.

13. The method of claim 12, wherein the notifications comprise at least one of a network status indicator, a user presence indicator, and a user control indicator.

14. A method of providing an uncoupled application extension to a remotely-accessed application, comprising:

providing a tiered remote access framework comprising an application tier, a server tier and a client tier, the tiered remote access framework communicating first information regarding the remotely-accessed application in a state model between client computing devices in the client tier that are in a collaborative session and that are accessing the remotely-accessed application, wherein the state model enables the client computing devices to natively display logical elements of the remotely-accessed application;

updating the state model by a server executing the remote-accessed application in accordance with the screen or application data of the remote-accessed application, and sending the updates of the state model to the client tier;

updating by a client access application program in the client tier the state model in accordance with input data received from a client tier user interface program and sending the updates of the state model to the server tier;

providing the uncoupled application extension in at least one of the server tier, the client tier and the application tier, the uncoupled application extension enhances the remotely-accessed application by providing a functionality at the client computing devices independent of the remotely-accessed application;

communicating second information regarding the uncoupled application extension in an extension state model such that the functionality associated with the uncoupled application extension is enabled at the client computing devices;

communicating the state model and the extension state model to synchronize a state of the application and the uncoupled application extension, respectively, during the collaboration session that includes the client computing devices;

receiving, by at least one of the client computing devices or a remote application server, the state model updates and the extension state model updates; and

composing an updated state model based on the state model updates and the extension state model updates, wherein the updated state model enables the client computing device to natively display modified logical elements of the remotely-accessed application.

15. The method claim 14, further comprising:

providing an interactive digital surface layer as the uncoupled application extension;

receiving inputs within the interactive digital surface layer; and

representing the inputs in the second information in the extension state model.

16. The method of claim 15, wherein the inputs are associated with users at each of the client computing devices, and wherein the inputs are color-coded to each user in the collaborative session.

17. The method of claim 15, further comprising separating mouse cursor position data from mouse input events to provide an indication of a location to users in the collaborative session, wherein the indication is associated with a particular user in the collaborative session.

18. The method of claim 15, further comprising providing a control mechanism to specify which user in the collaborative session is in control of the interactive digital surface layer.

19. The method claim 14, further comprising:

providing an interactive digital surface layer as the uncoupled application extension;

receiving the second information in the extension state model at the client computing devices; and

providing a notification in the interactive digital surface layer in accordance with the second information.

20. The method of claim 19, wherein the notification comprises at least one of a network status indicator, a user presence indicator, and a user control indicator.

21. A non-transitory computer readable medium comprising computer-executable instructions such that when executed by a processor of a computing device performs a method of providing an uncoupled application extension to a remotely-accessed application, comprising:

providing a tiered remote access framework comprising an application tier, a server tier and a client tier, the tiered remote access framework communicating first information regarding the remotely-accessed application in a state model between client computing devices in the client tier that are accessing the remotely-accessed application and second information regarding the uncoupled application extension, wherein the state model enables the client computing devices to natively display logical elements of the remotely-accessed application;

updating by a client access application program in the client tier the state model in accordance with input data received from a client tier user interface program and sending the updates of the state model to the server tier;

providing the uncoupled application extension at each client computing device, the uncoupled application extension enhances the remotely-accessed application by providing a functionality at the client computing device that is independent of the remotely-accessed application;

updating by a client access application in the client tier an extension state model to include second information regarding the uncoupled application extension such that the functionality associated with the uncoupled application extension is enabled in at least one of the client computing devices;

communicating the state model and the extension state model to synchronize a state of the application and the uncoupled application extension, respectively, during a session that includes the client computing devices;

receiving, by at least one of the client computing devices or a remote application server, the state model updates and the extension state model updates; and

composing an updated state model based on the state model updates and the extension state model updates, wherein the updated state model enables the client computing device to natively display modified logical elements of the remotely-accessed application.

22. The non-transitory computer readable medium of claim 21, further comprising instructions to execute the uncoupled application extension without accessing the first information.

23. The non-transitory computer readable medium of claim 21, further comprising instructions for providing an interactive digital surface layer as the uncoupled application extension.

24. The non-transitory computer readable medium of claim 23, further comprising instructions for:

receiving inputs within the interactive digital surface layer; and

representing the inputs in the second information in the extension state model.

25. The non-transitory computer readable medium of claim 23, further comprising instructions for providing notifications in the interactive digital surface layer in accordance with information contained in the extension state model.