Concurrent display of application between devices

• US 10,452,769 B1
• Filed: 06/03/2014
• Issued: 10/22/2019
• Est. Priority Date: 08/31/2012
• Status: Active Grant

• Current Assignee: United Services Automobile Association
• Original Assignee: United Services Automobile Association
• Inventors: Benavidez, Andy, Thomas, Heather, Springer, Ryan, Warnick, Mark Paxman
• Primary Examiner(s): Bates, Kevin T
• Assistant Examiner(s): Patel, Ronak
• Application Number:
  US14/294,912
• Time in Patent Office: 1,967 Days
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06Q 10/103 Workflow collaboration or project management
• CPC Class Codes: H04L 41/22 comprising specially adapted graphical user interfaces [GUI]
• CPC Class Codes: H04L 43/10 Active monitoring, e.g. heartbeat, ping or trace-route
• CPC Class Codes: H04L 65/401 wherein the services involve a main real-time session and one or more additional parallel real-time or time sensitive sessions, e.g. white board sharing or spawning of a subconference
• CPC Class Codes: H04L 67/02 based on web technology, e.g. hypertext transfer protocol [HTTP]
• CPC Class Codes: H04L 67/1095 Replication or mirroring of data, e.g. scheduling or transport for data synchronisation between network nodes

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for concurrent display of applications between devices. One of the methods includes establishing full duplex communication session with a first application on a first computing device using a second application. The method includes receiving a user interface image from the first application on the first computing device. The method includes displaying the user interface image on a second user interface of the computer system. The method includes generating an overlay image including one or more annotations to the user interface image. The method also includes sending the overlay image to the first computing device.

Claims

1. A computer implemented method, comprising:

establishing, by a computer system, full duplex communication session between a first application on a client device and a second application on the computer system, with the first application being configured to render a user interface that includes a plurality of fields;

sending, by the computer system to the client device, a request for one or more first fields of the plurality of fields of the first application, with the one or more first fields determined, at least in part, based on a security permission of a user of the computer system, with the security permission specifying that the user of the computer system has access to view only the one or more first fields;

receiving, by the computer system from the client device, rendering data generated on the client device, with the rendering data being configured to render, on a display device of the computer system, an image of the user interface of the first application on the client device, wherein the image of the user interface displays (i) the one or more first fields specified in the request and one or more values of the respective one or more first fields, and (ii) a second field in the plurality of the fields, with a value of the second field being obfuscated in accordance with the security permission of the user;

displaying, based on the received rendering data, the image of the user interface on a user interface of the computer system;

generating, by the computer system, overlay image data that when rendered displays one or more annotations as an overlay to the user interface of the first application on the client device; and

sending the overlay image data to the client device to cause the client device to display the one or more annotations as the overlay to the user interface of the first application.

2. The method of claim 1, further comprising:

polling the client device to request an updated image of the user interface of the first application on the client device.

3. The method of claim 2, further comprising:

receiving, by the computer system from the client device, additional rendering data generated on the client device, with the additional rendering data being configured to tender on the display device of the computer system an updated image of the user interface of the first application on the client device in response to a change in the user interface.

4. The method of claim 1, wherein establishing the full duplex communication comprises establishing a websockets session.

5. The method of claim 1, further comprising:

receiving, by the computer system and from the client device, additional rendering data generated on the client device, with the rendering data being configured to render on the display device of the computer system an updated image of the user interface, wherein the additional rendering data is generated in response to a change in an orientation of the client device.

16. The method of claim 1, further comprising causing the one or more annotations to be displayed on the client device until a predetermined action is taken by a user of the client device.

17. The method of claim 1, wherein the image of the user interface is in a first orientation determined based on an orientation of the client device and one or more annotations are in a second orientation determined by the user of the computer system.

6. A non-transitory computer storage medium encoded with computer program instructions that when executed by a computer system cause the computer system to perform operations comprising:

establishing, by the computer system, full duplex communication session between a first application on a client device and a second application on the computer system, with the first application being configured to render a user interface that includes a plurality of fields;

sending, by the computer system to the client device, a request for one or more first fields of the plurality of fields of the first application, with the one or more first fields determined, at least in part, based on a security permission of a user of the computer system, with the security permission specifying that the user of the computer system has access to view only the one or more first fields;

receiving, by the computer system from the client device, rendering data generated on the client device, with the rendering data being configured to render on a display device of the computer system an image of the user interface of the first application on the client device, wherein the image of the user interface displays (i) the one or more first fields specified in the request and one or more values of the respective one or more first fields, and (ii) a second field in the plurality of the fields, with a value of the second field being obfuscated in accordance with the security permission of the user;

displaying, based on the received rendering data, the image of the user interface on a user interface of the computer system;

generating, by the computer system, overlay image data that when rendered displays one or more annotations as an overlay to the user interface of the first application on the client device; and

sending the overlay image data to the client device to cause the client device to display the one or more annotations as the overlay to the user interface of the first application.

7. The medium of claim 6, further comprising:

polling the client device to request an updated image of the user interface of the first application on the client device.

8. The medium of claim 7, further comprising:

receiving, by the computer system from the client device, additional rendering data generated on the client device, with the additional rendering data being configured to tender on the display device of the computer system an updated image of the user interface of the first application on the client device in response to a change in the user interface.

9. The medium of claim 6, wherein establishing the full duplex communication comprises establishing a websockets session.

10. The medium of claim 6, further comprising:

receiving, by the computer system and from the client device, additional rendering data generated on the client device, with the rendering data being configured to render on the display device of the computer system an updated image of the user interface, wherein the additional rendering data is generated in response to a change in an orientation of the client device.

11. A system comprising:

a computer system having one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to;

establish full duplex communication session between a first application on a client device and a second application on the computer system, with the first application being configured to render a user interface that includes a plurality of fields;

send, by the one or more computers to the client device, a request for one or more first fields of the plurality of the first application, with the one or more first fields determined, at least in part, based on a security permission of a user of the one or more computers, with the security permission specifying that the user of the computer system has access to view only the one or more first fields;

generate, by the computer system from the client device, rendering data generated on the client device, with the rendering data being configured to render on a display device of the one or more computers an image of the user interface of the first application on the client device, wherein the image of the user interface displays (i) the one or more first fields specified in the request and one or more values of the respective one or more first fields, and (ii) a second field in the plurality of the fields, with a value of the second field being obfuscated in accordance with the security permission of the user;

display based on the received rendering data, the image of the user interface image on a user interface of the computer system, the user interface image displayed in the second orientation;

generate overlay image data that when rendered displays one or more annotations as an overlay to the user interface of the first application on the client device; and

send the overlay image data to the client device to cause the client device to display the one or more annotations as the overlay to the user interface of the first application.

12. The system of claim 11, wherein the instructions are further operable, when executed by the one or more computers, to cause the one or more computers to:

poll the client device to request an updated image of the user interface of the first application on the client device.

13. The system of claim 12, wherein the instructions are further operable, when executed by the one or more computers, to cause the one or more computers to:

receive, from the client device, additional rendering data generated on the client device, with the additional rendering data being configured to tender on the display device of the computer system an updated image of the user interface of the first application on the client device in response to a change in the user interface.

14. The system of claim 11, wherein establishing the full duplex communication comprises establishing a websockets session.

15. The system of claim 11, further comprising:

receiving, by the computer system and from the client device, additional rendering data generated on the client device, with the rendering data being configured to render on the display device of the computer system an updated image of the user interface, wherein the additional rendering data is generated in response to a change in an orientation of the client device.