Sharing screen content in a mobile environment

• US 10,491,660 B1
• Filed: 08/18/2017
• Issued: 11/26/2019
• Est. Priority Date: 07/11/2014
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David
• Primary Examiner(s): Kells, Asher D
• Application Number:
  US15/680,690
• Time in Patent Office: 830 Days
• CPC Class Codes: G06F 16/156 Query results presentation
• CPC Class Codes: G06F 16/2228 Indexing structures
• CPC Class Codes: G06F 16/24578 using ranking
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/5854 using shape and object relationship
• CPC Class Codes: G06F 16/5866 using information manually generated, e.g. tags, keywords, comments, manually generated location and time information
• CPC Class Codes: G06F 16/957 Browsing optimisation, e.g. caching or content distillation
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06F 9/451 Execution arrangements for user interfaces
• CPC Class Codes: G06Q 10/1097 Task assignment
• CPC Class Codes: G06V 30/416 Extracting the logical structure, e.g. chapters, sections or page numbers; Identifying elements of the document, e.g. authors
• CPC Class Codes: H04L 67/06 specially adapted for file transfer, e.g. file transfer protocol [FTP]
• CPC Class Codes: H04W 4/18 Information format or content conversion, e.g. adaptation by the network of the transmitted or received information for the purpose of wireless delivery to users or terminals

Abstract

Systems and methods are provided for sharing a screen from a mobile device. For example, a method includes receiving, at a second mobile device, an image of a screen captured from a first mobile device and determining whether to trigger an automated action. The method may also include displaying, responsive to not triggering the automated action, annotation data generated for the image with the image on a display of the second mobile device, the annotation data including at least one visual cue corresponding to content in the image relevant to a user of the second mobile device. The method may further include, responsive to triggering the automated action, determining that a mobile application associated with the image is installed on the second mobile device and replaying user input actions received with the image on the second mobile device starting from a reference screen associated with the mobile application.

Claims

1. A method performed by one or more processors, the method comprising:

receiving an identifier for a recipient and an image from a first mobile device, the image representing a captured screen of the first mobile device;

copying the image to an image repository associated with the recipient;

performing, using a recognition engine, recognition on the image to identify an entity referenced in the image;

generating annotation data for the image based on the recognition, the annotation data including a visual cue that corresponds to an area of actionable content for the entity identified based on the recognition,

wherein generating the visual cue that corresponds to the area of actionable content for the entity is based on the entity being determined, based on historical activity, to be relevant to the recipient indicated by the received identifier, and

wherein the visual cue is selectable to initiate an action associated with the entity; and

providing the image and the annotation data to a second mobile device, the second mobile device being associated with the recipient.

2. The method of claim 1, wherein the image represents less than the entire image as displayed on the first mobile device.

3. The method of claim 1, wherein the area of actionable content represents content that maps to a document that is associated with the entity and available via the Internet and wherein selection of the area causes the mobile device to navigate to the document.

4. The method of claim 1, wherein the visual cue corresponds with a structure element having the entity relevant to the recipient.

5. The method of claim 1, further comprising:

notifying the recipient after the annotation data has been generated; and

providing the image and the annotation data in response to a request for the image from the recipient.

6. The method of claim 1, further comprising:

receiving, at the second mobile device, the image of the screen captured from the display of the first mobile device, and the annotation data;

determining whether to trigger an automated action; and

displaying, responsive to not triggering the automated action, the annotation data generated for the image with the image on a display of the second mobile device, the annotation data including the visual cue.

7. The method of claim 6, wherein a set of user input actions is received with the image and the method further comprises, responsive to triggering the automated action:

determining that a mobile application associated with the image is installed on the second mobile device; and

replaying the user input actions on the second mobile device starting from a reference screen associated with the mobile application.

8. A system comprising:

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the at least one processor to;

receive an identifier for a recipient and an image from a first mobile device, the image representing a captured screen of the first mobile device;

copy the image to an image repository associated with the recipient;

perform, using a recognition engine, recognition on the image to identify an entity referenced in the image;

generate annotation data for the image based on the recognition, the annotation data including a visual cue that corresponds to an area of actionable content for the entity identified based on the recognition,

wherein generating the visual cue that corresponds to the area of actionable content for the entity is based on the entity being determined, based on historical activity, to be relevant to the recipient indicated by the received identifier, and

wherein the visual cue is selectable to initiate an action associated with the entity; and

provide the image and the annotation data to a second mobile device, the second mobile device being associated with the recipient.

9. The system of claim 8, wherein the image represents less than the entire image as displayed on the first mobile device.

10. The system of claim 8, wherein the area of actionable content represents content that maps to a document that is associated with the entity and available via the Internet, and wherein selection of the area causes the mobile device to navigate to the document.

11. The system of claim 8, wherein the visual cue corresponds with a structure element having the entity relevant to the recipient.

12. The system of claim 8, wherein the instructions, when executed by the at least one processor, further cause the at least one processor to:

notify the recipient after the annotation data has been generated; and

provide the image and the annotation data in response to a request for the image from the recipient.