Simulated hyperlinks on a mobile device based on user intent and a centered selection of text

• US 10,055,390 B2
• Filed: 11/18/2015
• Issued: 08/21/2018
• Est. Priority Date: 11/18/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Petrou, David
• Primary Examiner(s): Paula, Cesar
• Assistant Examiner(s): KULIKOWSKI, KONRAD J
• Application Number:
  US14/945,348
• Publication Number:
  US 20170139879A1
• Time in Patent Office: 1,007 Days
• CPC Class Codes: G06F 3/00 Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/04842 Selection of displayed objects or displayed text elements G06F3/0482 takes precedence
• CPC Class Codes: G06F 40/134 Hyperlinking
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06F 9/4843 by program, e.g. task dispatcher, supervisor, operating system

Abstract

Systems and methods simulate a hyperlink in regular content displayed on a screen. An example method can include generating, responsive to detecting a simulated hyperlink indication, a centered selection from content displayed on a display of a computing device, providing the centered selection to a simulated hyperlink model that predicts an operation given the centered selection, and initiating the operation using an intent associated with a mobile application. The simulated hyperlink model may also provide, from the centered selection, an intelligent selection used the intent'"'"'s parameter. Another method includes identifying documents having a hyperlink whitelisted websites, generating positive training examples for a simulated hyperlink model using the documents, each positive training example having a centered selection, a website, and a mobile application mapped to the website, and training the simulated hyperlink model, using the positive training examples, to predict an operation for the mobile application given the centered selection.

Claims

1. A mobile device comprising:

a display device;

at least one processor;

a simulated hyperlink model, the simulated hyperlink model being a neural-network trained to predict at least one operation when provided a preselected quantity of words as input, the operation being associated with an intent for a mobile application; and

memory storing instructions that, when executed by the at least one processor, cause the mobile device to;

detect an action on content displayed on the display device, the action having a location,

generate a centered selection responsive to detecting the action, the centered selection including a word nearest the location for the action and an equal quantity of words from the content appearing prior to the word nearest the location and following the word nearest the location so that the centered selection has the preselected quantity of words,

provide the centered selection to the simulated hyperlink model,

obtain a predicted operation from the simulated hyperlink model, and

initiate at least one of;

a) display of a user interface permitting selection of the operation, which uses the intent, or

b) performance of the operation using the intent.

2. The mobile device of claim 1, wherein the simulated hyperlink model is further trained to provide a sub-selection from the centered selection and performance of the operation occurs using the sub-selection as a parameter for the intent.

3. The mobile device of claim 2, wherein initiating display of the user interface includes displaying a menu with an icon for the mobile application and with an option to copy the sub-selection.

4. The mobile device of claim 2, wherein the memory further stores instructions that, when executed by the at least one processor, cause the mobile device to:

modify the appearance of text displayed on the display that corresponds to the sub-selection.

5. The mobile device of claim 1, wherein the memory further stores instructions that, when executed by the at least one processor, cause the mobile device to:

obtain a sub-selection of the centered selection from the simulated hyperlink model; and

initiate, responsive to receiving a selection of the operation from the user interface, performance of the operation using the sub-selection.

6. The mobile device of claim 1, wherein initiating display of the user interface includes displaying an icon for the mobile application.

7. The mobile device of claim 1, wherein the action is a drag and release of an icon at the location and an equal quantity of words from the content appearing prior to the word nearest the location and following the word nearest the location.

8. The mobile device of claim 1, wherein the words prior to and following the word nearest the location exclude words in different paragraphs and generating the centered selection includes:

determining that the quantity of words prior to the word nearest the location is less than the equal quantity; and

adding padding tokens to the centered selection, the padding tokens being counted as words prior to the word nearest the location, so that the quantity of words prior to the word nearest the location is the equal quantity.

9. The mobile device of claim 1, wherein the memory further stores instructions that, when executed by the at least one processor, cause the mobile device to:

receive the simulated hyperlink model from a server;

generate training examples from hyperlinks in a screen capture index for a user of the mobile device; and

use the training examples to train the simulated hyperlink model, resulting in a personalized simulated hyperlink model.

10. A method performed on a mobile device, the method comprising:

generating, responsive to detecting a simulated hyperlink indication, a centered selection from content displayed on a display of a mobile computing device, the centered selection including a word nearest a location for the simulated hyperlink indication and an equal quantity of words from the content appearing prior to the word nearest the location and following the word nearest the location so that the centered selection has a preselected quantity of words;

providing the centered selection to a simulated hyperlink model, the simulated hyperlink model being a neural-network trained to predict an operation given an input with the preselected quantity of words, the operation being associated with an intent for a mobile application, the simulated hyperlink model being stored on the mobile device;

obtaining a predicted operation from the simulated hyperlink model for the centered selection; and

initiating at least one of;

a) display of a user interface permitting selection of the operation, which uses the intent, or

b) performance of the operation using the intent.

11. The method of claim 10, wherein generating the centered selection includes:

performing text and image recognition on a screen capture of the content to determine words represented in the content; and

determining a word closest to a location of the simulated hyperlink indication as a center word in the centered selection.

12. The method of claim 10, wherein generating the centered selection further includes:

using padding tokens in place of words in the centered selection to avoid including words from different sections.

13. The method of claim 10, wherein the simulated hyperlink model is also trained to make a sub-selection from the centered selection, the sub-selection representing a portion of the centered selection and using the sub-selection as a parameter for the intent.

14. The method of claim 10, wherein the operation is a first operation and the method further comprises:

obtaining a second operation and the first operation from the simulated hyperlink model;

determining a probability score for the first operation and a probability score for the second operation are similar;

providing the user interface, which enables the user of the mobile computing device to select the first operation or the second operation;

receiving a selection of the first operation;

generating a positive training example from the centered selection and the first operation; and

initiating the first operation using the intent.

15. The method of claim 10, further comprising:

obtaining, responsive to detecting the simulated hyperlink indication, a partial screen capture image,

wherein the centered selection is obtained from the screen capture image.