Action suggestions for user-selected content

• US 10,970,646 B2
• Filed: 10/01/2015
• Issued: 04/06/2021
• Est. Priority Date: 10/01/2015
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Sharifi, Matthew, Ramage, Daniel, Petrou, David
• Primary Examiner(s): Gonzales, Vincent
• Application Number:
  US14/872,582
• Publication Number:
  US 20170098159A1
• Time in Patent Office: 2,014 Days
• Field of Search: None
• CPC Class Codes: G06F 16/245 Query processing
• CPC Class Codes: G06F 16/3322 using system suggestions G06F16/3325 takes precedence
• CPC Class Codes: G06F 3/0481 based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
• CPC Class Codes: G06F 3/0482 Interaction with lists of selectable items, e.g. menus
• CPC Class Codes: G06F 3/0484 for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
• CPC Class Codes: G06F 3/04895 Guidance during keyboard input operation, e.g. prompting
• CPC Class Codes: G06F 9/453 Help systems
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

Systems and methods are provided for suggesting actions for selected text based on content displayed on a mobile device. An example method can include converting a selection made via a display device into a query, providing the query to an action suggestion model that is trained to predict an action given a query, each action being associated with a mobile application, receiving one or more predicted actions, and initiating display of the one or more predicted actions on the display device. Another example method can include identifying, from search records, queries where a website is highly ranked, the website being one of a plurality of websites in a mapping of websites to mobile applications. The method can also include generating positive training examples for an action suggestion model from the identified queries, and training the action suggestion model using the positive training examples.

Claims

1. A mobile device comprising:

a display device;

at least one processor; and

memory storing instructions that, when executed by the at least one processor, cause the mobile device to;

convert a selection made via the display device into a query,

provide the query to an action suggestion model, the action suggestion model being trained to predict one or more actions given the query, each of the one or more actions being associated with a corresponding mobile application,

receive predicted actions and probability scores that each correspond to a respective one of the predicted actions,

identify metadata about the mobile device, wherein the metadata about the mobile device indicates whether the mobile device is moving,

determine a certain threshold that is dependent on the metadata about the mobile device,

select a subset of the predicted actions based on the subset satisfying the certain threshold, wherein the selected subset includes a lower quantity of mobile applications when the mobile device is moving,

initiate display of the subset of the predicted actions on the display device,

receive a user selection of one of the predicted actions of the subset, and

initiate an intent using the query for the mobile application associated with the user selection.

2. The mobile device of claim 1, wherein initiating display of the subset of the predicted actions includes for each predicted action, displaying an icon for the mobile application associated with the predicted action.

3. The mobile device of claim 2, wherein the icons for the subset of the predicted actions overlay content displayed on the display device.

4. The mobile device of claim 1, wherein the selection is an image and converting the selection into a query includes:

identifying text related to the image; and

using at least some of the text as the query.

5. The mobile device of claim 1, wherein the selection is an image and converting the selection into a query includes:

identifying an entity in the image; and

using the entity as the query.

6. The mobile device of claim 1, wherein the memory further stores instructions that, when executed by the at least one processor, causes the mobile device to:

receive the action suggestion model from a server;

identify whitelisted websites in search records for a user of the mobile device;

generate training examples from queries that include the whitelisted websites as highly ranked, the training examples having a weight higher than a ranking of the whitelisted website in the search records; and

train the action suggestion model using the training examples, resulting in a personalized action suggestion model.

7. The mobile device of claim 1, wherein the selection is a voice command and converting the selection into a query includes:

performing word recognition on the voice command; and

using recognized words as the query.

8. The mobile device of claim 1, wherein converting the selection into a query includes:

determining extensions for the selection to generate at least one extended query;

providing the extended query to the action suggestion model to generate second predicted actions;

initiating display of the at least one extended query, wherein initiating display of the at least one extended query causes the at least one extended query to be displayed, along with the subset of the predicted actions, in response to the selection; and

responsive to receiving a selection of the extended query, initiating display of second predicted actions.

9. The mobile device of claim 1, wherein at least one of the mobile applications is not installed on the mobile device.

10. The method of claim 1, wherein the action suggestion model is trained using one or more positive training examples to predict each of the one or more actions, and

wherein each of the positive training examples used in training the action suggestion model includes a corresponding training query and a corresponding one of the one or more actions based on the corresponding one of the one or more actions being associated with a highly ranked search result for the corresponding training query.

11. A method comprising:

converting, responsive to receiving an action suggestion indication, a selection made via a display device of a mobile computing device into a query;

predicting mobile applications for the query using an action suggestion model, the action suggestion model being trained to predict mobile applications given a query;

determining probability scores for each of the mobile applications;

identifying metadata about the mobile computing device, wherein the metadata about the mobile computing device indicates whether the mobile computing device is moving;

determining a certain threshold that is dependent on the metadata about the mobile computing device;

selecting a subset of the mobile applications based on the subset satisfying the certain threshold, wherein the selected subset includes a lower quantity of mobile applications when the mobile computing device is moving;

initiating display of a selectable control for the subset of the mobile applications on the display device;

receiving a user selection of one of the predicted mobile applications of the subset; and

initiating an intent using the query for the mobile applications associated with the user selection.

12. The method of claim 11, wherein initiating display of the selectable control includes displaying an icon for the subset of the mobile applications.

13. The method of claim 11, wherein converting the selection into a query includes:

determining extensions for the selection to generate at least one extended query;

providing the extended query to the action suggestion model to generate an additional mobile application;

initiating display of the at least one extended query, wherein initiating display of the at least one extended query causes the at least one extended query to be displayed, along with the subset of the mobile applications, in response to the selection; and

responsive to receiving a selection of the extended query, initiating display of a selectable control for the additional mobile application.

14. The method of claim 11, wherein at least one of the mobile applications is not installed on the mobile computing device and the method further includes:

receiving a selection of the selectable control; and

initiating installation of the mobile application on the mobile computing device.