Actionable search results for visual queries

• US 8,977,639 B2
• Filed: 08/11/2010
• Issued: 03/10/2015
• Est. Priority Date: 12/02/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Petrou, David, Flamholz, Avi, Casey, Matthew R., Power, Theodore
• Primary Examiner(s): Yen, Syling
• Application Number:
  US12/854,793
• Publication Number:
  US 20110131241A1
• Time in Patent Office: 1,672 Days
• Field of Search: 707/706, 707/722, 707/736, 707/758, 707/759, 707/764, 707/765, 707/766, 707/769, 707/781, 707999002-999003
• US Class Current: 707/766
• CPC Class Codes: G06F 16/95 Retrieval from the web

Abstract

A server system receives a visual query and identifies an entity in the visual query. The server system further identifies a client-side action corresponding to the identified entity and creates an actionable search result element configured to launch the client-side action. Examples of actionable search result elements are buttons to initiate a telephone call, to initiate email message, to map an address, to make a restaurant reservation, and to provide an option to purchase a product. The entity identified in the visual query may be indirectly associated with a client-side action whose contact address or appropriate link is found in a search result associated with the identified entity. The client system receives and displays the actionable search result element, and upon a user selection of the actionable search result element, launches the client-side action in an application distinct from the visual query client application.

Claims

1. A computer-implemented method of processing a visual query comprising:

at a server system having one or more processors and memory storing one or more programs for execution by the one or more processors;

receiving a visual query from a client system, wherein the visual query comprises an image;

in response to receiving the visual query;

obtaining a plurality of search results to the visual query;

analyzing the image to identify an entity in the visual query;

identifying one or more client-side actions corresponding to the identified entity based on information in the plurality of search results;

creating one or more actionable search result elements configured to launch respective client-side actions, wherein the actionable search result element includes a user selectable element that identifies the particular client-side action with respect to the identified entity; and

sending (i) the one or more actionable search result elements and (ii) at least one search result in the plurality of search results to the client system configured to display a search results list including one or more search results and to separately display of the one or more actionable search result elements in a display area;

wherein the at least one search result is formatted for display in the search result portion of a display on the client system;

wherein the actionable search result element is formatted for display in the search result element portion of the display, and

the search result element portion is different than the search result portion of the display.

2. The method of claim 1, including creating and sending to the client system a plurality of actionable search result user interface elements, each configured to launch a unique client-side action.

3. The method of claim 1, further comprising:

identifying a plurality of distinct client-side actions corresponding to the identified entity;

creating two or more actionable search result elements, each configured to launch a respective client-side action of the identified plurality of client-side actions; and

sending the two or more actionable search result elements to the client system.

4. The method of claim 1, wherein identifying the entity comprises using a non-OCR image matching process to identify the entity in the visual query.

5. The method of claim 1, wherein the respective client-side action is selected from the group consisting of:

initiating a call to a telephone number, instant messaging, paging, faxing, emailing, a social network communication, and communicating by another communication mechanism.

6. The method of claim 1, wherein the identified entity in the visual query is one or more of:

a person, a name or other identifier of the person, a business, an organization, and a service.

7. The method of claim 1, wherein the sending includes sending to the client system a representation of the visual query with the actionable search result element overlaying at least a portion of the representation of the visual query.

8. The method of claim 1, wherein the sending includes sending to the client system information for visually presenting the actionable search result element overlaying at least a portion of the visual query.

9. The method of claim 1, wherein the identified entity is a phone number and the actionable search result element is a button for initiating a telephone call to the phone number.

10. The method of claim 1, wherein the identified entity is an email address and the actionable search result element is a button for initiating composition of an email message to the email address.

11. The method of claim 1, wherein the actionable search result element is configured to add information to a contacts list, wherein the information is selected from a group consisting of one or more of:

a name, an email address, a phone number, a fax number, a postal address, an instant messaging address, a company name, an organization name, a URL, and a social networking contact.

12. The method of claim 1, wherein the identified entity is a person or an identifier associated with the person, the plurality of search results includes a communication address associated with the person, and the actionable search result element is configured to launch a communication using the communication address.

13. The method of claim 12, wherein the actionable search result includes an identifier associated with the person, and the identifier is one or more of the name of the person, a facial image of the person, an identification number associated with the person, a phone number associated with the person, a fax number associated with the person, a social networking identifier associated with the person, and an email address associated with the person.

14. The method of claim 1, further including sending an actionable element configured to share or upload at least a portion of the visual query.

15. The method of claim 1, wherein the one or more actionable search result elements are selected based on capabilities of the client system.

16. The method of claim 1, wherein the one or more actionable search results are scored based on one or more factors.

17. A computer-implemented method of processing a visual query comprising:

at a client system having one or more processors, a display, and memory storing one or more programs for execution by the one or more processors;

receiving an image;

creating a visual query from the image;

sending the visual query to a visual query search system;

in response to sending the visual query;

receiving from the visual query search system (i) one or more actionable search result elements configured to launch a client-side action, wherein the actionable search result element corresponds to an entity in the visual query; and

(ii) one or more search results corresponding to the visual query;

displaying (i) the one or more actionable search result elements and (ii) a search results list including the one or more search results on the client system;

wherein the search results list including the one or more search results are displayed in a search result portion of a display on the client system;

wherein the one or more actionable search result elements are formatted for display in a distinct search result element portion of the display, and

wherein the search result element portion is different than the search result portion of the display.

18. A server system, for processing a visual query, comprising:

one or more central processing units for executing programs;

memory storing one or more programs be executed by the one or more central processing units;

the one or more programs comprising instructions for;

receiving a visual query from a client system, wherein the visual query comprises an image;

in response to receiving the visual query;

obtaining a plurality of search results to the visual query;

analyzing the image to identify an entity in the visual query;

identifying one or more client-side actions corresponding to the identified entity based on information in the plurality of search results;

creating one or more actionable search result elements configured to launch respective client-side actions, wherein the actionable search result element includes a user selectable element that identifies the particular client-side action with respect to the identified entity; and

sending (i) the one or more actionable search result elements and (ii) at least one search result in the plurality of search results to the client system configured to display a search results list including one or more search results and to separately display of the one or more actionable search result elements in a display area;

wherein the at least one search result is formatted for display in the search result portion of a display on the client system;

wherein the actionable search result element is formatted for display in the search result element portion of the display, and

the search result element portion is different than the search result portion of the display.

19. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

receiving a visual query from a client system, wherein the visual query comprises an image;

in response to receiving the visual query;

obtaining a plurality of search results to the visual query;

analyzing the image to identify an entity in the visual query;

identifying one or more client-side actions corresponding to the identified entity based on information in the plurality of search results;

creating one or more actionable search result elements configured to launch respective client-side actions, wherein the actionable search result element includes a user selectable element that identifies the particular client-side action with respect to the identified entity; and

sending (i) the one or more actionable search result elements and (ii) at least one search result in the plurality of search results to the client system configured to display a search results list including one or more search results and to separately display of the one or more actionable search result elements in a display area;

wherein the at least one search result is formatted for display in the search result portion of a display on the client system;

wherein the actionable search result element is formatted for display in the search result element portion of the display, and

the search result element portion is different than the search result portion of the display.