Actionable Search Results for Visual Queries

• US 20110131241A1
• Filed: 08/11/2010
• Published: 06/02/2011
• Est. Priority Date: 12/02/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Flamholz, Avi, Petrou, David, Power, Theodore, Casey, Matthew R.
• Granted Patent: US 8,977,639 B2
• Time in Patent Office: Days
• US Class Current: 707/770
• CPC Class Codes: G06F 16/95 Retrieval from the web

Abstract

A server system receives a visual query and identifies an entity in the visual query. The server system further identifies a client-side action corresponding to the identified entity and creates an actionable search result element configured to launch the client-side action. Examples of actionable search result elements are buttons to initiate a telephone call, to initiate email message, to map an address, to make a restaurant reservation, and to provide an option to purchase a product. The entity identified in the visual query may be indirectly associated with a client-side action whose contact address or appropriate link is found in a search result associated with the identified entity. The client system receives and displays the actionable search result element, and upon a user selection of the actionable search result element, launches the client-side action in an application distinct from the visual query client application.

Claims

1. A computer-implemented method of processing a visual query comprising:

at a server system having one or more processors and memory storing one or more programs for execution by the one or more processors;

receiving a visual query from a client system;

processing the visual query by sending the visual query to at least one search system implementing a visual query search process;

receiving a plurality of search results from the at least one search system;

identifying an entity in the visual query;

identifying one or more client-side actions corresponding to the identified entity;

creating an actionable search result element configured to launch a respective client-side action of the one or more client-side actions; and

sending the actionable search result element and at least one of the plurality of search results to the client system.

2. The method of claim 1, wherein the actionable search result element is distinct from the plurality of search results.

3. The method of claim 1, including creating and sending to the client system a plurality of actionable search result user interface elements, each configured to launch a unique client action.

4. A computer-implemented method of processing a visual query comprising:

at a server system having one or more processors and memory storing one or more programs for execution by the one or more processors;

receiving a visual query from a client system;

identifying an entity in the visual query;

identifying one or more client-side actions corresponding to the identified entity;

creating an actionable search result element configured to launch a respective client-side action of the identified one or more client-side actions; and

sending the actionable search result element to the client system.

5. The method of claim 4, further comprising:

identifying a plurality of distinct client-side actions corresponding to the identified entity;

creating two or more actionable search result elements, each configured to launch a respective client-side action of the identified plurality of client-side actions; and

sending the two or more actionable search result elements to the client system.

6. The method of claim 4, wherein identifying the entity comprises using a non-OCR image matching process to identify the entity in the visual query.

7. The method of claim 4, wherein a respective client-side action of the one or more client-side actions is selected from the group consisting of:

initiating a call to a telephone number, instant messaging, paging, faxing, emailing, a social network communication, and communicating by another communication mechanism.

8. The method of claim 4, wherein the identified entity in the visual query is one or more of:

a person, a name or other identifier of the person, a bar code, a logo, a business, an organization, a building, a group of buildings or physical structures, a postal address, a landmark, a geographical entity, a product, and service.

9. The method of claim 4, wherein the sending includes sending to the client system a representation of the visual query with the actionable search result element overlaying at least a portion of the representation of the visual query.

10. The method of claim 4, wherein the sending includes sending to the client system information for visually presenting the actionable search result element overlaying at least a portion of the visual query.

11. The method of claim 4, wherein the identified entity is a phone number and the actionable search result element is a button for initiating a telephone call to the phone number.

12. The method of claim 4, wherein the identified entity is an email address and the actionable search result element is a button for initiating composition of an email message to the email address.

13. The method of claim 4, wherein the identified entity is a postal address and the actionable search result element is a button for mapping the address, wherein the mapping includes at least one of:

providing a map identifying the location of the postal address, providing driving directions to the postal address, providing driving directions from the postal address, providing an aerial photograph including the postal address, and providing a street view image corresponding to the postal address.

14. The method of claim 4, wherein the actionable search result element is configured to add information to a contacts list, wherein the information is selected from a group consisting of one or more of:

a name, an email address, a phone number, a fax number, a postal address, an instant messaging address, a company name, an organization name, a URL, and a social networking contact.

15. The method of claim 4, wherein the identified entity is a product and the actionable search result element is configured to provide one or more of the following:

a product review, an option to initiate purchase of the product, and option to initiate a bid on the product, a list of similar products, and a list of related products.

16. The method of claim 4, wherein the identified entity is a person or an identifier associated with the person, the plurality of search results includes a communication address associated with the person, and the actionable search result element is configured to launch a communication using the communication address.

17. The method of claim 16, wherein the actionable search result includes an identifier associated with the person, and the identifier is one or more of the name of the person, a facial image of the person, an identification number associated with the person, a phone number associated with the person, a fax number associated with the person, a social networking identifier associated with the person, and an email address associated with the person.

18. The method of claim 4, further including sending an actionable element configured to share or upload at least a portion of the visual query.

19. A computer-implemented method of processing a visual query comprising:

at a client system having one or more processors, a display, and memory storing one or more programs for execution by the one or more processors;

receiving an image;

creating a visual query from the image;

sending the visual query to a visual query search system;

receiving from the visual query search system an actionable search result element configured to launch a client-side action, wherein the actionable search result element corresponds to an entity in the visual query;

displaying the actionable search result element on the display using a visual query client application;

receiving a user selection of the actionable search result element; and

in a client-side application distinct from the visual query client application, launching the client-side action corresponding to the selected actionable search result element.

20. The method of claim 19, wherein the client-side application distinct from the visual query client application is selected from the group consisting of:

an email application, a browser application, a phone application, an instant messaging application, a social networking application, and a mapping application.

21. A server system, for processing a visual query, comprising:

one or more central processing units for executing programs;

memory storing one or more programs be executed by the one or more central processing units;

the one or more programs comprising instructions for;

receiving a visual query from a client system;

identifying an entity in the visual query;

identifying one or more client-side actions corresponding to the identified entity;

creating an actionable search result element configured to launch a respective client-side action of the identified one or more client-side actions; and

sending the actionable search result element to the client system.

22. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

receiving a visual query from a client system;

identifying an entity in the visual query;

identifying one or more client-side actions corresponding to the identified entity;

creating an actionable search result element configured to launch a respective client-side action of the identified one or more client-side actions; and

sending the actionable search result element to the client system.