Actionable search results for street view visual queries

• US 9,405,772 B2
• Filed: 08/10/2010
• Issued: 08/02/2016
• Est. Priority Date: 12/02/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Petrou, David, Flynn, John, Power, Theodore
• Primary Examiner(s): Yen, Syling
• Application Number:
  US12/853,878
• Publication Number:
  US 20110131235A1
• Time in Patent Office: 2,184 Days
• Field of Search: 707/706, 707/722, 707/736, 707/758, 707/759, 707/764, 707/765, 707/766, 707/769, 707/781, 7079992-9993, 707/999.5
• US Class Current: 1/1
• CPC Class Codes: G06F 16/532 Query formulation, e.g. graphical querying
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content

Abstract

A server system receives a visual query and information about the current location of the querying device. The server system sends the query and location information to a visual query search system and receives one or more search results in accordance with the visual query and the location information. From the one or more search results, the server system identifies an entity in the visual query. The server system further identifies a client-side action corresponding to the identified entity and creates an actionable search result element configured to launch the client-side action. For example, if the identified entity were a restaurant the actionable search result elements could include buttons to initiate a telephone call, to read reviews, to map the address, or to make a reservation. If the actionable search result is selected, the client system launches the client-side action in an application distinct from the visual query client application.

Claims

1. A computer-implemented method of processing a visual query comprising:

at a server system having one or more processors and memory storing one or more programs for execution by the one or more processors;

receiving a visual query from a client system, the visual query comprising an image;

analyzing the image to identify an entity associated with the visual query;

receiving location information indicating a location of the client system, from the client system;

obtaining search results for the visual query, wherein obtaining search results includes;

determining whether the image of the visual query matches an image in a corpus of images;

in response to determining a matching image in the corpus of images, obtaining enhanced location information associated with the matching image, wherein the enhanced location information provides a greater specificity than the received location information from the client system; and

obtaining search results in accordance with the visual query and the enhanced location information, wherein the search results include one or more search results associated with one or more second entities related to the first entity according to particular criteria; and

sending at least (i) an image representing the first entity, (ii) information identifying the first entity, (iii) an image representing each of one or more second entities related to the first entity, and (iv) information identifying the one or more second entities, to the client system for concurrent display by the client system.

2. The method of claim 1, wherein the location information is obtained from one or more of the following:

GPS information, cell tower information, and local wireless network information.

3. The method of claim 1, further comprising:

sending, along with the image representing the first entity, one or more visual query search results associated with the first entity to the client system.

4. The method of claim 3, wherein the one or more visual query search results associated with the first entity include search results within a specified distance from a location identified by the location information.

5. The method of claim 3, wherein the one or more visual query search results associated with the first entity further include search results associated with the one or more second entities.

6. The method of claim 3, wherein the one or more visual query search results associated with the first entity include an actionable search results element configured to launch a client-side action corresponding to the first entity.

7. The method of claim 6, wherein the first entity is a restaurant and the client-side action is selected from the group consisting of:

initiating a phone call, providing a review;

initiating a reservation request, providing mapping information, launching the restaurant'"'"'s website, providing additional information, and sharing any of the above.

8. The method of claim 1, wherein the first entity is selected from the group consisting of:

a person, a name or other identifier associated with the person, a bar code, a logo, a business, an organization, a building, a group of buildings or physical structures, a postal address, a landmark, a geographical entity, a product, and a service.

9. The method of claim 6, wherein the first entity is a postal address, and the actionable search result element is a button for mapping the postal address, wherein the mapping includes at least one of:

providing a map identifying the location of the postal address, providing an aerial photograph including the postal address, and providing a street view image corresponding to the postal address.

10. The computer-implemented method of claim 1, wherein the information identifying the first entity is a name of the first entity.

11. The computer-implemented method of claim 1, wherein the information identifying the second entity is a name of the second entity.

12. The computer-implemented method of claim 1, wherein the information identifying the first entity is formatted for display on the client system next to the image representing the first entity.

13. The computer-implemented method of claim 1, wherein the information identifying the second entity is formatted for display on the client system next to the image representing the second entity.

14. The computer-implemented method of claim 1 wherein

the image representing the first entity is formatted for display in a place match portion of a display of the client system,

the image representing the second entity is formatted for display in a related place match portion of the display of the client system, and

the place match portion of the display is different than the related place match portion of the display.

15. A method of processing a visual query comprising:

at a client system having one or more processors, a display, and memory storing one or more programs for execution by the one or more processors;

obtaining or creating an image;

receiving or calculating location information indicating a location of the client system;

creating a visual query from the image;

sending the visual query and the location information to a visual query search system;

in response to sending the visual query and the location information;

receiving from the visual query search system (i) an image representing a first place entity obtained in accordance with the visual query and the location information, (ii) information identifying the first place entity, (iii) an image representing each of one or more second place entities geographically next to the first place entity, and (iv) information identifying the second place entities; and

concurrently displaying on the client system (i) the image representing the first place entity, (ii) the information identifying the first place entity, (iii) the image representing the second place entities, and (iv) the information identifying the second place entities.

16. The method of claim 15, further comprising:

in response to sending the visual query and the location information;

receiving from the visual query search system one or more visual query search results associated with the first place entity in accordance with both the visual query and the location information; and

displaying on the display, along with the image representing the first place entity, the one or more visual query search results associated with the first place entity.

17. A server system, for processing a visual query, comprising:

one or more central processing units for executing programs;

memory storing one or more programs be executed by the one or more central processing units;

the one or more programs comprising instructions for;

receiving a visual query from a client system, the visual query comprising an image;

analyzing the image to identify an entity associated with the visual query;

receiving location information indicating a location of the client system, from the client system;

obtaining search results for the visual query, wherein obtaining search results includes;

determining whether the image of the visual query matches an image in a corpus of images;

in response to determining a matching image in the corpus of images, obtaining enhanced location information associated with the matching image, wherein the enhanced location information provides a greater specificity than the received location information from the client system; and

obtaining search results in accordance with the visual query and the enhanced location information, wherein the search results include one or more search results associated with second entities related to the first entity according to particular criteria; and

sending at least (i) an image representing the first entity, (ii) information identifying the first entity, (iii) an image representing each of one or more second entities related to the first entity, and (iv) information identifying the one or more second entities, to the client system for concurrent display on a display associated with the client system.

18. A client system, for processing a visual query, comprising:

one or more central processing units for executing programs;

a display; and

memory storing one or more programs be executed by the one or more central processing units;

the one or more programs comprising instructions for;

obtaining or creating an image;

receiving or calculating location information indicating a location of the client system;

creating a visual query from the image;

sending the visual query and the location information to a visual query search system;

in response to sending the visual query and the location information;

receiving from the visual query search system (i) an image representing a first place entity obtained in accordance with the visual query and the location information, (ii) information identifying the first place entity, (iii) an image representing each of one or more second place entities geographically next to the first place entity, and (iv) information identifying the second place entities; and

concurrently displaying on the client system (i) the image representing the first place entity, (ii) the information identifying the first place entity, (iii) the image representing the each of one or more second place entities, and (iv) the information identifying the second place entities.

19. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

receiving a visual query from a client system, the visual query comprising an image;

analyzing the image to identify an entity associated with the visual query;

receiving location information indicating a location of the client system, from the client system;

obtaining search results for the visual query, wherein obtaining search results includes;

determining whether the image of the visual query matches an image in a corpus of images;

in response to determining a matching image in the corpus of images, obtaining enhanced location information associated with the matching image, wherein the enhanced location information provides a greater specificity than the received location information from the client system; and

obtaining search results in accordance with the visual query and the enhanced location information, wherein the search results include one or more search results associated with second entities related to the first entity according to particular criteria; and

sending at least (i) an image representing the first entity, (ii) information identifying the first entity, (iii) an image representing each of one or more second entities related to the first entity, and (iv) information identifying the one or more second entities, to the client system for concurrent display on a display associated with the client system.

20. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

obtaining or creating an image;

receiving or calculating location information indicating a location of a client system;

creating a visual query from the image;

sending the visual query and the location information to a visual query search system;

in response to sending the visual query and the location information;

receiving from the visual query search system (i) an image representing a first place entity obtained in accordance with the visual query and the location information, (ii) information identifying the first place entity, (iii) an image representing each of one or more second place entities geographically next to the first place entity, and (iv) information identifying the second place entities; and

concurrently displaying on the client system (i) the image representing the first place entity, (ii) the information identifying the first place entity, (iii) the image representing the second place entities, and (iv) the information identifying the second place entities.