Hybrid Use of Location Sensor Data and Visual Query to Return Local Listings for Visual Query

• US 20110137895A1
• Filed: 08/12/2010
• Published: 06/09/2011
• Est. Priority Date: 12/03/2009
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google LLC (Alphabet Inc.)
• Inventors: Petrou, David, Adam, Hartwig, Flynn, John, Neven, Hartmut
• Granted Patent: US 9,852,156 B2
• Time in Patent Office: Days
• US Class Current: 707/723
• CPC Class Codes: G06F 16/242 Query formulation
• CPC Class Codes: G06F 16/24575 using context
• CPC Class Codes: G06F 16/29 Geographical information databases
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: H04W 4/02 Services making use of location information
• CPC Class Codes: H04W 4/029 Location-based management or tracking services
• CPC Class Codes: H04W 64/00 Locating users or terminals or network equipment for network management purposes, e.g. mobility management

Abstract

A visual query is received from a client system, along with location information for the client system, and processed by a server system. The server system sends the visual query and the location information to a visual query search system, and receives from the visual query search system enhanced location information based on the visual query and the location information. The server system then sends a search query, including the enhanced location information, to a location-based search system. The search system receives and provides to the client one or more search results to the client system.

Claims

1. A computer-implemented method of processing a visual query comprising:

at a server system having one or more processors and memory storing one or more programs for execution by the one or more processors;

receiving a visual query from a client system;

receiving location information from the client system;

sending the visual query and the location information to a visual query search system;

receiving from the visual query search system enhanced location information for the client system based on the visual query and the location information;

sending a search query to a location-based search system, the search query including the enhanced location information;

receiving the one or more search results in accordance with the enhanced location information; and

sending at least one of the search results to the client system.

2. The method of claim 1, wherein the enhanced location information includes pose/azimuth information.

3. The method of claim 2, wherein the at least one search result includes a result in the direction of the pose.

4. The method of claim 2, wherein the device has an orientation and the visual query has an asymmetrical aspect ratio, the method further comprising:

constructing a viewing frustum based on the pose and based the orientation of the device determined based on one or more of;

the device sensors and the asymmetrical aspect ratio of the visual query, and

sending at least one search result within the viewing frustum to the client system.

5. The method of claim 4, including receiving a plurality of initial search results from location-based search system and filtering the initial search results to exclude search results outside the viewing frustum.

6. The method of claim 1, wherein the enhanced location information has greater associated accuracy than the location information received from the client system.

7. The method of claim 1, further comprising:

identifying an accuracy value for the enhanced location information;

favoring search results near the enhanced location when the enhanced location has an accuracy value at or above a threshold; and

favoring search results with a high prominence value when the enhanced location has an accuracy value below the threshold; and

sending at least one favored search result to the client system.

8. The method of claim 1, further comprising:

creating an interactive results document comprising a bounding box outlining a respective sub-portion of the visual query and including at least one user selectable link to at least one of the search results, wherein the bounding box is created by projecting earth coordinates of a search result onto screen coordinates of the visual query; and

sending the interactive results document to the client system.

9. The method of claim 2, wherein the enhanced location information comprises first enhanced location information, the method further comprising:

receiving a second visual query from the client system;

receiving second location information from the client system;

requesting, from a visual query search system, second enhanced location information for the client system based on the second visual query and the second location information;

when the request for second enhanced location information is successful, resulting in receipt of second enhanced location information having greater accuracy than the second location information received from the client system,

sending a second search query to a location-based search system, the second search query including the second enhanced location information, and

receiving one or more search results in accordance with the second search query;

when the request for second enhanced location information is not successful,

sending a third search query to the location-based search system, the third search query including the first enhanced location information; and

receiving one or more search results in accordance with the third search query; and

sending at least one of the search results to the client system.

10. The method of claim 1, further comprising:

receiving a plurality of search results from the location-based search system in response to the search query, each of the search results having an associated positional accuracy;

selecting one or more of the search results having highest associated positional accuracy, and returning the selected search results to the client system.

11. The method of claim 1, wherein each of the search results from the location-based search system comprises a respective local listing having an associated position and positional accuracy, the method further comprising:

selecting one or more first search results, each comprising a local listing having an associated position that A) satisfies a first positional closeness requirement with respect to the enhanced location information for the client system, and B) satisfies an accuracy requirement that the local listing'"'"'s associated position has positional accuracy that is equal to or greater than a threshold; and

sending the first results to the client system.

12. The method of claim 11, further comprising:

selecting one or more second search results in accordance with a requirement that each identified second search result satisfy a second positional closeness requirement with respect to at least one of the first search results; and

sending the first search results and second search results to the client system.

13. The method of claim 1, wherein each of the search results from the location-based search system comprises a respective local listing having an associated position and positional accuracy, the method further comprising:

selecting search results to send to the client system in accordance with the associated position and positional accuracy of each of the search results, the selecting including excluding from the selected search results those search results that A) have positional accuracy less than a threshold, and B) do not satisfy a positional closeness requirement with respect to at least one of the selected search results that has positional accuracy equal to or greater than the threshold and that satisfies a first positional closeness requirement with respect to the enhanced location information for the client system.

14. The method of claim 1, including sending to the client system a street view image determined by the visual query search system to match the visual query.

15. A computer-implemented method of processing a visual query comprising:

at a server system having one or more processors and memory storing one or more programs for execution by the one or more processors;

receiving a visual query from a client system;

receiving location information from the client system;

requesting, from a visual query search system, enhanced location information for the client system based on the visual query and the location information;

when the request for enhanced location information is successful, resulting in receipt of enhanced location information having greater accuracy than the location information received from the client system,

sending a first search query to a location-based search system, the first search query including the enhanced location information, and

receiving one or more search results in accordance with the first search query;

when the request for enhanced location information is not successful,

sending a second search query to the location-based search system, the second search query including the received location information from the client system; and

receiving one or more search results in accordance with the second search query; and

sending at least one of the search results to the client system.

16. A server system, for processing a visual query, comprising:

one or more central processing units for executing programs;

memory storing one or more programs be executed by the one or more central processing units;

the one or more programs comprising instructions for;

receiving visual query from a client system;

receiving location information from the client system;

sending the visual query and the location information to a visual query search system;

receiving from the visual query search system enhanced location information for the client system based on the visual query and the location information;

sending a search query to a location-based search system, the search query including the enhanced location information;

receiving the one or more search results in accordance with the enhanced location information; and

sending at least one of the search results to the client system.

17. A server system, for processing a visual query, comprising:

one or more central processing units for executing programs;

memory storing one or more programs be executed by the one or more central processing units;

the one or more programs comprising instructions for;

receiving visual query from a client system;

receiving location information from the client system;

requesting, from a visual query search system, enhanced location information for the client system based on the visual query and the location information;

when the request for enhanced location information is successful, resulting in receipt of enhanced location information having greater accuracy than the location information received from the client system,

sending a first search query to a location-based search system, the first search query including the enhanced location information, and

receiving one or more search results in accordance with the first search query;

when the request for enhanced location information is not successful,

sending a second search query to the location-based search system, the second search query including the received location information from the client system; and

receiving one or more search results in accordance with the second search query; and

sending at least one of the search results to the client system.

18. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

receiving a visual query from a client system;

receiving location information from the client system;

sending the visual query and the location information to a visual query search system;

receiving from the visual query search system enhanced location information for the client system based on the visual query and the location information;

sending a search query to a location-based search system, the search query including the enhanced location information;

receiving the one or more search results in accordance with the enhanced location information; and

sending at least one of the search results to the client system.

19. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

receiving a visual query from a client system;

receiving location information from the client system;

requesting, from a visual query search system, enhanced location information for the client system based on the visual query and the location information;

when the request for enhanced location information is successful, resulting in receipt of enhanced location information having greater accuracy than the location information received from the client system,

sending a first search query to a location-based search system, the first search query including the enhanced location information, and

receiving one or more search results in accordance with the first search query;

when the request for enhanced location information is not successful,

sending a second search query to the location-based search system, the second search query including the received location information from the client system; and

receiving one or more search results in accordance with the second search query; and

sending at least one of the search results to the client system.