Web mining to build a landmark database and applications thereof

• US 9,323,792 B2
• Filed: 06/25/2014
• Issued: 04/26/2016
• Est. Priority Date: 11/12/2008
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Zhao, Ming, Zheng, Yantao, Song, Yang, Adam, Hartwig
• Primary Examiner(s): Shanmugasundaram, Kannan
• Application Number:
  US14/314,948
• Publication Number:
  US 20140310319A1
• Time in Patent Office: 671 Days
• Field of Search: 707/741, 707/915
• US Class Current: 1/1
• CPC Class Codes: G06F 16/21 Design, administration or maintenance of databases
• CPC Class Codes: G06F 16/22 Indexing; Data structures therefor; Storage structures
• CPC Class Codes: G06F 16/58 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
• CPC Class Codes: G06F 16/583 using metadata automatically derived from the content
• CPC Class Codes: G06F 16/95 Retrieval from the web

Abstract

This invention relates to building a landmark database from web data. In one embodiment, a computer-implemented method builds a landmark database. Web data including a web page is received from one or more websites via one or more networks. The web data is interpreted using at least one processor to determine landmark data describing a landmark. At least a portion of the landmark data identifies a landmark. Finally, a visual model is generated using the landmark data. A computing device is able to recognize the landmark in an image based on the visual model.

Claims

1. A computer-implemented method for building a landmark database, comprising:

receiving web data including a web page from one or more websites via one or more networks;

interpreting the web data using at least one processor to extract landmark data describing a landmark from the web data, the landmark data including a name that identifies the landmark and a portion of a street address where the landmark is located;

applying one or more error checking rules to remove inaccurate landmark data from the landmark data;

sending a search query to an image search service based on the landmark data;

receiving a set of image search results from the image search service;

filtering using one or more image checkers to remove non-photographic images from the set of image search results;

generating a visual model using the filtered image search results;

sending, to a geocoding service, a request with the name of the landmark and the portion of the street address where the landmark is located;

receiving, in response to the request, a location of the landmark from the geocoding service, the location determined based on the name of the landmark and the portion of the street address where the landmark is located; and

recognizing the landmark in an image different from the images in the filtered image search results by correlating the image with the visual model, and the location.

2. A computer-implemented method for building a landmark database, comprising:

receiving web data including a web page comprising a first image of a landmark and text from one or more websites via one or more networks;

interpreting the web data using at least one processor to extract landmark data describing the landmark in the first image from the text in the web data, wherein the landmark data comprises landmark name data that identifies the landmark;

applying one or more error checking rules to the landmark name data to remove inaccurate landmark name data, wherein the applying comprises checking name length of the landmark name data and capitalization of the landmark name data;

geocoding a location of the landmark in the first image by sending a request to a geocoding service, the request based on the landmark name data;

sending a search query to an image search service, the search query based on the landmark name data;

receiving a set of image search results from the image search service;

filtering using one or more image checkers to remove images of landmarks obstructed by one or more faces, non-photographic images, and images of maps from the set of image search results;

generating a visual model for the landmark using the filtered image search results; and

recognizing the landmark in a target image different from images in the filtered image search results by correlating the target image with the visual model and the location of the landmark.

3. The method of claim 2, wherein interpreting the web data comprises parsing the web data to determine the landmark data according to one or more extraction rules.

4. The method of claim 2, further comprising storing the visual model and a portion of the landmark data in a database of landmarks.

5. The method of claim 2, wherein the landmark data further comprises a portion of a street address where the landmark is located.

6. The method of claim 5, wherein geocoding the location of the landmark in the first image by sending the request to the geocoding service comprises sending the request comprising the landmark name data and the portion of the street address where in the landmark is located, the method further comprising:

receiving, from the geocoding service, in response to the request, a location of the landmark, the location determined based on the landmark name data and the portion of the street address where the landmark is located.

7. The method of claim 2, further comprising storing, in a database of landmarks, the visual model and a portion of the landmark data identifying the landmark.

8. The method of claim 2, wherein receiving the set of image search results from the image search service comprises receiving at least one image search result that comprises an image of the landmark.

9. The method of claim 2, wherein generating the visual model further comprises applying a detector to determine that a result in the set of image search results does not include an image of the landmark.

10. The method of claim 2 further comprising recognizing the landmark in an image from a frame of a video.

19. The method of claim 2, wherein interpreting the web data to extract landmark data comprises identifying one or more tags and one or more headers in the web data.

11. A system for building a landmark database, comprising:

one or more processors;

a web data retriever module implemented on at least one computing device configured to receive web data including a web page comprising a first image of a landmark from one or more websites via one or more networks;

a web data interpreter module implemented on the at least one computing device configured to interpret the web data to extract landmark data describing the landmark in the first image from text in the web data, wherein the landmark data comprises landmark name data that identifies a landmark;

an error checker module implemented on the at least one computing device configured to apply one or more error checking rules to the landmark name data to remove inaccurate landmark name data, wherein the error checking rules comprise checking name length of landmark name data and checking capitalization of landmark name data;

a geocoding module implemented on the at least one computing device configured to geocode a location of the landmark in the first image by sending a request to a geocoding service, the request based on the landmark name data;

a visual model generator implemented on the at least one computing device configured to send a search query to an image search service based on the landmark name data, receive a set of image search results from the image search service, filter the received image search results using one or more image checkers to remove images of landmarks obstructed by one or more faces, non-photographic images, and images of maps, and generate a visual model using the filtered image search results; and

a landmark recognition engine implemented on the at least one computing device configured to recognize the landmark in a target image different from the images in the filtered image search results by correlating the target image with the visual model and the location of the landmark.

12. The system of claim 11, wherein the web data interpreter module parses the web data to determine the landmark data according to one or more extraction rules.

13. The system of claim 11, further comprising a landmark database that stores the visual model and a portion of the landmark data identifying the landmark.

14. The system of claim 11, wherein the landmark data further comprises a portion of a street address where the landmark is located, the geocoding module further configured to send the request based on the landmark name data and the portion of a street address where the landmark is located to the geocoding service and to receive the location of the landmark from the geocoding service in response to the request.

15. The system of claim 14, further comprising a landmark database that stores the visual model and a portion of the landmark data.

16. The system of claim 11, wherein at least one result in the image search results comprises an image of the landmark.

17. The system of claim 11, wherein the visual model generator further comprises detector to determine that a result in the set of image search results does not include the landmark.

18. The system of claim 11, wherein the landmark recognition engine is further configured to recognize the landmark in an image from a frame of a video.

20. The system of claim 11, wherein the web data interpreter module is configured to interpret the web data by identifying one or more tags and one or more headers in the web data.