System and method for extracting structured data from classified websites

• US 8,682,881 B1
• Filed: 09/07/2011
• Issued: 03/25/2014
• Est. Priority Date: 09/07/2011
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Xu, Cheng, Feng, Gang, Li, Xin
• Primary Examiner(s): NGUYEN, KIM T
• Application Number:
  US13/227,379
• Time in Patent Office: 930 Days
• Field of Search: 707/706, 707/749
• US Class Current: 707/706
• CPC Class Codes: G06F 16/9535 Search customisation based on user profiles and personalisation
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking

Abstract

Systems, methods, and computer readable storage mediums are provided for automatically extracting data from a classified website. A website is determined to be a classified website based on a set of heuristics. Then page models for other classified websites are accessed. The page models may include listing page models, detail page models, and/or city page models. A listing page in the classified website is determined based on similarity of the listing page to the page models for the other classified websites. Then a listing page model for the listing page in the classified website is created. After the model has been created data from the classified website is extracted based at least in part on the listing page model. Similar processes are performed for determining a details page, creating a details page model, and extracting data from the classified website using a details page model.

Claims

1. A computer implemented method of automatically extracting data from a classified website comprising:

on a server system having one or more processors and memory storing one or more programs for execution by the one or more processors;

determining that a website is an area specific classified website based at least in part upon determining that the website is geographically localized;

accessing page models for other classified websites;

identifying a listing page in the classified website based on similarity of the listing page to the page models;

creating a listing page model for the listing page comprising;

identifying one or more dynamic regions within the listing page;

determining a type of information associated with a respective dynamic region of the one or more identified dynamic regions;

creating a listing page template that identifies the one or more dynamic regions and their associated type of information; and

storing the listing page template;

extracting data from the classified website based at least in part on the listing page model; and

saving the extracted data in a database responsive to a classified site query by a user.

2. The method of claim 1, further comprising:

identifying a details page associated with the listing page in the classified website;

creating a details page model for the details page; and

extracting data from the classified website based at least in part on the details page model.

3. The method of claim 2, wherein creating the details page model comprises:

identifying one or more dynamic regions within the details page;

determining a type of information associated with a respective dynamic region of the one or more identified dynamic regions;

creating a details page template that identifies the one or more dynamic regions and their associated type of information; and

storing the details page template.

4. The method of claim 3, wherein extracting data from the classified website comprises:

accessing the details page template;

extracting content from a dynamic region identified in the details page template; and

storing in the database the extracted content as respective values for the types of information associated with the one or more dynamic regions identified in the details page template, wherein the database includes a plurality of attribute-value pairs, the values being derived from the extracted content, and the attributes being derived from the type of information associated with respective ones of the values.

5. The method of claim 1, further comprising:

identifying one or more additional pages in the classified website based in part on user access data;

creating at least one additional model based on a respective page of the one or more additional pages; and

extracting data from the classified website based at least in part on the additional model.

6. The method of claim 5, wherein the access data comprises Internet activity data for a group of users.

7. The method of claim 1, wherein the page models for other classified websites include one or more of:

listing page models, detail page models, and city page models.

8. The method of claim 1, wherein extracting data from the classified website comprises:

accessing the listing page template;

extracting content from a dynamic region identified in the listing page template; and

storing in the database the extracted content as respective values for the types of information associated with the one or more dynamic regions identified in the listing page template, wherein the database includes a plurality of attribute-value pairs, the values being derived from the extracted content, and the attributes being derived from the type of information associated with respective ones of the values.

9. The method of claim 1, wherein the website is determined to be a classified website based on a set of heuristics including at least determining that the website contains common classified site terms.

10. The method of claim 1, wherein the listing page is identified based at least in part on having a set of rows exhibiting a repeating pattern.

11. A server system, for automatically extracting data from a classified website, comprising:

one or more processors; and

memory storing one or more programs to be executed by the one or more processors;

the one or more programs comprising instructions for;

determining that a website is an area specific classified website based at least in part upon determining that the website is geographically localized;

accessing page models for other classified websites;

identifying a listing page in the classified website based on similarity of the listing page to the page models;

creating a listing page model for the listing page comprising;

identifying one or more dynamic regions within the listing page;

determining a type of information associated with a respective dynamic region of the one or more identified dynamic regions;

creating a listing page template that identifies the one or more dynamic regions and their associated type of information; and

storing the listing page template;

extracting data from the classified website based at least in part on the listing page model; and

saving the extracted data in a database responsive to a classified site query by a user.

12. The system of claim 11, further comprising instructions for:

identifying a details page associated with the listing page in the classified website;

creating a details page model for the details page; and

extracting data from the classified website based at least in part on the details page model.

13. The system of claim 12, wherein instructions for creating the details page model comprises instructions for:

identifying one or more dynamic regions within the details page;

determining a type of information associated with a respective dynamic region of the one or more identified dynamic regions;

creating a details page template that identifies the one or more dynamic regions and their associated type of information; and

storing the details page template.

14. The system of claim 13, wherein instructions for extracting data from the classified website comprises instructions for:

accessing the details page template;

extracting content from a dynamic region identified in the details page template; and

storing in the database the extracted content as respective values for the types of information associated with the one or more dynamic regions identified in the details page template, wherein the database includes a plurality of attribute-value pairs, the values being derived from the extracted content, and the attributes being derived from the type of information associated with respective ones of the values.

15. The system of claim 11, further comprising instructions for:

identifying one or more additional pages in the classified website based in part on user access data;

creating at least one additional model based on a respective page of the one or more additional pages; and

extracting data from the classified website based at least in part on the additional model.

16. The system of claim 15, wherein the access data comprises Internet activity data for a group of users.

17. The system of claim 11, wherein instructions for extracting data from the classified website comprises instructions for:

accessing the listing page template;

extracting content from a dynamic region identified in the listing page template; and

storing in the database the extracted content as respective values for the types of information associated with the one or more dynamic regions identified in the listing page template, wherein the database includes a plurality of attribute-value pairs, the values being derived from the extracted content, and the attributes being derived from the type of information associated with respective ones of the values.

18. A non-transitory computer readable storage medium storing one or more programs configured for execution by a computer, the one or more programs comprising instructions for:

determining that a website is an area specific classified website based at least in part upon determining that the website is geographically localized;

accessing page models for other classified websites;

identifying a listing page in the classified website based on similarity of the listing page to the page models;

creating a listing page model for the listing page comprising;

identifying one or more dynamic regions within the listing page;

determining a type of information associated with a respective dynamic region of the one or more identified dynamic regions;

creating a listing page template that identifies the one or more dynamic regions and their associated type of information; and

storing the listing page template;

extracting data from the classified website based at least in part on the listing page model; and

saving the extracted data in a database responsive to a classified site query by a user.

19. The non-transitory computer readable storage medium of claim 18, further comprising instructions for:

identifying a details page associated with the listing page in the classified website;

creating a details page model for the details page; and

extracting data from the classified website based at least in part on the details page model.

20. The non-transitory computer readable storage medium of claim 19, wherein instructions for creating the details page model comprises instructions for:

identifying one or more dynamic regions within the details page;

determining a type of information associated with a respective dynamic region of the one or more identified dynamic regions;

creating a details page template that identifies the one or more dynamic regions and their associated type of information; and

storing the details page template.

21. The non-transitory computer readable storage medium of claim 20, wherein instructions for extracting data from the classified website comprises instructions for:

accessing the details page template;

extracting content from a dynamic region identified in the details page template; and

storing in the database the extracted content as respective values for the types of information associated with the one or more dynamic regions identified in the details page template, wherein the database includes a plurality of attribute-value pairs, the values being derived from the extracted content, and the attributes being derived from the type of information associated with respective ones of the values.

22. The non-transitory computer readable storage medium of claim 18, further comprising instructions for:

identifying one or more additional pages in the classified website based in part on user access data;

creating at least one additional model based on a respective page of the one or more additional pages; and

extracting data from the classified website based at least in part on the additional model.

23. The non-transitory computer readable storage medium of claim 22, wherein the access data comprises Internet activity data for a group of users.

24. The non-transitory computer readable storage medium of claim 18, wherein instructions for extracting data from the classified website comprises instructions for:

accessing the listing page template;

extracting content from a dynamic region identified in the listing page template; and

storing in the database the extracted content as respective values for the types of information associated with the one or more dynamic regions identified in the listing page template, wherein the database includes a plurality of attribute-value pairs, the values being derived from the extracted content, and the attributes being derived from the type of information associated with respective ones of the values.