USING STRUCTURED DATABASE FOR WEBPAGE INFORMATION EXTRACTION

• US 20080281827A1
• Filed: 05/10/2007
• Published: 11/13/2008
• Est. Priority Date: 05/10/2007
• Status: Abandoned Application

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Microsoft Corporation
• Inventors: Acero, Alejandro, Wang, Ye-Yi, Rahurkar, Mandar A.
• Application Number:
  US11/746,790
• Publication Number:
  US 20080281827A1
• Time in Patent Office: Days
• US Class Current: 707/10
• CPC Class Codes: G06F 40/295 Named entity recognition
• CPC Class Codes: G06Q 10/00 Administration; Management

Abstract

A structured database is used for webpage information extraction, and in particular, to obtain training data from the webpage for training a statistical model. The structured database has a plurality of entries, wherein each entry comprises a plurality of fields. One of the fields comprises a URL (uniform resource locater), while another field comprises information at least similar to other information to be located in a webpage associated with the URL. For at least some of the entries in the structured database, a web page associated with the URL is retrieved. The webpage is analyzed and if information is found in the webpage similar to the information in the structured database, the webpage is identified as being suitable to be considered as a training sample.

Claims

1. A computer-implemented method of obtaining webpage training samples, the method comprising:

accessing a structured database having a plurality of entries, wherein each entry comprises a plurality of fields, one of the fields comprising a URL (uniform resource locater) and another one of the fields comprising first information at least similar to second information to be located in a webpage associated with the URL; and

for each of the plurality of entries in the structured database, retrieving a webpage associated with the URL; and

analyzing the webpage to find the second information therein corresponding to the first information in the structured database, and if the second information is found in the webpage storing information indicative of the webpage as a training sample.

2. The computer-implemented method of claim 1 wherein retrieving the webpage associated with the URL includes retrieving a root webpage associated with the URL.

3. The computer-implemented method of claim 2 wherein retrieving the webpage associated with the URL includes retrieving a plurality of webpages of varying hierarchy associated with the URL.

4. The computer-implemented method of claim 3 and further comprising generating a document object model (DOM) for each of the webpages.

5. The computer-implemented method of claim 4 wherein a score is calculated indicative of similarity of the first information with the second information.

6. The computer-implemented method of claim 5 wherein the score is based on an edit-distance between the first information and the second information.

7. The computer-implemented method of claim 6 wherein the score is based on a number of matches of tokens in the second information with that of tokens in the first information relative to a number of tokens in the first information, and the number of matches of tokens in the second information with that of tokens in the first information relative to a number of tokens in the second information.

8. The computer-implemented method of claim 5 and further comprising analyzing the webpages having a score above a selected threshold indicating close correspondence between the first information and the second information so as to obtain values of markup language related features pertaining to the second information.

9. The computer-implemented method of claim 8 wherein one of the markup language features comprises the last portion of the URL.

10. The computer-implemented method of claim 8 wherein the markup language features relates to at least one of size, font and color of the second information when rendered.

11. The computer-implemented method of claim 8 and further comprising analyzing surrounding text of the second information to obtain values of markup language related features pertaining to the second information.

12. A computer-implemented method of obtaining webpage training samples, the method comprising:

accessing a structured database having a plurality of entries, wherein each entry comprises a plurality of fields, one of the fields comprising a URL (uniform resource locater) and another one of the fields comprising first information at least similar to second information to be located in a webpage associated with the URL; and

for each of the plurality of entries in the structured database, retrieving a webpage associated with the URL; and

analyzing the webpage to obtain an indication of the similarity of the second information therein with the first information in the structured database, and if the indication indicates substantial correspondence analyzing the webpage so as to obtain values of markup language related features pertaining to the second information.

13. The computer-implemented method of claim 12 wherein one of the markup language features comprises the last portion of the URL.

14. The computer-implemented method of claim 12 wherein the markup language features relates to a size of the second information when rendered.

15. The computer-implemented method of claim 12 wherein the markup language features relates to a font of the second information when rendered.

16. The computer-implemented method of claim 12 wherein the markup language features relates to a color of the second information when rendered.

17. The computer-implemented method of claim 12 and further comprising analyzing surrounding text of the second information to obtain values of markup language related features pertaining to the second information.

18. A system for obtaining webpage training samples, the system comprising:

a structured database having a first plurality of entries and a second plurality of entries, wherein each entry of the first plurality of entries and the second plurality of entries comprises a plurality of fields, one of the fields comprising a URL (uniform resource locater) and another one of the fields in the first plurality of entries comprises first information at least similar to second information to be located in a webpage associated with the URL, and wherein said another one of the fields in the second plurality of entries lacks information;

a webpage processing module configured to operate with the structured database and access the Internet, the webpage processing module configured to retrieve a webpage associated with the URL for each entry of only the first plurality of entries in the database and not the second plurality of entries, configured to obtain a score for each webpage retrieved and rank the webpages based on the score.

19. The system of claim 18 wherein the score is based on an edit-distance between the first information and the second information.

20. The system of claim 19 wherein the score is based on a number of matches of tokens in the second information with that of tokens in the first information relative to a number of tokens in the first information, and the number of matches of tokens in the second information with that of tokens in the first information relative to a number of tokens in the second information.