Structured metadata extraction

• US 8,954,438 B1
• Filed: 05/31/2012
• Issued: 02/10/2015
• Est. Priority Date: 05/31/2012
• Status: Active Grant

• Current Assignee: Google LLC (Alphabet Inc.)
• Original Assignee: Google Inc. (Alphabet Inc.)
• Inventors: Tang, Alvin, Khandelwal, Nitin, Mao, Yiqiang
• Primary Examiner(s): Beausoliel, Jr., Robert
• Assistant Examiner(s): TRAN, BAO G
• Application Number:
  US13/484,803
• Time in Patent Office: 985 Days
• Field of Search: 707/737, 707/776
• US Class Current: 707/737
• CPC Class Codes: G06F 16/907 Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually

Abstract

Structured metadata extraction may include accessing one or more documents from which to extract the structured metadata from each of a plurality of hosts. A plurality of entity names can be extracted from the one or more documents from one of the plurality of hosts using an entity name pattern. A first element list can be extracted from the one or more documents based at least in part on the plurality of entity names and based at least in part on one or more heuristic rules. An element list pattern may be generated based at least in part on the first element list, and a second element list may be extracted from the one or more documents based at least in part on the element list pattern.

Claims

1. A method comprising:

accessing, using processing equipment, one or more documents from which to extract structured metadata from each of a plurality of hosts;

extracting, using the processing equipment, a plurality of entity names from one or more documents from a first host of the plurality of hosts using an entity name pattern;

determining whether to extract a first element list based at least in part on one or more heuristic rules or based at least in part on one or more pattern-based rules;

in response to determining to extract the first element list based at least in part on one or more heuristic rules, extracting, using the processing equipment, the first element list from the one or more documents based at least in part on a first entity name of the plurality of entity names and based at least in part on one or more heuristic rules;

validating the first element list based at least in part on a comparison of the first element list with one or more reference lists, wherein the first element list and the one or more reference lists are each associated with a same one of the plurality of entity names;

generating, using the processing equipment, an element list pattern based at least in part on the first element list and on the structured metadata of at least the first host, wherein generating the element list pattern includes determining an element node pattern, wherein the element node pattern is based at least in part on a document object model tree path from a document root node to an element node;

determining whether to extract a second element list based at least in part on one or more heuristic rules or based at least in part on one or more pattern-based rules; and

in response to determining to extract the second element list based at least in part on one or more pattern-based rules, extracting, using the processing equipment, the second element list from the one or more documents based at least in part on a second entity name and one or more pattern-based rules comprising the generated element list pattern.

2. The method of claim 1, further comprising clustering the one or more documents from each of the plurality of hosts.

3. The method of claim 1, further comprising validating the second element list based at least in part on a comparison of the second element list with one or more second reference lists, wherein the second element list and the one or more second reference lists are each associated with the second entity name.

4. The method of claim 1, wherein the one or more documents include one or more web pages, and wherein the accessing the one or more documents includes using a uniform resource locator pattern.

5. The method of claim 4, wherein the one or more web pages include one or more web documents, the method further comprising rendering the one or more web documents.

6. The method of claim 1, wherein extracting the plurality of entity names from the one of more documents includes locating a title tag in the one or more documents.

7. The method of claim 1, further comprising clustering the one or more documents by language.

8. The method of claim 1, wherein the one or more heuristic rules is based at least in part on a user input.

9. A system comprising:

one or more computers configured to;

access, using processing equipment, one or more documents from which to extract structured metadata from each of a plurality of hosts;

extract, using the processing equipment, a plurality of entity names from one or more documents from a first host of the plurality of hosts using an entity name pattern;

determine whether to extract a first element list based at least in part on one or more heuristic rules or based at least in part on one or more pattern-based rules;

in response to the determination to extract the first element list based at least in part on one or more heuristic rules, extract, using the processing equipment, a first element list from the one or more documents based at least in part on a first entity name of the plurality of entity names and based at least in part on one or more heuristic rules;

validate the first element list based at least in part on a comparison of the first element list with one or more reference lists, wherein the first element list and the one or more reference lists are each associated with a same one of the plurality of entity names;

generate, using the processing equipment, an element list pattern based at least in part on the first element list and on the structured metadata of at least the first host, wherein generating the element list pattern includes determining an element node pattern, wherein the element node pattern is based at least in part on a document object model tree path from a document root node to an element node;

determine whether to extract a second element list based at least in part on one or more heuristic rules or based at least in part on one or more pattern-based rules; and

in response to determining to extract the second element list based at least in part on one or more pattern-based rules, extract, using the processing equipment, a second element list from the one or more documents based at least in part on a second entity name and one or more pattern-based rules comprising the generated element list pattern.

10. The system of claim 9, wherein the processing equipment is further configured to cluster the one or more documents from each of the plurality of hosts.

11. The system of claim 9, wherein the processing equipment is further configured to validate the second element list based at least in part on a comparison of the second element list with one or more second reference lists, wherein the second element list and the one or more second reference lists are each associated with the second entity name.

12. The system of claim 9, wherein the one or more documents include one or more web pages, and wherein the processing equipment is further configured to use a uniform resource locator pattern.

13. The system of claim 12, wherein the one or more web pages include one or more web documents, and wherein the processing equipment is further configured to render the one or more web documents.

14. The system of claim 9, wherein the processing equipment is further configured to extract the plurality of entity names from the one of more documents by locating a title tag in the one or more documents.

15. The system of claim 9, wherein the processing equipment is further configured to cluster the one or more documents by language.

16. The system of claim 9, further comprising a user interface, wherein the one or more heuristic rules is based at least in part on a user input received by the user interface.