Systems and methods for content extraction

• US 20070050708A1
• Filed: 03/30/2006
• Published: 03/01/2007
• Est. Priority Date: 03/30/2005
• Status: Active Grant

• Current Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Original Assignee: Trustees Of Columbia University In The City Of New York (Columbia University)
• Inventors: Kaiser, Gail, Gupta, Suhit, Stolfo, Salvatore
• Granted Patent: US 8,468,445 B2
• Time in Patent Office: Days
• US Class Current: 715/235
• CPC Class Codes: G06F 16/80 of semi-structured data, e.g. markup language structured data such as SGML, XML or HTML content-based retrieval of web data G06F16/95
• CPC Class Codes: G06F 16/84 Mapping; Conversion
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/143 Markup, e.g. Standard Generalized Markup Language [SGML] or Document Type Definition [DTD]

Abstract

Systems and methods are presented for content extraction from markup language text. The content extraction process may parse markup language text into a hierarchical data model and then apply one or more filters. Output filters may be used to make the process more versatile. The operation of the content extraction process and the one or more filters may be controlled by one or more settings set by a user, or automatically by a classifier. The classifier may automatically enter settings by classifying markup language text and entering settings based on this classification. Automatic classification may be performed by clustering unclassified markup language texts with previously classified markup language texts.

Claims

1. A method for extracting content from input markup language text comprising:

(a) parsing the input markup language text into a first hierarchical data model;

(b) generating a second hierarchical data model based on the first hierarchical data model using one or more filters to remove content from the first hierarchical data model; and

(c) generating output markup language text from the second hierarchical data model.

2. The method of claim 1, wherein one of the one or more filters removes content from the first hierarchical data model using a ratio of the number of links to the number of non-linked words.

3. The method of claim 2, further comprising adding content to the second hierarchical data model corresponding to the content removed from the first hierarchical data model by one of the one or more filters.

4. The method of claim 3, wherein the content added to the hierarchical data model is a plurality of links.

5. The method of claim 4, further comprising comparing the first hierarchical data model to the second hierarchical data model and ignoring the result of a filter pass based on the comparison.

6. The method of claim 1, wherein the operation of at least one of the one or more filters is based on one or more settings.

7. The method of claim 6, wherein the one or more settings are determined by a user.

8. The method of claim 6, wherein the one or more settings are determined automatically.

9. The method of claim 8, wherein the one or more settings are determined automatically based on the classification of the input markup language text.

10. The method of claim 9, wherein the classification of the input markup language text is determined automatically.

11. The method of claim 10, wherein the automatic determination is done using clustering.

12. The method of claim 2, wherein one of the one or more filters removes at least one of tables, programming script, styles and image links from the first hierarchical data model based on one or more predetermined rules.

13. The method of claim 12, wherein one of the one or more predetermined rules is removing tables containing less than a threshold number of characters.

14. The method of claim 13, wherein one of the one or more predetermined rules is removing image links that have a source equivalent to any source within a set of blocked sources.

15. The method of claim 2, wherein the output markup language text generated from the second hierarchical data model is in a different markup language than the input markup language text.

16. A method for automatically classifying a markup language text that is accessible at an Internet domain comprising:

(a) retrieving from one or more data repositories, data associated with the Internet domain;

(b) computing a first identifier for the Internet domain based on at least the data associated with the Internet domain and the markup language text;

(c) computing a measure of similarity between the computed first identifier and each of a first plurality of previously classified identifiers; and

(d) assigning the markup language text a classification based on the computed measure of similarity between the computed first identifier and each of the first plurality of previously classified identifiers.

17. The method of claim 16, wherein the classification assigned to the markup language text is the same classification as that of the previously classified identifier with the best measure of similarity to the computed first identifier.

18. The method of claim 16, wherein the classification assigned to the markup language text is a new classification.

19. The method of claim 17, wherein the one or more data repositories are search engines and the data associated with the Internet domain is a search result.

20. The method of claim 19, wherein computing the first identifier comprises computing, for each of a plurality of words in a predetermined set of words, a frequency of each word in the markup language text and the search result.

21. The method of claim 20, wherein the predetermined set of words are generated by:

(a) retrieving markup language text from an Internet domain;

(b) retrieving search results associated with the Internet domain from one or more search engines;

(c) computing a frequency for each of a plurality of words in the search results and the markup language text; and

(d) adding to the predetermined set of words each of the plurality of words whose frequency is greater than a threshold.

22. The method of claim 21, further comprising adding to the predetermined set of words each of the plurality of words whose frequency is one.

23. The method of claim 20, wherein computing the measure of similarity comprises computing the Manhattan distance between the computed first identifiers and each of the first plurality of previously classified and previously computed identifiers.

24. The method of claim 16, further comprising retrieving settings for a filter based on the classification assigned to the markup language text.

25. The method of claim 16, further comprising:

(a) computing a second identifier for the markup language text based on the layout of the markup language text;

(b) computing a measure of similarity between the second identifier and each of a second plurality of previously classified identifiers; and

(c) assigning the markup language text a classification based on both the first identifier and the second identifier.

26. A system for extracting content from markup language text comprising:

(a) a parser for converting the markup language text into a hierarchical data model;

(b) one or more filters for removing content from the hierarchical data model; and

(c) a data repository containing settings for each of the one or more filters.

27. The system of claim 26, wherein the hierarchical data model is a document object model.

28. The system of claim 26, further comprising a user interface for changing the settings for at least one of the one or more filters.

29. The system of claim 28, further comprising an output filter for converting the markup language text into a different markup language.

30. The system of claim 29, further comprising a data repository of previously determined settings.

31. The system of claim 30, further comprising a classifier for automatically changing the settings of one of the one or more filters using the data repository of previously determined settings.

32. The system of claim 31, wherein the classifier comprises a key word list and data repository of identifiers.

33. The system of claim 32, further comprising one or more data repositories containing data associated with a plurality of Internet domains.