Systems and methods for modular information extraction

• US 7,987,416 B2
• Filed: 11/14/2007
• Issued: 07/26/2011
• Est. Priority Date: 11/14/2007
• Status: Active Grant

• Current Assignee: SAP SE
• Original Assignee: SAP AG (SAP SE)
• Inventors: Barczyński, Wojciech Michal, Loeser, Alexander, Brauer, Falk
• Primary Examiner(s): HUYNH, CONG LAC T
• Application Number:
  US11/939,794
• Publication Number:
  US 20090125542A1
• Time in Patent Office: 1,350 Days
• Field of Search: 715/230, 715/200, 715/255, 707/706, 707/713
• US Class Current: 715/200
• CPC Class Codes: G06F 16/951 Indexing; Web crawling techniques
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes

Abstract

Embodiments of the present invention include a computer-implemented method of extracting information. In one embodiment, the present invention comprises defining a plurality of reusable operators, wherein each operator performs a predefined information extraction task different from the other operators. Composite annotators may be created by specifying a composition of the reusable operators. Each operator may receive a searchable item, such as a web page or an annotation, and may generate one or more output annotations. The output annotations may be further processed by other reusable operators and the annotations may be stored in a repository for use during a search.

Claims

1. A computer-implemented method of extracting information comprising:

defining a plurality of reusable operators, wherein each operator performs a predefined information extraction task different from the other operators;

specifying a composition of said reusable operators to form a composite annotator, wherein each operator receives a searchable item and generates one or more output annotations; and

storing the output annotations for use during a search,

wherein the plurality of reusable operators include an extraction operator, wherein the extraction operator identifies features based on predefined criteria and generates one or more output annotations comprising the features extracted from one or more searchable items,

the searchable items comprising text, wherein the extraction operator extracts specified text by matching text in each of said searchable items against a first rule and a second rule, wherein if text satisfies the first rule and the second rule, text between the text satisfying the first rule and text satisfying the second rule is stored in an output annotation, and wherein the extraction operator assigns a specified type to each of the one or more output annotations.

2. The method of claim 1, the searchable items comprising text, wherein the extraction operator extracts specified text using a specified rule, and assigns a specified type to each of the one or more output annotations, wherein extracted features in the one or more output annotations comprise text that satisfies the rule.

3. The method of claim 1, the searchable items comprising text, wherein the extraction operator extracts specified text by matching text in each of said searchable items against a specified list of text and assigns a specified type to each of the one or more output annotations.

4. The method of claim 1, wherein the plurality of reusable operators include a context operator, wherein the context operator identifies features based on one or more other annotations and generates one or more output annotations comprising the identified features extracted from one or more searchable items.

5. The method of claim 4 wherein the context operator identifies features based on predefined analysis criteria.

6. The method of claim 4, the searchable items comprising text, and wherein the context operator receives an input annotation and a reference annotation and generates an output annotation comprising a plurality of text adjacent to the reference annotation.

7. The method of claim 6 wherein the context operator receives a specification of the output annotation type.

8. The method of claim 7 wherein the context operator receives an analysis window specifying the number of characters to be extracted from the input annotation.

9. The method of claim 6 wherein the context operator extracts text to the left of said reference annotation.

10. The method of claim 6 wherein the context operator extracts text to the right of said reference annotation.

11. The method of claim 1, wherein the plurality of reusable operators include a composition operator, wherein the composition operator generates one or more output annotations based on a relationship between a plurality of input annotations.

12. The method of claim 1, wherein the plurality of reusable operators include a composition operator, wherein the composition operator generates an output annotation based on a relationship between an input annotation and an input annotation type.

13. A computer-implemented method of extracting information comprising:

defining a plurality of reusable operators, wherein each operator performs a predefined information extraction task different from the other operators;

specifying a composition of said reusable operators to form a composite annotator, wherein each operator receives a searchable item and generates one or more output annotations; and

storing the output annotations for use during a search,

wherein the plurality of reusable operators include an composition operator, the searchable items comprising text, and wherein the composition operator receives an input annotation and two reference annotations and generates one or more output annotations comprising text between the two reference annotations.

14. The method of claim 13 wherein the composition operator receives a specification of an output annotation type.

15. The method of claim 13 wherein the composition operator receives a specification of a context window.

16. A computer-implemented method of extracting information comprising:

defining a plurality of reusable operators, wherein each operator performs a predefined information extraction task different from the other operators;

specifying a composition of said reusable operators to form a composite annotator, wherein each operator receives a searchable item and generates one or more output annotations; and

storing the output annotations for use during a search,

wherein the plurality of reusable operators include an composition operator, the searchable items comprising text, and wherein the composition operator receives an input annotation and two reference annotation types and generates one or more output annotations comprising text between the two reference annotation types.

17. The method of claim 16, the two reference annotation types each comprising a set comprising a plurality of annotations.

18. A computer-implemented method of extracting information comprising:

defining a plurality of reusable operators, wherein each operator performs a predefined information extraction task different from the other operators;

specifying a composition of said reusable operators to form a composite annotator, wherein each operator receives a searchable item and generates one or more output annotations; and

storing the output annotations for use during a search,

wherein the plurality of reusable operators include an composition operator, the searchable items comprising text, and wherein the composition operator receives an input annotation, two reference annotations, and a rule, and generates one or more output annotations comprising text between the two reference annotations that satisfy the rule.

19. A computer-implemented method of extracting information comprising:

defining a plurality of reusable operators, wherein each operator performs a predefined information extraction task different from the other operators;

specifying a composition of said reusable operators to form a composite annotator, wherein each operator receives a searchable item and generates one or more output annotations; and

storing the output annotations for use during a search,

wherein the plurality of reusable operators include an composition operator, the searchable items comprising text, and wherein the composition operator receives an input annotation, two reference annotation types, and a rule, and generates one or more output annotations comprising text between the two reference annotation types that satisfy the rule.

20. A computer-implemented method of extracting information comprising:

defining a plurality of reusable operators, wherein each operator performs a predefined information extraction task different from the other operators;

specifying a composition of said reusable operators to form a composite annotator, wherein each operator receives a searchable item and generates one or more output annotations; and

storing the output annotations for use during a search,

wherein the plurality of reusable operators include an composition operator, and

wherein the composition operator receives an input annotation, first and second reference annotations, and an expression specifying a semantic relationship between the first reference annotation and the second reference annotation, and wherein the composition operator checks if the first and second reference annotations are within the input annotation and selects annotations matching the expression.

21. The method of claim 20 wherein the first reference annotation is a left annotation and the second reference annotation is a right annotation.

22. The method of claim 20 wherein the composition operation further receives a context window, and wherein both the first and second reference annotations are within the context window.

23. The method of claim 20 wherein the first and second reference annotations are one of (i) a single first reference annotation and a single second reference annotation, (ii) first and second annotation types, or (iii) first and second sets of annotations.

24. A computer-implemented method of extracting information comprising:

defining a plurality of reusable operators, wherein each operator performs a predefined information extraction task different from the other operators;

specifying a composition of said reusable operators to form a composite annotator, wherein each operator receives a searchable item and generates one or more output annotations; and

storing the output annotations for use during a search,

wherein the plurality of reusable operators include an composition operator, and

wherein the composition operator receives a first input annotation, a second input annotation, a first parameter for specifying the output annotation type, and a second parameter for specifying a type of relationship that the output annotation will express, wherein the second parameter specifies the type of relationship as one of (i) a hierarchy, wherein the output annotation was created from the first input annotation and the second input annotation, or (ii) an attribute, wherein the output annotation is a composition of the first input annotation and the second input annotation.

25. The method of claim 24, wherein the output annotation is an object, the output annotation object storing a common parent to the first and second input annotations.

26. A computer-readable medium containing instructions for controlling a computer system to perform a method of extracting information comprising:

receiving a plurality of first annotations, the first annotations comprising text;

processing the plurality of first annotations using a reusable extraction operator, and in accordance therewith, generating a plurality of second annotations, wherein the extraction operator identifies text in the first annotations that satisfy one or more rules or match text in a predefined list of text, and generates one or more output annotations comprising the identified text; and

processing the plurality of second annotations using a reusable context operator, and in accordance therewith, generating a plurality of third annotations, the second annotations comprising text, wherein the context operator receives each of the second annotations and a reference annotation and generates output annotations comprising a plurality of text adjacent to the reference annotation;

processing the plurality of third annotations using a reusable composition operator, wherein the composition operator receives each of the third annotations, one or more first reference annotations, and one or more second reference annotations, and generates output annotations comprising text between the first and second reference annotations.

27. The computer-readable medium of claim 26 wherein the composition operator receives a rule, and wherein output annotations comprise text that satisfies the rule, and if an annotation in the plurality of third annotations does not include the first reference annotation or the second reference annotation or does not satisfy said rule, then no output annotation is generated by the composition operator.

28. The computer-readable medium of claim 26, further comprising

storing the plurality of third annotations in a repository;

and

accessing the plurality of third annotations in response to a search request.

29. A computer-readable medium containing instructions for controlling a computer system to perform a method of extracting information comprising:

receiving a plurality of first annotations;

processing the plurality of first annotations using an reusable extraction operator, and in accordance therewith, generating a plurality of second annotations; and

processing the plurality of second annotations using a reusable composition operator, and in accordance therewith, generating a plurality of third annotations, the second annotations comprising text, wherein the composition operator receives each of the second annotations, one or more first reference annotations, and one or more second reference annotations, and generates output annotations comprising text between the first and second reference annotations.