Computer-implemented method, system, and program product for evaluating annotations to content

• US 20070005592A1
• Filed: 06/21/2005
• Published: 01/04/2007
• Est. Priority Date: 06/21/2005
• Status: Abandoned Application

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Naphade, Milind, Kender, John
• Application Number:
  US11/158,223
• Publication Number:
  US 20070005592A1
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06N 5/02 Knowledge representation; Symbolic representation

Abstract

A method, system and program product for evaluating annotations to content are described. Under aspects of the present invention, annotations made to content are received and evaluated for accuracy. Each annotation typically includes at least one element (e.g., terms) describing the content. The evaluation includes a syntactic level evaluation and at least one of a semantic level evaluation, a source level evaluation, a content level evaluation, or an annotator level evaluation. Based on the evaluations, feedback can be provided to an annotator making the annotations.

Claims

1. A computer-implemented method for evaluating annotations to content, comprising:

receiving a set of annotations for content, wherein each of the set of annotations includes at least one element that describes the content;

performing a syntactic level evaluation and at least one of a semantic level evaluation, a source level evaluation, a content level evaluation, or an annotator level evaluation to determine accuracy of the set of annotations; and

providing feedback based on the evaluations.

2. The computer-implemented method of claim 1, wherein the syntactic level evaluation comprises:

determining whether the set of annotations includes a required element based on a context of the content;

checking a spelling of the at least one element in each of the set of annotations; and

determining whether the at least one element in each of the set of annotations meets a length requirement.

3. The computer-implemented method of claim 1, wherein the semantic level evaluation comprises:

determining whether the at least one element in the set of annotations meets with an expected frequency of use; and

determining whether any groupings of the at least one element in each of the set of annotations are commonly used together or are rarely used together.

4. The computer-implemented method of claim 1, wherein the source level evaluation comprises:

determining whether the at least one element in each of the set of annotations matches a temporal setting of the content; and

determining whether the at least one element in each of the set of annotations is reflective of a provider of the content.

5. The computer-implemented method of claim 1, wherein the content level evaluation comprises:

locating other content that is related to the content based on the set of annotations; and

clustering the content and the other content.

6. The computer-implemented method of claim 1, wherein the annotator level evaluation comprises:

determining whether the at least one element includes an element that is unique to a particular annotator; and

identifying a new element to replace the element.

7. A system for evaluating annotations to content, comprising:

an annotation reception system configured to receive a set of annotations for content, wherein each of the set of annotations includes at least one element that describes the content;

an annotator evaluation system configured to perform a syntactic level evaluation and at least one of a semantic level evaluation, a source level evaluation, a content level evaluation, or an annotator level evaluation to determine accuracy of the set of annotations; and

a feedback system configured to provide feedback based on the evaluations.

8. The system of claim 7, wherein the syntactic level evaluation determines whether the set of annotations includes a required element based on a context of the content, checks a spelling of the at least one element in each of the set of annotations, and determines whether the at least one element in each of the set of annotations meets a length requirement.

9. The system of claim 7, wherein the semantic level evaluation determines whether the at least one element in the set of annotations meets with an expected frequency of use, and determines whether any groupings of the at least one element in each of the set of annotations are commonly used together.

10. The system of claim 7, wherein the source level evaluation determines whether the at least one element in each of the set of annotations matches a temporal setting of the content, and determines whether the at least one element in each of the set of annotations is reflective of a provider of the content.

11. The system of claim 7, wherein the content level evaluation locates other content that is related to content based on the set of annotations, and clusters the content and the other content.

12. The system of claim 7, wherein the annotator level evaluation determines whether the at least one element includes an element that is unique to a particular annotator, and identifies a new element to replace the element.

13. The system of claim 7, wherein the feedback is provided to an annotator to guide the creation of the set of annotations

14. A program product stored on a computer useable medium for evaluating annotations to content, the program product comprising program code for causing a computer system to perform the following functions:

receiving a set of annotations for content, wherein each of the set of annotations includes at least one element that describes the content;

performing a syntactic level evaluation and at least one of a semantic level evaluation, a source level evaluation, a content level evaluation, or an annotator level evaluation to determine accuracy of the set of annotations; and

providing feedback based on the evaluations.

15. The program product of claim 14, wherein the syntactic level evaluation determines whether the set of annotations includes a required element based on a context of the content, checks a spelling of the at least one element in each of the set of annotations, and determines whether the at least one element in each of the set of annotations meets a length requirement.

16. The program product of claim 14, wherein the semantic level evaluation determines whether the at least one element in the set of annotations meets with an expected frequency of use, and determines whether any groupings of the at least one element in each of the set of annotations are commonly used together.

17. The program product of claim 14, wherein the source level evaluation determines whether the at least one element in each of the set of annotations matches a temporal setting of the content, and determines whether the at least one element in each of the set of annotations is reflective of a provider of the content.

18. The program product of claim 14, wherein the content level evaluation locates other content that is related to content based on the set of annotations, and clusters the content and the other content.

19. The program product of claim 14, wherein the annotator level evaluation determines whether the at least one element includes an element that is unique to a particular annotator, identifies a new element to replace the terms.

20. A method for deploying an application for evaluating annotations to content, comprising:

providing a computer infrastructure being operable to;

receive a set of annotations for content, wherein each of the set of annotations includes at least one element that describes the content;

perform a syntactic level evaluation and at least one of a semantic level evaluation, a source level evaluation, a content level evaluation, or an annotator level evaluation to determine accuracy of the set of annotations; and

provide feedback based on the evaluations.