Annotation Collision Detection in a Question and Answer System

• US 20140359421A1
• Filed: 06/03/2013
• Published: 12/04/2014
• Est. Priority Date: 06/03/2013
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Allen, Corville O., Carrier, Scott R., Sizemore, Robert C., Salmon, Richard A.
• Granted Patent: US 10,642,928 B2
• Time in Patent Office: Days
• US Class Current: 715/230
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06Q 10/101 Collaborative creation, e.g. joint development of products or services

Abstract

Mechanisms are provided for resolving annotation conflicts in a portion of content. A portion of content is received and annotation operations are performed on it using a plurality of annotators. Each annotator in the plurality of annotators generates a set of annotations for the portion of content, thereby generating a plurality of sets of annotations for the portion of content. The plurality of sets of annotations are analyzed to identify a conflict between a first annotation and a second annotation in the plurality of sets of annotations. The conflict between the first annotation and the second annotation is resolved to generate a resultant set of annotations for the portion of content. The resultant set of annotations is stored in an annotations data structure associated with the portion of content.

Claims

1. A method, in a data processing system comprising a processor and a memory, for resolving annotation conflicts in a portion of content, the method comprising:

receiving, in the data processing system, a portion of content for annotating;

performing, by the data processing system, annotation operations on the portion of content using a plurality of annotators, wherein each annotator in the plurality of annotators generates a set of annotations for the portion of content, thereby generating a plurality of sets of annotations for the portion of content;

analyzing, by the data processing system, the plurality of sets of annotations for the portion of content to identify a conflict between a first annotation and a second annotation in the plurality of sets of annotations;

resolving, by the data processing system, the conflict between the first annotation and the second annotation to generate a resultant set of annotations for the portion of content; and

storing, by the data processing system, the resultant set of annotations in an annotations data structure associated with the portion of content.

2. The method of claim 1, further comprising:

utilizing the resultant set of annotations for the portion of content to perform natural language processing operation on the portion of content.

3. The method of claim 1, further comprising:

performing a question and answer operation, of a question and answer system implemented by the data processing system, on the portion of content using the resultant set of annotations.

4. The method of claim 1, wherein resolving the conflict between the first annotation and the second annotation comprises at least one of automatically selecting one of the first annotation or the second annotation to include in the resultant set of annotations, presenting conflict information identifying the conflict between the first annotation and the second annotation to a user via a user interface to request user input identifying a desired method to rectify the identified conflict.

5. The method of claim 1, further comprising identifying a domain of the portion of content, and wherein resolving the conflict between the first annotation and the second annotation comprises selecting one of the first annotation or second annotation based on the identified domain of the portion of content.

6. The method of claim 5, wherein resolving the conflict between the first annotation and the second annotation comprises at least one of:

performing a reverse lookup of terms in the first annotation and the second annotation, in a domain specific language model for the domain of the portion of content to determine if there are matching terms in the domain specific language model;

or

performing a lookup of terms in the first annotation and the second annotation, in a domain specific corpora of information to determine if there are matching terms in the domain specific corpora of information.

7. The method of claim 6, further comprising:

in response to a term in one of the first annotation or the second annotation having a matching term in the domain specific language model or the domain specific corpora of information, determining whether a usage of the matching term meets a predetermined criteria; and

in response to the matching term meeting the predetermined criteria, maintaining a corresponding one of the first annotation or second annotation in the resultant set of annotations.

8. The method of claim 7, wherein determining whether a usage of the matching term meets a predetermined criteria comprises:

determining a number of instances of the matching term in one of the domain specific language model or the domain specific corpora of information;

comparing the number of instances to a threshold value; and

in response to the number of instances having a predetermined relationships to the threshold value, determining that the usage of the matching term meets the predetermined criteria.

9. The method of claim 1, wherein the portion of content is an input question to a question and answer (QA) system to be answered by the QA system by searching a corpus of information.

10. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a computing device, causes the computing device to:

receive a portion of content for annotating;

perform annotation operations on the portion of content using a plurality of annotators, wherein each annotator in the plurality of annotators generates a set of annotations for the portion of content, thereby generating a plurality of sets of annotations for the portion of content;

analyze the plurality of sets of annotations for the portion of content to identify a conflict between a first annotation and a second annotation in the plurality of sets of annotations;

resolve the conflict between the first annotation and the second annotation to generate a resultant set of annotations for the portion of content; and

store the resultant set of annotations in an annotations data structure associated with the portion of content.

11. The computer program product of claim 10, wherein the computer readable program further causes the computing device to:

utilize the resultant set of annotations for the portion of content to perform natural language processing operation on the portion of content.

12. The computer program product of claim 10, wherein the computer readable program further causes the computing device to:

perform a question and answer operation, of a question and answer system implemented by the data processing system, on the portion of content using the resultant set of annotations.

13. The computer program product of claim 10, wherein the computer readable program further causes the computing device to resolve the conflict between the first annotation and the second annotation by at least one of automatically selecting one of the first annotation or the second annotation to include in the resultant set of annotations, presenting conflict information identifying the conflict between the first annotation and the second annotation to a user via a user interface to request user input identifying a desired method to rectify the identified conflict.

14. The computer program product of claim 10, wherein the computer readable program further causes the computing device to identify a domain of the portion of content, and wherein the computer readable program further causes the computing device to resolve the conflict between the first annotation and the second annotation at least by selecting one of the first annotation or second annotation based on the identified domain of the portion of content.

15. The computer program product of claim 14, wherein the computer readable program further causes the computing device to resolve the conflict between the first annotation and the second annotation by at least one of:

performing a reverse lookup of terms in the first annotation and the second annotation, in a domain specific language model for the domain of the portion of content to determine if there are matching terms in the domain specific language model;

or

performing a lookup of terms in the first annotation and the second annotation, in a domain specific corpora of information to determine if there are matching terms in the domain specific corpora of information.

16. The computer program product of claim 13, wherein the computer readable program further causes the computing device to:

determine, in response to a term in one of the first annotation or the second annotation having a matching term in the domain specific language model or the domain specific corpora of information, whether a usage of the matching term meets a predetermined criteria; and

in response to the matching term meeting the predetermined criteria, maintaining a corresponding one of the first annotation or second annotation in the resultant set of annotations.

17. The computer program product of claim 16, wherein the computer readable program further causes the computing device to determine whether a usage of the matching term meets predetermined criteria at least by:

determining a number of instances of the matching term in one of the domain specific language model or the domain specific corpora of information;

comparing the number of instances to a threshold value; and

in response to the number of instances having a predetermined relationships to the threshold value, determining that the usage of the matching term meets the predetermined criteria.

18. An apparatus comprising:

a processor; and

a memory coupled to the processor, wherein the memory comprises instructions which, when executed by the processor, cause the processor to;

receive a portion of content for annotating;

perform annotation operations on the portion of content using a plurality of annotators, wherein each annotator in the plurality of annotators generates a set of annotations for the portion of content, thereby generating a plurality of sets of annotations for the portion of content;

analyze the plurality of sets of annotations for the portion of content to identify a conflict between a first annotation and a second annotation in the plurality of sets of annotations;

resolve the conflict between the first annotation and the second annotation to generate a resultant set of annotations for the portion of content; and

store the resultant set of annotations in an annotations data structure associated with the portion of content.

19. The apparatus of claim 18, wherein the instructions further cause the processor to identify a domain of the portion of content, and wherein the instructions further cause the processor to resolve the conflict between the first annotation and the second annotation at least by selecting one of the first annotation or second annotation based on the identified domain of the portion of content.

20. The apparatus of claim 19, wherein the instructions further cause the processor to resolve the conflict between the first annotation and the second annotation by at least one of:

performing a reverse lookup of terms in the first annotation and the second annotation, in a domain specific language model for the domain of the portion of content to determine if there are matching terms in the domain specific language model;

or

performing a lookup of terms in the first annotation and the second annotation, in a domain specific corpora of information to determine if there are matching terms in the domain specific corpora of information.