Detect annotation error by segmenting unannotated document segments into smallest partition

• US 10,176,157 B2
• Filed: 01/03/2015
• Issued: 01/08/2019
• Est. Priority Date: 01/03/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Allen, Corville O., Freed, Andrew R., Lim, Kyungae
• Primary Examiner(s): Paula, Cesar B
• Assistant Examiner(s): Nguyen, Luu-Phuong T
• Application Number:
  US14/588,928
• Publication Number:
  US 20160196249A1
• Time in Patent Office: 1,466 Days
• Field of Search: 715230
• CPC Class Codes: G06F 16/176 Support for shared access to files; File sharing support
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/106 Display of layout of documents; Previewing
• CPC Class Codes: G06F 40/131 Fragmentation of text files, e.g. creating reusable text-blocks; Linking to fragments, e.g. using XInclude; Namespaces
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

An approach is provided in which a knowledge manager detects an annotation error while attempting to annotate a document and, as such, segments the document into multiple document segments. The knowledge manager then detects another annotation error while attempting to annotate the multiple document segments, resulting in a set of unannotated document segments. In turn, the knowledge manager generates a notification that identifies the set of unannotated document segments.

Claims

1. An information handling system comprising:

one or more processors;

a memory coupled to at least one of the processors; and

a set of computer program instructions stored in the memory and executed by at least one of the processors in order to perform actions of;

detecting a first annotation error generated by an annotator while annotating a first document segment;

in response to detecting the first annotation error, segmenting the first document segment into a plurality of second document segments;

detecting one or more second annotation errors generated by the annotator while annotating at least one of the plurality of second document segments, resulting in one or more unannotated second document segments; and

in response to detecting the one or more second annotation errors, generating a notification that identifies at least one of the one or more unannotated second document segments.

2. The information handling system of claim 1 wherein the one or more processors perform additional actions comprising:

identifying a smallest segment size of the annotator, wherein the smallest segment size corresponds to a smallest document partition that the annotator is configured to annotate;

selecting one of the unannotated second document segments; and

segmenting the selected unannotated second document segment into a plurality of third document segments in response to determining that the selected unannotated second document segment is larger than the smallest segment size.

3. The information handling system of claim 2 wherein the one or more processors perform additional actions comprising:

detecting a third annotation error in response to attempting to annotate a selected one of the plurality of third document segments;

segmenting the selected third document segment into a plurality of fourth document segments that each correspond to the smallest segment size;

determining that a selected one of the plurality of fourth document segments generates a fourth annotation error; and

including an identifier in the notification that identifies the selected fourth document segment.

4. The information handling system of claim 2 wherein the one or more processors perform additional actions comprising:

annotating each of the plurality of third document segments;

determining that the annotator successfully annotated each of the plurality of third document segments; and

modifying the smallest segment size to correspond with the selected unannotated second document segment in response to the determining.

5. The information handling system of claim 1 wherein, prior to the detection of the first annotation error, the one or more processors perform additional actions comprising:

annotating, by the annotator, one or more baseline documents included in a corpus of documents;

logging a baseline annotation time of the annotator based upon the annotating of the one or more baseline documents; and

logging an annotator feature set corresponding to the annotator, wherein the annotation feature set includes at least one document size and at least one amount of annotations corresponding to at least one of the one or more baseline documents.

6. The information handling system of claim 5 wherein the one or more processors perform additional actions comprising:

identifying a document feature set of the first document segment;

computing an estimated annotation time to annotate the document based upon the baseline annotation time in response to determining that the document feature set matches the annotator feature set;

monitoring a current in-process annotation time taken by the annotator while in-process of annotating the first document segment; and

generating the first annotation error in response to determining that the in-process annotation time is longer than the estimated annotation time.

7. The information handling system of claim 6 wherein the one or more processors perform additional actions comprising:

computing an annotation time adjustment value by comparing the document feature set against the annotator feature set; and

generating the first annotation error in response to determining that the in-process annotation time is longer than a combination of the estimated annotation time and the annotation time adjustment value.

8. A computer program product stored in a non-transitory computer readable storage medium, comprising computer program code that, when executed by an information handling system, causes the information handling system to perform actions comprising:

detecting a first annotation error generated by an annotator while annotating a first document segment;

in response to detecting the first annotation error, segmenting the first document segment into a plurality of second document segments;

detecting one or more second annotation errors generated by the annotator while annotating at least one of the plurality of second document segments, resulting in one or more unannotated second document segments; and

in response to detecting the one or more second annotation errors, generating a notification that identifies at least one of the one or more unannotated second document segments.

9. The computer program product of claim 8 wherein the information handling system performs additional actions comprising:

identifying a smallest segment size of the annotator, wherein the smallest segment size corresponds to a smallest document partition that the annotator is configured to annotate;

selecting one of the unannotated second document segments; and

segmenting the selected unannotated second document segment into a plurality of third document segments in response to determining that the selected unannotated second document segment is larger than the smallest segment size.

10. The computer program product of claim 9 wherein the information handling system performs additional actions comprising:

detecting a third annotation error in response to attempting to annotate a selected one of the plurality of third document segments;

segmenting the selected third document segment into a plurality of fourth document segments that each correspond to the smallest segment size;

determining that a selected one of the plurality of fourth document segments generates a fourth annotation error; and

including an identifier in the notification that identifies the selected fourth document segment.

11. The computer program product of claim 9 wherein the information handling system performs additional actions comprising:

annotating each of the plurality of third document segments; and

determining that the annotator successfully annotated each of the plurality of third document segments; and

modifying the smallest segment size to correspond with the selected unannotated second document segment in response to the determining.

12. The computer program product of claim 8 wherein, prior to the detection of the first annotation error, the information handling system performs additional actions comprising:

annotating, by the annotator, one or more baseline documents included in a corpus of documents;

logging a baseline annotation time of the annotator based upon the annotating of the one or more baseline documents; and

logging an annotator feature set corresponding to the annotator, wherein the annotation feature set includes at least one document size and at least one amount of annotations corresponding to at least one of the one or more baseline documents.

13. The computer program product of claim 12 wherein the information handling system performs additional actions comprising:

identifying a document feature set of the first document segment;

computing an estimated annotation time to annotate the document based upon the baseline annotation time in response to determining that the document feature set matches the annotator feature set;

monitoring a current in-process annotation time taken by the annotator while in-process of annotating the first document segment; and

generating the first annotation error in response to determining that the in-process annotation time is longer than the estimated annotation time.