Detect annotation error locations through unannotated document segment partitioning

• US 10,235,350 B2
• Filed: 09/29/2015
• Issued: 03/19/2019
• Est. Priority Date: 01/03/2015
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Allen, Corville O., Freed, Andrew R., Lim, Kyungae
• Primary Examiner(s): Paula, Cesar B
• Assistant Examiner(s): Nguyen, Luu-Phuong T
• Application Number:
  US14/868,816
• Publication Number:
  US 20160196250A1
• Time in Patent Office: 1,267 Days
• Field of Search: 715230
• CPC Class Codes: G06F 16/176 Support for shared access to files; File sharing support
• CPC Class Codes: G06F 16/215 Improving data quality; Data cleansing, e.g. de-duplication, removing invalid entries or correcting typographical errors
• CPC Class Codes: G06F 16/285 Clustering or classification
• CPC Class Codes: G06F 16/35 Clustering; Classification
• CPC Class Codes: G06F 16/958 Organisation or management of web site content, e.g. publishing, maintaining pages or automatic linking
• CPC Class Codes: G06F 40/103 Formatting, i.e. changing of presentation of documents automatic justification G06F40/189; automatic line break hyphenation G06F40/191
• CPC Class Codes: G06F 40/106 Display of layout of documents; Previewing
• CPC Class Codes: G06F 40/131 Fragmentation of text files, e.g. creating reusable text-blocks; Linking to fragments, e.g. using XInclude; Namespaces
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/20 Natural language analysis semantic analysis of natural language G06F40/30
• CPC Class Codes: G06F 40/289 Phrasal analysis, e.g. finite state techniques or chunking
• CPC Class Codes: G06F 40/30 Semantic analysis

Abstract

An approach is provided in which a knowledge manager detects an annotation error while attempting to annotate a document and, as such, segments the document into multiple document segments. The knowledge manager then detects another annotation error while attempting to annotate the multiple document segments, resulting in a set of unannotated document segments. In turn, the knowledge manager generates a notification that identifies the set of unannotated document segments.

Claims

1. A method implemented by an information handling system that includes a memory and a processor, the method comprising:

detecting a first annotation error generated by an annotator while annotating a first document segment;

in response to detecting the first annotation error, segmenting, by a processor, the first document segment into a plurality of second document segments;

detecting one or more second annotation errors generated by the annotator while annotating at least one of the plurality of second document segments, resulting in one or more unannotated second document segments;

segmenting a selected one of the one or more unannotated second document segments into a plurality of third document segments;

annotating each of the plurality of third document segments; and

in response to determining that the annotator successfully annotated each of the plurality of third document segments, generating a notification that links the selected unannotated second document segment to at least one of the one or more second annotation errors.

2. The method of claim 1 further comprising:

identifying a smallest segment size of the annotator, wherein the smallest segment size corresponds to a smallest document partition that the annotator is configured to annotate;

segmenting the selected unannotated second document segment into the plurality of third document segments in response to determining that the selected unannotated second document segment is larger than the smallest segment size.

3. The method of claim 2 further comprising:

detecting a third annotation error in response to attempting to annotate a selected one of the plurality of third document segments;

segmenting the selected third document segment into a plurality of fourth document segments that each correspond to the smallest segment size;

determining that a selected one of the plurality of fourth document segments generates a fourth annotation error; and

including an identifier in the notification that identifies the selected fourth document segment.

4. The method of claim 1 wherein, prior to the detection of the first annotation error, the method further comprises:

annotating, by the annotator, one or more baseline documents included in a corpus of documents;

logging a baseline annotation time of the annotator based upon the annotating of the one or more baseline documents; and

logging an annotator feature set corresponding to the annotator, wherein the annotation feature set includes at least one document size and at least one amount of annotations corresponding to at least one of the one or more baseline documents.

5. The method of claim 4 further comprising:

identifying a document feature set of the first document segment;

computing an estimated annotation time to annotate the document based upon the baseline annotation time in response to determining that the document feature set matches the annotator feature set;

monitoring a current in-process annotation time taken by the annotator while in-process of annotating the first document segment; and

generating the first annotation error in response to determining that the in-process annotation time is longer than the estimated annotation time.

6. The method of claim 5 further comprising:

computing an annotation time adjustment value by comparing the document feature set against the annotator feature set; and

generating the first annotation error in response to determining that the in-process annotation time is longer than a combination of the estimated annotation time and the annotation time adjustment value.