NLU TRAINING WITH MERGED ENGINE AND USER ANNOTATIONS

• US 20150356057A1
• Filed: 06/04/2014
• Published: 12/10/2015
• Est. Priority Date: 06/04/2014
• Status: Active Grant

• Current Assignee: Microsoft Technology Licensing LLC (Microsoft Corporation)
• Original Assignee: Nuance Communications, Inc. (Microsoft Corporation)
• Inventors: Sarkar, Debjani, Subramanian, Aparna
• Granted Patent: US 10,331,763 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 40/10 Text processing natural language analysis G06F40/20; semantic analysis G06F40/30; processing or translation of natural language G06F40/40
• CPC Class Codes: G06F 40/117 Tagging; Marking up details of markup languages G06F40/143; Designating a block; Setting of attributes style sheets, e.g. eXtensible Stylesheet Language Transformation [XSLT], G06F40/154
• CPC Class Codes: G06F 40/169 Annotation, e.g. comment data or footnotes
• CPC Class Codes: G06F 40/284 Lexical analysis, e.g. tokenisation or collocates
• CPC Class Codes: G16H 10/60 for patient-specific data, e.g. for electronic patient records
• CPC Class Codes: G16H 15/00 ICT specially adapted for medical reports, e.g. generation or transmission thereof

Abstract

Techniques for training a natural language understanding (NLU) engine may include generating a first annotation of free-form text documenting a healthcare patient encounter and a link between the first annotation and a corresponding portion of the text, using the NLU engine. A second annotation of the text and a link between the second annotation and a corresponding portion of the text may be received from a human user. The first annotation and its corresponding link may be merged with the second annotation and its corresponding link. Training data may be provided to the engine in the form of the text and the merged annotations and links.

Claims

1. A method comprising:

generating one or more engine annotations and engine links of a corpus of free-form text documenting a clinical patient encounter by applying a natural language understanding engine to the corpus of free-form text, wherein the one or more engine links associate each of the one or more engine annotations with a corresponding portion of the corpus of free-form text;

receiving, from a human user, one or more user annotations and user links of the corpus of free-form text, wherein the one or more user links associate each of the one or more user annotations with a corresponding portion of the corpus of free-form text;

merging the one or more engine annotations and engine links with the one or more user annotations and user links into one or more merged annotations and merged links; and

providing training data to the natural language understanding engine in the form of the corpus of free-form text and the one or more merged annotations and merged links.

2. The method of claim 1, wherein the one or more engine annotations, the one or more user annotations, and the one or more merged annotations comprise medical codes representing medical diagnoses and/or medical procedures.

3. The method of claim 1, further comprising, prior to receiving from the human user, the one or more user annotations and user links, providing to the human user one or more suggested annotations and suggested links in the form of the one or more engine annotations and engine links.

4. The method of claim 1, wherein merging the one or more engine annotations and engine links with the one or more user annotations and user links comprises comparing the one or more engine annotations and engine links with the one or more user annotations and user links and removing one or more redundant annotations.

5. The method of claim 1, wherein merging the one or more engine annotations and engine links with the one or more user annotations and user links comprises comparing an order of annotations including medical codes between the one or more engine annotations and the one or more user annotations and generating information identifying differences in order between the one or more engine annotations and the one or more user annotations.

6. The method of claim 1, further comprising collecting information for annotations in the one or more user annotations, the information comprising an order of annotations including medical codes in the one or more user annotations, an amount of time taken to input at least one annotation in the one or more user annotations, and/or a clinical practitioner associated with at least one annotation in the one or more user annotations.

7. The method of claim 1, wherein at least two documents of the corpus of documents pertain to a common medical field.

8. A system comprising:

a user interface configured to receive, from a human user, one or more user annotations and user links of a corpus of free-form text documenting a clinical patient encounter, wherein the one or more user links associate each of the one or more user annotations with a corresponding portion of the corpus of free-form text; and

a natural language understanding engine implemented on a processor configured to;

generate one or more engine annotations and engine links of the corpus of free-form text by applying the natural language understanding engine to the corpus of free-form text, wherein the one or more engine links associate each of the one or more engine annotations with a corresponding portion of the corpus of free-form text; and

receive training data in the form of the corpus of free-form text and one or more merged annotations and merged links of the corpus of free-form text.

9. The system of claim 8, wherein the one or more engine annotations, the one or more user annotations, and the one or more merged annotations comprise medical codes representing medical diagnoses and/or medical procedures.

10. The system of claim 8, wherein the user interface is further configured, prior to receiving, from the human user, the one or more user annotations and user links, to provide to the human user one or more suggested annotations and suggested links in the form of the one or more engine annotations and engine links.

11. The system of claim 8, wherein the one or more merged annotations lacks redundant annotations between the one or more user annotations and the one or more engine annotations.

12. The system of claim 8, further comprising a processor configured to form the one or more merged annotations by comparing an order of annotations including medical codes between the one or more engine annotations and engine links and the one or more user annotations and user links and generating information identifying differences in order between the one or more engine annotations and engine links and the one or more user annotations and user links.

13. The system of claim 8, wherein the user interface is further configured to collect information for annotations in the one or more user annotations, the information comprising an order of annotations including medical codes in the one or more user annotations, an amount of time taken to input at least one annotation in the one or more user annotations, and/or a clinical practitioner associated with at least one annotation in the one or more user annotations.

14. The system of claim 8, wherein at least two documents of the corpus of documents pertain to a common medical field.

15. A computer-readable storage medium having instructions that, when executed by a processor, cause performance of a method comprising:

generating one or more engine annotations and engine links of a corpus of free-form text documenting a clinical patient encounter by applying a natural language understanding engine to the corpus of free-form text, wherein the one or more engine links associate each of the one or more engine annotations with a corresponding portion of the corpus of free-form text;

receiving, from a human user, one or more user annotations and user links of the corpus of free-form text, wherein the one or more user links associate each of the one or more user annotations with a corresponding portion of the corpus of free-form text;

merging the one or more engine annotations and engine links with the one or more user annotations and user links into one or more merged annotations and merged links; and

providing training data to the natural language understanding engine in the form of the corpus of free-form text and the one or more merged annotations and merged links.

16. The computer-readable storage medium of claim 15, wherein the one or more engine annotations, the one or more user annotations, and the one or more merged annotations comprise medical codes representing medical diagnoses and/or medical procedures.

17. The computer-readable storage medium of claim 15, wherein the method further comprises, prior to receiving, from the human user, the one or more user annotations and user links, providing to the human user one or more suggested annotations and suggested links in the form of the one or more engine annotations and engine links.

18. The computer-readable storage medium of claim 15, wherein merging the one or more engine annotations and engine links with the one or more user annotations and user links comprises comparing the one or more engine annotations and engine links with the one or more user annotations and user links and removing one or more redundant annotations.

19. The computer-readable storage medium of claim 15, wherein merging the one or more engine annotations and engine links with the one or more user annotations and user links comprises comparing an order of annotations including medical codes between the one or more engine annotations and the one or more user annotations and generating information identifying differences in order between the one or more engine annotations and the one or more user annotations.

20. The computer-readable storage medium of claim 15, wherein the method further comprises collecting information for annotations in the one or more user annotations, the information comprising an order of annotations including medical codes in the one or more user annotations, an amount of time taken to input at least one annotation in the one or more user annotations, and/or a clinical practitioner associated with at least one annotation in the one or more user annotations.