IMAGE ANALYTICS QUESTION ANSWERING

• US 20170293725A1
• Filed: 03/10/2017
• Published: 10/12/2017
• Est. Priority Date: 04/07/2016
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Liu, Wen, Modi, Ashutosh, Georgescu, Bogdan, Pereira, Francisco
• Granted Patent: US 9,984,772 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 16/24522 Translation of natural language queries to structured queries
• CPC Class Codes: G06F 16/3329 Natural language query formulation or dialogue systems
• CPC Class Codes: G06F 40/211 Syntactic parsing, e.g. based on context-free grammar [CFG] or unification grammars
• CPC Class Codes: G06F 40/30 Semantic analysis
• CPC Class Codes: G06F 40/35 Discourse or dialogue representation
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

A computer-implemented method for predicting answers to questions concerning medical image analytics reports includes splitting a medical image analytics report into a plurality of sentences and generating a plurality of sentence embedding vectors by applying a natural language processing framework to the plurality of sentences. A question related to subject matter included in the medical image analytics report is received and a question embedding vector is generated by applying the natural language processing framework to the question. A subset of the sentence embedding vectors most similar to the question embedding vector is identified by applying a similarity matching process to the sentence embedding vectors and the question embedding vector. A trained recurrent neural network (RNN) is used to determine a predicted answer to the question based on the subset of the sentence embedding vectors.

Claims

1. A computer-implemented method for predicting answers to questions concerning medical image analytics reports, the method comprising:

splitting a medical image analytics report into a plurality of sentences;

generating a plurality of sentence embedding vectors by applying a natural language processing framework to the plurality of sentences;

receiving a question related to subject matter included in the medical image analytics report;

generating a question embedding vector by applying the natural language processing framework to the question;

identifying a subset of the sentence embedding vectors most similar to the question embedding vector by applying a similarity matching process to the sentence embedding vectors and the question embedding vector; and

using a trained recurrent neural network (RNN) to determine a predicted answer to the question based on the subset of the sentence embedding vectors.

2. The method of claim 1, wherein each sentence embedding vector is generated by:

generating an embedding for each word in a particular sentence using the natural language processing framework; and

averaging the embedding of each word in the particular sentence to yield the sentence embedding vector.

3. The method of claim 1, wherein the natural language processing framework identifies target medical syntax by matching the plurality of sentences to one or more ontology references.

4. The method of claim 3, wherein the target medical syntax comprises a description of one or more diseases.

5. The method of claim 3, wherein the target medical syntax comprises a description of one or more anatomical entities.

6. The method of claim 3, wherein the one or more ontology references include a Unified Medical Language System (UMLS) dataset.

7. The method of claim 1, further comprising:

training the natural language processing framework with one or more medical knowledge datasets.

8. The method of claim 7, wherein the one or more medical knowledge datasets include radiology guidelines from the American College of Radiology.

9. The method of claim 1, wherein the similarity matching process comprises:

calculating a cosine product between each sentence embedding vector and the question embedding vector to yield a similarity score for each sentence embedding vector;

ranking the sentence embedding vectors according to similarity score;

selecting a predetermined number of the highest ranking sentence embedding vectors as the subset of the sentence embedding vectors.

10. The method of claim 1, wherein the trained RNN is a long short-term memory (LSTM) RNN and the predicted answer is determined by:

dividing the subset of the sentence embedding vectors into a first sequence of words;

dividing the subset of the question embedding vector into a second sequence of words;

passing the first sequence of words and the second sequence of words through a plurality of LSTM cells sequentially to yield a plurality of outputs corresponding to different states;

combining the plurality of outputs using linear operations into a single input vector; and

applying a softmax function to the single input vector to generate the predicted answer.

11. A computer-implemented method for predicting answers to questions concerning medical image analytics reports, the method comprising:

generating a plurality of sentence embedding vectors by applying a natural language processing framework to sentences in a medical image analytics report;

receiving a question related to subject matter included in the medical image analytics report;

generating a question embedding vector by applying the natural language processing framework to the question;

identifying a subset of the sentence embedding vectors most similar to the question embedding vector;

dividing the subset of the sentence embedding vectors into a first sequence of words;

dividing the subset of the question embedding vector into a second sequence of words;

passing the first sequence of words and the second sequence of words through a plurality of long short-term memory (LSTM) cells sequentially to yield a plurality of outputs corresponding to different states;

combining the plurality of outputs into a single input vector; and

applying a softmax function to the single input vector to generate a predicted answer.

12. The method of claim 11, wherein the subset of the sentence embedding vectors most similar to the question embedding vector are identified by applying a similarity matching process to the sentence embedding vectors and the question embedding vector.

13. The method of claim 11, wherein each sentence embedding vector is generated by:

generating an embedding for each word in a particular sentence using the natural language processing framework; and

averaging the embedding of each word in the particular sentence to yield the sentence embedding vector.

14. The method of claim 11, wherein the natural language processing framework identifies target medical syntax by matching the plurality of sentences to one or more ontology references.

15. The method of claim 14, wherein the target medical syntax comprises a description of one or more diseases.

16. The method of claim 14, wherein the target medical syntax comprises a description of one or more anatomical entities.

17. The method of claim 11, further comprising:

training the natural language processing framework with one or more medical knowledge datasets.

18. The method of claim 17, wherein the one or more medical knowledge datasets include radiology guidelines from the American College of Radiology.

19. The method of claim 11, wherein the similarity matching process comprises:

calculating a cosine product between each sentence embedding vector and the question embedding vector to yield a similarity score for each sentence embedding vector;

ranking the sentence embedding vectors according to similarity score;

selecting a predetermined number of highest ranking sentence embedding vectors as the subset of the sentence embedding vectors.

20. A system for predicting answers to questions concerning medical image analytics reports, the system comprising:

a host computer comprising one or more processor configured to;

generate a plurality of sentence embedding vectors by applying a natural language processing framework to the plurality of sentences included in a medical image analytics report;

receive a question related to subject matter included in the medical image analytics report;

generate a question embedding vector by applying the natural language processing framework to the question;

identify a subset of the sentence embedding vectors most similar to the question embedding vector by applying a similarity matching process to the sentence embedding vectors and the question embedding vector; and

a device computer comprising a graphics processing unit (GPU) configured to use a LSTM-based RNN to determine a predicted answer to the question based on the subset of the sentence embedding vectors, wherein each element included in the subset of the sentence embedding vectors is processed by a LSTM cell in parallel using processing resources of the GPU.