Generative adversarial network medical image generation for training of a classifier

• US 10,592,779 B2
• Filed: 12/21/2017
• Issued: 03/17/2020
• Est. Priority Date: 12/21/2017
• Status: Active Grant

• Current Assignee: Merative US LP
• Original Assignee: International Business Machines Corporation
• Inventors: Madani, Ali, Moradi, Mehdi, Syeda-Mahmood, Tanveer F.
• Primary Examiner(s): Osifade, Idowu O
• Application Number:
  US15/850,007
• Publication Number:
  US 20190197358A1
• Time in Patent Office: 817 Days
• Field of Search: None
• CPC Class Codes: A61B 6/5217 extracting a diagnostic or physiological parameter from medical diagnostic data
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 5/022 Knowledge engineering; Knowledge acquisition
• CPC Class Codes: G06T 2207/10116 X-ray image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30048 Heart; Cardiac
• CPC Class Codes: G06T 2207/30096 Tumor; Lesion
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/7753 Incorporation of unlabelled data, e.g. multiple instance learning [MIL]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems

Abstract

Mechanisms are provided to implement a machine learning training model. The machine learning training model trains an image generator of a generative adversarial network (GAN) to generate medical images approximating actual medical images. The machine learning training model augments a set of training medical images to include one or more generated medical images generated by the image generator of the GAN. The machine learning training model trains a machine learning model based on the augmented set of training medical images to identify anomalies in medical images. The trained machine learning model is applied to new medical image inputs to classify the medical images as having an anomaly or not.

Claims

1. A method, in a data processing system comprising a processor and a memory, the memory comprising instructions that are executed by the processor to configure the processor to implement a machine learning training model, the method comprising:

training, by the machine learning training model, an image generator of a generative adversarial network (GAN) to generate medical images approximating actual medical images;

augmenting, by the machine learning training model, a set of training medical images to include one or more generated medical images generated by the image generator of the GAN;

training, by the machine learning training model, a machine learning model based on the augmented set of training medical images to identify anomalies in medical images; and

applying the trained machine learning model to new medical image inputs to classify the medical images as having an anomaly or not, wherein the machine learning model is a discriminator of the GAN which is configured to receive, as input, actual labeled medical image data, actual unlabeled medical image data, and generated medical image data generated by the image generator of the GAN, and wherein training the machine learning model comprises training the discriminator to generate an output comprising an output value for each of a plurality of classifications, and wherein the plurality of classifications comprises a first classification for real-normal image data indicating input image data to be actual image data representing a normal medical condition, at least one second classification for real-abnormal image data indicating input image data to be actual image data representing a corresponding abnormal medical condition, and a third classification for generated image data indicating input image data to be image data generated by the image generator.

2. The method of claim 1, wherein training a generator of the GAN to generate medical images approximating actual medical images, further comprises:

receiving, by the image generator of the GAN, at least one medical image showing an anatomical structure;

inputting, by a noise generator, a noise input to the image generator;

generating, by the image generator, at least one generated image based on a combination of the at least one medical image and the noise input;

providing the at least one medical image and the at least one generated image to the discriminator of the machine learning training image;

analyzing, by the discriminator, the at least one medical image and the at least one generated image to label the at least one medical image and the at least one generated image as either being an actual medical image or a generated medical image; and

modifying an operational parameter of the generator based on the results of analyzing the at least one medical image and the at least one generated image by the discriminator.

3. The method of claim 1, wherein the GAN is trained to classify medical images with regard to whether or not the medical images contain an abnormality.

4. The method of claim 1, wherein the discriminator is trained, using an adversarial training technique based on the augmented set of training medical images, to discriminate between medical images showing anomalies and medical images showing normal medical conditions.

5. The method of claim 1, wherein training the image generator comprises training the image generator based on both labeled training medical images and non-labeled medical images, and wherein the training is based on a feedback obtained from the output of the discriminator.

6. The method of claim 1, wherein training, by the machine learning training model, the image generator to generate medical images approximating actual medical images comprises:

training the discriminator of the GAN to separate generated medical images from actual medical images and actual medical images showing a normal medical condition from actual medical images showing an abnormal medical condition; and

training the image generator to generate medical images that cause the discriminator to fail to correctly discriminate the generated medical images from actual medical images for a predetermined ratio of instances indicating convergence of the training of the GAN.

7. The method of claim 1, wherein the discriminator of the GAN is a convolutional neural network with K+1 output nodes, where K is a number of classes of abnormalities that the machine learning model is configured to identify in input medical images and the K+1 class is a class indicating an input medical image to be a generated medical image from the image generator, and wherein the discriminator outputs, for each input medical image, a corresponding vector output having K+1 values indicating into which class the discriminator classifies the input medical image.

8. The method of claim 1, wherein the discriminator is a convolutional neural network with 2K output nodes, where K is a number of classes of abnormalities that the machine learning model is configured to identify in input medical images and for each class in the K number of classes, there is a separate output node indicating whether the input medical image is a generated medical image from the image generator, or an actual medical image, and wherein the discriminator outputs, for each input medical image, a corresponding vector output having 2K values indicating into which class the discriminator classifies the input medical image.

9. The method of claim 1, further comprising:

configuring a loss function of the discriminator of the GAN to include error components for each of actual labeled medical images, actual unlabeled medical images, and generated medical images, wherein training the image generator of the GAN to generate medical images approximating actual medical images comprises minimizing the loss function of the discriminator.

10. The method of claim 1, wherein the generator is one of a plurality of generators, and wherein at least one generator in the plurality of generators is trained to generate medical images approximating actual medical images having a normal medical condition depicted, and wherein at least one other generator in the plurality of generators is trained to generate medical images approximating actual medical images having an abnormal medical condition depicted.

11. A computer program product comprising a computer readable storage medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to implement a machine learning training model that operates to:

train an image generator of a generative adversarial network (GAN) to generate medical images approximating actual medical images;

augment a set of training medical images to include one or more generated medical images generated by the image generator of the GAN;

train a machine learning model based on the augmented set of training medical images to identify anomalies in medical images; and

apply the trained machine learning model to new medical image inputs to classify the medical images as having an anomaly or not, wherein the machine learning model is a discriminator of the GAN which is configured to receive, as input, actual labeled medical image data, actual unlabeled medical image data, and generated medical image data generated by the image generator of the GAN, and wherein training the machine learning model comprises training the discriminator to generate an output comprising an output value for each of a plurality of classifications, and wherein the plurality of classifications comprises a first classification for real-normal image data indicating input image data to be actual image data representing a normal medical condition, at least one second classification for real-abnormal image data indicating input image data to be actual image data representing a corresponding abnormal medical condition, and a third classification for generated image data indicating input image data to be image data generated by the image generator.

12. The computer program product of claim 11, wherein the computer readable program further causes the machine learning training model to train the generator of the GAN to generate medical images approximating actual medical images, at least by:

receiving, by the image generator of the GAN, at least one medical image showing an anatomical structure;

inputting, by a noise generator, a noise input to the image generator;

generating, by the image generator, at least one generated image based on a combination of the at least one medical image and the noise input;

providing the at least one medical image and the at least one generated image to the discriminator of the machine learning training image;

analyzing, by the discriminator, the at least one medical image and the at least one generated image to label the at least one medical image and the at least one generated image as either being an actual medical image or a generated medical image; and

modifying an operational parameter of the generator based on the results of analyzing the at least one medical image and the at least one generated image by the discriminator.

13. The computer program product of claim 11, wherein the GAN is trained to classify medical images with regard to whether or not the medical images contain an abnormality.

14. The computer program product of claim 11, wherein the discriminator is trained, using an adversarial training technique based on the augmented set of training medical images, to discriminate between medical images showing anomalies and medical images showing normal medical conditions.

15. The computer program product of claim 11, wherein the computer readable program further causes the machine learning training model to train the image generator at least by training the image generator based on both labeled training medical images and non-labeled medical images, and wherein the training is based on a feedback obtained from the output of the discriminator.

16. The computer program product of claim 11, wherein the computer readable program further causes the machine learning training model to train the image generator to generate medical images approximating actual medical images at least by:

training the discriminator of the GAN to separate generated medical images from actual medical images and actual medical images showing a normal medical condition from actual medical images showing an abnormal medical condition; and

training the image generator to generate medical images that cause the discriminator to fail to correctly discriminate the generated medical images from actual medical images for a predetermined ratio of instances indicating convergence of the training of the GAN.

17. The computer program product of claim 11, wherein the discriminator of the GAN is a convolutional neural network with K+1 output nodes, where K is a number of classes of abnormalities that the machine learning model is configured to identify in input medical images and the K+1 class is a class indicating an input medical image to be a generated medical image from the image generator, and wherein the discriminator outputs, for each input medical image, a corresponding vector output having K+1 values indicating into which class the discriminator classifies the input medical image.

18. The computer program product of claim 11, wherein the discriminator is a convolutional neural network with 2K output nodes, where K is a number of classes of abnormalities that the machine learning model is configured to identify in input medical images and for each class in the K number of classes, there is a separate output node indicating whether the input medical image is a generated medical image from the image generator, or an actual medical image, and wherein the discriminator outputs, for each input medical image, a corresponding vector output having 2K values indicating into which class the discriminator classifies the input medical image.

19. The computer program product of claim 11, wherein the computer readable program further causes the machine learning training model to:

configure a loss function of the discriminator of the GAN to include error components for each of actual labeled medical images, actual unlabeled medical images, and generated medical images, wherein training the image generator of the GAN to generate medical images approximating actual medical images comprises minimizing the loss function of the discriminator.

20. An apparatus comprising:

at least one processor; and

at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to implement a machine learning training model that operates to;

train an image generator of a generative adversarial network (GAN) to generate medical images approximating actual medical images;

augment a set of training medical images to include one or more generated medical images generated by the image generator of the GAN;

train a machine learning model based on the augmented set of training medical images to identify anomalies in medical images; and

apply the trained machine learning model to new medical image inputs to classify the medical images as having an anomaly or not, wherein the machine learning model is a discriminator of the GAN which is configured to receive, as input, actual labeled medical image data, actual unlabeled medical image data, and generated medical image data generated by the image generator of the GAN, and wherein training the machine learning model comprises training the discriminator to generate an output comprising an output value for each of a plurality of classifications, and wherein the plurality of classifications comprises a first classification for real-normal image data indicating input image data to be actual image data representing a normal medical condition, at least one second classification for real-abnormal image data indicating input image data to be actual image data representing a corresponding abnormal medical condition, and a third classification for generated image data indicating input image data to be image data generated by the image generator.