Adapting a generative adversarial network to new data sources for image classification

• US 10,540,578 B2
• Filed: 12/21/2017
• Issued: 01/21/2020
• Est. Priority Date: 12/21/2017
• Status: Active Grant

• Current Assignee: Merative US LP
• Original Assignee: International Business Machines Corporation
• Inventors: Madani, Ali, Moradi, Mehdi, Syeda-Mahmood, Tanveer F.
• Primary Examiner(s): Alavi, Amir
• Application Number:
  US15/850,193
• Publication Number:
  US 20190197368A1
• Time in Patent Office: 761 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/047 Probabilistic or stochastic networks
• CPC Class Codes: G06N 3/048 Activation functions
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/774 Generating sets of training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G16H 30/20 for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients

Abstract

Mechanisms are provided to implement a generative adversarial network (GAN) that is trained based on labeled image data, unlabeled image data, and generated image data generated by a generator of the GAN. The GAN comprises a loss function that comprises error components for each of the labeled image data, unlabeled image data, and generated image data which is used to train the GAN. A new data source for which the trained GAN is to be adapted is identified and the trained GAN is adapted for the new data source. Image data in the new data source is classified by applying the adapted GAN to the data in the new data source. Adapting the trained GAN includes obtaining a minimized set of labeled images and utilizing the minimized set of images to perform the adapting of the trained GAN.

Claims

1. A method, in a data processing system comprising a processor and a memory, for re-training a classification engine for a new data source, the memory comprising instructions that are executed by the processor to configure the processor to implement a generative adversarial network (GAN), the method comprising:

training the GAN based on labeled image data, unlabeled image data, and generated image data generated by a generator of the GAN, wherein the GAN comprises a loss function that comprises error components for each of the labeled image data, unlabeled image data, and generated image data which is used to train the GAN;

identifying the new data source for which the trained GAN is to be adapted;

adapting the trained GAN for the new data source; and

classifying image data in the new data source by applying the adapted GAN to the data in the new data source, wherein adapting the trained GAN comprises obtaining a minimized set of labeled images and utilizing the minimized set of images to perform the adapting of the trained GAN.

2. The method of claim 1, wherein adapting the trained GAN for the new data source comprises re-training the trained GAN based on the minimized set of labeled images.

3. The method of claim 1, wherein the minimized set of labeled images is obtained from a known trusted third party source computing system providing labeled image data, and wherein the minimized set of labeled images are labeled images in a same domain as unlabeled images provided by the new data source.

4. The method of claim 3, wherein adapting the trained GAN for the new data source comprises re-training the trained GAN based on the minimized set of labeled images from the known trusted third party source computing system, and the unlabeled images in the image data in the new data source.

5. The method of claim 1, wherein the minimized set of images is a first subset of the image data in the new data source, and wherein the minimized set of images is smaller than a second subset of the image data in the new data source that consists of unlabeled image data.

6. The method of claim 5, wherein adapting the trained GAN for the new data source comprises re-training the trained GAN based on the first subset and second subset of image data in the new data source using an adversarial training technique between a generator of the trained GAN and a discriminator of the trained GAN.

7. The method of claim 1, wherein the GAN is trained to classify medical images with regard to whether or not the medical images contain an abnormality.

8. The method of claim 1, wherein training the GAN based on labeled image data, unlabeled image data, and generated image data generated by the generator of the GAN comprises:

training a discriminator of the GAN to separate generated medical images from actual medical images and actual medical images showing a normal medical condition from actual medical images showing an abnormal medical condition; and

training the generator to generate medical images that cause the discriminator to fail to correctly discriminate the generated medical images from actual medical images for a predetermined ratio of instances indicating convergence of the training of the GAN.

9. The method of claim 1, wherein training the GAN based on labeled image data, unlabeled image data, and generated image data generated by the generator of the GAN comprises training a discriminator of the GAN to classify medical images into corresponding ones of a first class indicating an image representing a normal medical condition, one or more second classes indicating one or more abnormal medical conditions, and a third class indicating a generated image.

10. The method of claim 9, wherein training the discriminator further comprises:

for unlabeled images, training the discriminator to correctly identify each of the unlabeled images as either a generated image or an actual image regardless of whether or not the unlabeled image shows an abnormality or a normal medical condition;

for labeled images, training the discriminator to correctly identify each of the labeled images as either an image showing a normal medical condition or an image showing an abnormality, and further correctly identifying each of the labeled images as either being a generated image or an actual image; and

for generated images, training the discriminator to correctly identify each generated image as being a generated image and not an actual image.

11. A computer program product comprising a non-transitory computer readable medium having a computer readable program stored therein, wherein the computer readable program, when executed on a data processing system, causes the data processing system to implement a generative adversarial network (GAN) and to:

train the GAN based on labeled image data, unlabeled image data, and generated image data generated by a generator of the GAN, wherein the GAN comprises a loss function that comprises error components for each of the labeled image data, unlabeled image data, and generated image data which is used to train the GAN;

identify a new data source for which the trained GAN is to be adapted;

adapt the trained GAN for the new data source; and

classify image data in the new data source by applying the adapted GAN to the data in the new data source, wherein adapting the trained GAN comprises obtaining a minimized set of labeled images and utilizing the minimized set of images to perform the adapting of the trained GAN.

12. The computer program product of claim 11, wherein the computer readable program further causes the data processing system to adapt the trained GAN for the new data source at least by re-training the trained GAN based on the minimized set of labeled images.

13. The computer program product of claim 11, wherein the minimized set of labeled images is obtained from a known trusted third party source computing system providing labeled image data, and wherein the minimized set of labeled images are labeled images in a same domain as unlabeled images provided by the new data source.

14. The computer program product of claim 13, wherein the computer readable program further causes the data processing system to adapt the trained GAN for the new data source at least by re-training the trained GAN based on the minimized set of labeled images from the known trusted third party source computing system, and the unlabeled images in the image data in the new data source.

15. The computer program product of claim 11, wherein the minimized set of images is a first subset of the image data in the new data source, and wherein the minimized set of images is smaller than a second subset of the image data in the new data source that consists of unlabeled image data.

16. The computer program product of claim 15, wherein the computer readable program further causes the data processing system to adapt the trained GAN for the new data source at least by re-training the trained GAN based on the first subset and second subset of image data in the new data source using an adversarial training technique between a generator of the trained GAN and a discriminator of the trained GAN.

17. The computer program product of claim 11, wherein the GAN is trained to classify medical images with regard to whether or not the medical images contain an abnormality.

18. The computer program product of claim 11, wherein the computer readable program further causes the data processing system to train the GAN based on labeled image data, unlabeled image data, and generated image data generated by the generator of the GAN at least by:

training a discriminator of the GAN to separate generated medical images from actual medical images and actual medical images showing a normal medical condition from actual medical images showing an abnormal medical condition; and

training the generator to generate medical images that cause the discriminator to fail to correctly discriminate the generated medical images from actual medical images for a predetermined ratio of instances indicating convergence of the training of the GAN.

19. The computer program product of claim 11, wherein the computer readable program further causes the data processing system to train the GAN based on labeled image data, unlabeled image data, and generated image data generated by the generator of the GAN at least by training a discriminator of the GAN to classify medical images into corresponding ones of a first class indicating an image representing a normal medical condition, one or more second classes indicating one or more abnormal medical conditions, and a third class indicating a generated image.

20. An apparatus comprising:

at least one processor; and

at least one memory coupled to the at least one processor, wherein the at least one memory comprises instructions which, when executed by the at least one processor, cause the at least one processor to implement a generative adversarial network (GAN) and to;

train the GAN based on labeled image data, unlabeled image data, and generated image data generated by a generator of the GAN, wherein the GAN comprises a loss function that comprises error components for each of the labeled image data, unlabeled image data, and generated image data which is used to train the GAN;

identify a new data source for which the trained GAN is to be adapted;

adapt the trained GAN for the new data source; and

classify image data in the new data source by applying the adapted GAN to the data in the new data source, wherein adapting the trained GAN comprises obtaining a minimized set of labeled images and utilizing the minimized set of images to perform the adapting of the trained GAN.