Adversarial and Dual Inverse Deep Learning Networks for Medical Image Analysis

• US 20180225823A1
• Filed: 01/11/2018
• Published: 08/09/2018
• Est. Priority Date: 02/09/2017
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Zhou, Shaohua Kevin, Chen, Mingqing, Xu, Daguang, Xu, Zhoubing, Yang, Dong
• Granted Patent: US 10,636,141 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 2207/10072 Tomographic images
• CPC Class Codes: G06T 2207/10081 Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/10088 Magnetic resonance imaging [MRI]
• CPC Class Codes: G06T 2207/10104 Positron emission tomography [PET]
• CPC Class Codes: G06T 2207/10116 X-ray image
• CPC Class Codes: G06T 2207/10132 Ultrasound image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

Methods and apparatus for automated medical image analysis using deep learning networks are disclosed. In a method of automatically performing a medical image analysis task on a medical image of a patient, a medical image of a patient is received. The medical image is input to a trained deep neural network. An output model that provides a result of a target medical image analysis task on the input medical image is automatically estimated using the trained deep neural network. The trained deep neural network is trained in one of a discriminative adversarial network or a deep image-to-image dual inverse network.

Claims

1. A method for automatically performing a medical image analysis task on a medical image of a patient, comprising:

receiving a medical image of a patient;

inputting the medical image to a trained deep neural network; and

automatically estimating an output model that provides a result of a target medical image analysis task on the input medical image using the trained deep neural network, wherein the trained deep neural network is trained in one of a discriminative adversarial network or a deep image-to-image dual inverse network.

2. The method of claim 1, wherein the trained deep neural network is an estimator network that directly maps the output model from the input medical image and is trained in the discriminative adversarial network, which includes the estimator network and a discriminator network that distinguishes between estimated output models estimated by the estimator network from input training images and real ground-truth output models, conditioned on the input training images.

3. The method of claim 2, wherein the estimator network and the discriminator network of the discriminative adversarial network are trained together based on a set of training samples including input training images and corresponding ground-truth output models to learn parameters for the estimator network and the discriminator network that optimize a minimax objective function over the set of training samples, wherein the minimax objective function includes a first cost term related to classification, by the discriminator network, of the ground truth output models and a second cost term related to classification, by the discriminator network, of the estimated output models estimated by the estimator network from the input training images.

4. The method of claim 3, wherein the minimax objective function further includes a third cost term computed using a cost function that calculates an error between the ground truth output models and the estimated output models estimated by the estimator network from the input training images.

5. The method of claim 4, further comprising:

training the estimator network and the discriminator network to learn parameters to optimize the minimax objective function by repeating the following training operations for a plurality of iterations;

with the parameters of the estimator network fixed, learning the parameters of the discriminator network to optimize the minimax objective function; and

with the parameters of the discriminator network fixed, learning the parameters of the estimator network to optimize the minimax objective function.

6. The method of claim 5, wherein:

learning the parameters of the discriminator network to optimize the minimax objective function comprises adjusting the parameters of the discriminator network to maximize probability scores computed by the discriminator network for the ground truth output models and to minimize probability scores computed by the discriminator for the estimated output models estimated by the estimator network from the input training images over the set of training samples; and

learning the parameters of the estimator network to optimize the minimax objective function comprises adjusting the parameters of the estimator network to minimize the error between the ground truth output models and estimated output models and estimated by the estimator network and to maximize the probability scores computed by the discriminator for the estimated models estimated by the estimator network over the set of training samples.

7. The method of claim 1, wherein the trained deep neural network is a first deep image-to-image network trained in the deep image-to-image dual inverse network and automatically estimating an output model that provides a result of a target medical image analysis task on the input medical image using the trained deep neural network comprises:

automatically generating an output image that provides a result of the target medical image analysis task on the input medical image using the first deep image-to-image network.

8. The method of claim 7, wherein the deep image-to-image dual inverse network includes the first deep image-to-image network trained to perform the target medical image analysis task and including an encoder that coverts an input medical image to a feature representation and a decoder that generates a predicted output image from the feature representation of the input medical image, and a second deep image-to-image network trained to perform an inverse task to the target medical image analysis task and including an encoder that coverts an output image for the target medical image analysis task to a feature representation and a decoder that generates a predicted input image from the feature representation of the output image.

9. The method of claim 8, wherein the first deep image-to-image network and the second deep image-to-image network are trained together based on a set of training samples including ground truth input images and corresponding ground truth output images to minimize a cost function including a first loss function that calculates an error between the ground truth output images and the predicted output images generated by the first deep image-to-image network from the ground truth input images, a second loss function that calculates an error between the ground truth output images and the predicted input images generated by the second deep image-to-image network from the ground truth output images, and a third loss function that calculates and error between the feature representation of the ground truth input images generated by the encoder of the first deep image-to-image network and the feature representation of the ground truth output images generated by the encoder of the second deep image-to-image network.

10. The method of claim 9, further comprising:

training the first deep image-to-image network and the second deep image-to-image network together to minimize the cost function by repeating the following training operations for a plurality of iterations;

with parameters of the second deep image-to-image network fixed, learning parameters of the first deep image-to-image network to minimize the first loss function and the third loss function; and

with the parameters of the first deep image-to-image network fixed, learning the parameters of the second deep image-to-image network to minimize the second loss function and the third loss function.

11. An apparatus for automatically performing a medical image analysis task on a medical image of a patient, comprising:

means for receiving a medical image of a patient;

means for inputting the medical image to a trained deep neural network; and

means for automatically estimating an output model that provides a result of a target medical image analysis task on the input medical image using the trained deep neural network, wherein the trained deep neural network is trained in one of a discriminative adversarial network or a deep image-to-image dual inverse network.

12. The apparatus of claim 11, wherein the trained deep neural network is an estimator network that directly maps the output model from the input medical image and is trained in the discriminative adversarial network, which includes the estimator network and a discriminator network that distinguishes between estimated output models estimated by the estimator network from input training images and real ground-truth output models, conditioned on the input training images.

13. The apparatus of claim 11, wherein the trained deep neural network is a first deep image-to-image network trained in the deep image-to-image dual inverse network and the means for automatically estimating an output model that provides a result of a target medical image analysis task on the input medical image using the trained deep neural network comprises:

means for automatically generating an output image that provides a result of the target medical image analysis task on the input medical image using the first deep image-to-image network.

14. The apparatus of claim 13, wherein the deep image-to-image dual inverse network includes the first deep image-to-image network trained to perform the target medical image analysis task and including an encoder that coverts an input medical image to a feature representation and a decoder that generates a predicted output image from the feature representation of the input medical image, and a second deep image-to-image network trained to perform an inverse task to the target medical image analysis task and including an encoder that coverts an output image for the target medical image analysis task to a feature representation and a decoder that generates a predicted input image from the feature representation of the output image.

15. A non-transitory computer readable medium storing computer program instructions for automatically performing a medical image analysis task on a medical image of a patient, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

receiving a medical image of a patient;

inputting the medical image to a trained deep neural network; and

automatically estimating an output model that provides a result of a target medical image analysis task on the input medical image using the trained deep neural network, wherein the trained deep neural network is trained in one of a discriminative adversarial network or a deep image-to-image dual inverse network.

16. The non-transitory computer readable medium of claim 15, wherein the trained deep neural network is an estimator network that directly maps the output model from the input medical image and is trained in the discriminative adversarial network, which includes the estimator network and a discriminator network that distinguishes between estimated output models estimated by the estimator network from input training images and real ground-truth output models, conditioned on the input training images.

17. The non-transitory computer readable medium of claim 16, wherein the estimator network and the discriminator network of the discriminative adversarial network are trained together based on a set of training samples including input training images and corresponding ground-truth output models to learn parameters for the estimator network and the discriminator network that optimize a minimax objective function over the set of training samples, wherein the minimax objective function includes a first cost term related to classification, by the discriminator network, of the ground truth output models and a second cost term related to classification, by the discriminator network, of the estimated output models estimated by the estimator network from the input training images, and a third cost term computed using a cost function that calculates an error between the ground truth output models and the estimated output models estimated by the estimator network from the input training images.

18. The non-transitory computer readable medium of claim 17, wherein the operations further comprise:

training the estimator network and the discriminator network to learn parameters to optimize the minimax objective function by repeating the following training operations for a plurality of iterations;

with the parameters of the estimator network fixed, learning the parameters of the discriminator network to optimize the minimax objective function; and

with the parameters of the discriminator network fixed, learning the parameters of the estimator network to optimize the minimax objective function.

19. The non-transitory computer readable medium of claim 15, wherein the trained deep neural network is a first deep image-to-image network trained in the deep image-to-image dual inverse network and automatically estimating an output model that provides a result of a target medical image analysis task on the input medical image using the trained deep neural network comprises:

automatically generating an output image that provides a result of the target medical image analysis task on the input medical image using the first deep image-to-image network.

20. The non-transitory computer readable medium of claim 19, wherein the deep image-to-image dual inverse network includes the first deep image-to-image network trained to perform the target medical image analysis task and including an encoder that coverts an input medical image to a feature representation and a decoder that generates a predicted output image from the feature representation of the input medical image, and a second deep image-to-image network trained to perform an inverse task to the target medical image analysis task and including an encoder that coverts an output image for the target medical image analysis task to a feature representation and a decoder that generates a predicted input image from the feature representation of the output image.

21. The non-transitory computer readable medium of claim 20, wherein the first deep image-to-image network and the second deep image-to-image network are trained together based on a set of training samples including ground truth input images and corresponding ground truth output images to minimize a cost function including a first loss function that calculates an error between the ground truth output images and the predicted output images generated by the first deep image-to-image network from the ground truth input images, a second loss function that calculates an error between the ground truth output images and the predicted input images generated by the second deep image-to-image network from the ground truth output images, and a third loss function that calculates and error between the feature representation of the ground truth input images generated by the encoder of the first deep image-to-image network and the feature representation of the ground truth output images generated by the encoder of the second deep image-to-image network.

22. The non-transitory computer readable medium of claim 21, wherein the operations further comprise:

training the first deep image-to-image network and the second deep image-to-image network together to minimize the cost function by repeating the following training operations for a plurality of iterations;

with parameters of the second deep image-to-image network fixed, learning parameters of the first deep image-to-image network to minimize the first loss function and the third loss function; and

with the parameters of the first deep image-to-image network fixed, learning the parameters of the second deep image-to-image network to minimize the second loss function and the third loss function.