Adversarial and dual inverse deep learning networks for medical image analysis

• US 10,636,141 B2
• Filed: 01/11/2018
• Issued: 04/28/2020
• Est. Priority Date: 02/09/2017
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Zhou, Shaohua Kevin, Chen, Mingqing, Xu, Daguang, Xu, Zhoubing, Yang, Dong
• Primary Examiner(s): Liu, Li
• Application Number:
  US15/868,062
• Publication Number:
  US 20180225823A1
• Time in Patent Office: 838 Days
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 2207/10072 Tomographic images
• CPC Class Codes: G06T 2207/10081 Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/10088 Magnetic resonance imaging [MRI]
• CPC Class Codes: G06T 2207/10104 Positron emission tomography [PET]
• CPC Class Codes: G06T 2207/10116 X-ray image
• CPC Class Codes: G06T 2207/10132 Ultrasound image
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 30/19173 Classification techniques

Abstract

Methods and apparatus for automated medical image analysis using deep learning networks are disclosed. In a method of automatically performing a medical image analysis task on a medical image of a patient, a medical image of a patient is received. The medical image is input to a trained deep neural network. An output model that provides a result of a target medical image analysis task on the input medical image is automatically estimated using the trained deep neural network. The trained deep neural network is trained in one of a discriminative adversarial network or a deep image-to-image dual inverse network.

Claims

1. A method for automatically performing a medical image analysis task on a medical image of a patient, comprising:

receiving a medical image of a patient;

inputting the medical image to a trained deep neural network; and

automatically estimating an output model that provides a result of a target medical image analysis task on the input medical image using the trained deep neural network, wherein the trained deep neural network is trained in a discriminative adversarial network based on a minimax objective function comprising

     1) a first cost term related to classification, by a discriminator network of the discriminative adversarial network, of ground truth output models,

     2) a second cost term related to classification, by the discriminator network, of estimated output models estimated by an estimator network of the discriminative adversarial network from input training images, and

     3) a third cost term computed using a cost function that calculates an error between the ground truth output models and the estimated output models.

2. The method of claim 1, wherein the trained deep neural network is the estimator network that directly maps the output model from the input medical image.

3. The method of claim 2, wherein the estimator network and the discriminator network of the discriminative adversarial network are trained together based on a set of training samples including the input training images and the ground truth output models to learn parameters for the estimator network and the discriminator network that optimize the minimax objective function over the set of training samples.

4. The method of claim 1, further comprising:

training the estimator network and the discriminator network to learn parameters to optimize the minimax objective function by repeating the following training operations for a plurality of iterations;

with the parameters of the estimator network fixed, learning the parameters of the discriminator network to optimize the minimax objective function; and

with the parameters of the discriminator network fixed, learning the parameters of the estimator network to optimize the minimax objective function.

5. The method of claim 4, wherein:

learning the parameters of the discriminator network to optimize the minimax objective function comprises adjusting the parameters of the discriminator network to maximize probability scores computed by the discriminator network for the ground truth output models and to minimize probability scores computed by the discriminator for the estimated output models estimated by the estimator network from the input training images over a set of training samples; and

learning the parameters of the estimator network to optimize the minimax objective function comprises adjusting the parameters of the estimator network to minimize the error between the ground truth output models and the estimated output models estimated by the estimator network and to maximize the probability scores computed by the discriminator for the estimated models estimated by the estimator network over the set of training samples.

6. An apparatus for automatically performing a medical image analysis task on a medical image of a patient, comprising:

means for receiving a medical image of a patient;

means for inputting the medical image to a trained deep neural network; and

means for automatically estimating an output model that provides a result of a target medical image analysis task on the input medical image using the trained deep neural network, wherein the trained deep neural network is trained in a discriminative adversarial network based on a minimax objective function comprising

     1) a first cost term related to classification, by a discriminator network of the discriminative adversarial network, of ground truth output models,

     2) a second cost term related to classification, by the discriminator network, of estimated output models estimated by an estimator network of the discriminative adversarial network from input training images, and

     3) a third cost term computed using a cost function that calculates an error between the ground truth output models and the estimated output models.

7. The apparatus of claim 6, wherein the trained deep neural network is the estimator network that directly maps the output model from the input medical image.

8. The apparatus of claim 7, wherein the estimator network and the discriminator network of the discriminative adversarial network are trained together based on a set of training samples including the input training images and the ground truth output models to learn parameters for the estimator network and the discriminator network that optimize the minimax objective function over the set of training samples.

9. The apparatus of claim 6, further comprising:

training the estimator network and the discriminator network to learn parameters to optimize the minimax objective function by repeating the following training operations for a plurality of iterations;

with the parameters of the estimator network fixed, learning the parameters of the discriminator network to optimize the minimax objective function; and

with the parameters of the discriminator network fixed, learning the parameters of the estimator network to optimize the minimax objective function.

10. The apparatus of claim 9, wherein:

learning the parameters of the discriminator network to optimize the minimax objective function comprises adjusting the parameters of the discriminator network to maximize probability scores computed by the discriminator network for the ground truth output models and to minimize probability scores computed by the discriminator for the estimated output models estimated by the estimator network from the input training images over a set of training samples; and

learning the parameters of the estimator network to optimize the minimax objective function comprises adjusting the parameters of the estimator network to minimize the error between the ground truth output models and the estimated output models estimated by the estimator network and to maximize the probability scores computed by the discriminator for the estimated models estimated by the estimator network over the set of training samples.

11. A non-transitory computer readable medium storing computer program instructions for automatically performing a medical image analysis task on a medical image of a patient, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

receiving a medical image of a patient;

inputting the medical image to a trained deep neural network; and

automatically estimating an output model that provides a result of a target medical image analysis task on the input medical image using the trained deep neural network, wherein the trained deep neural network is trained in a discriminative adversarial network based on a minimax objective function comprising

     1) a first cost term related to classification, by a discriminator network of the discriminative adversarial network, of ground truth output models,

     2) a second cost term related to classification, by the discriminator network, of estimated output models estimated by an estimator network of the discriminative adversarial network from input training images, and

     3) a third cost term computed using a cost function that calculates an error between the ground truth output models and the estimated output models.

12. The non-transitory computer readable medium of claim 11, wherein the trained deep neural network is the estimator network that directly maps the output model from the input medical image.

13. The non-transitory computer readable medium of claim 12, wherein the estimator network and the discriminator network of the discriminative adversarial network are trained together based on a set of training samples including the input training images and the ground truth output models to learn parameters for the estimator network and the discriminator network that optimize the minimax objective function over the set of training samples.

14. The non-transitory computer readable medium of claim 11, wherein the operations further comprise:

training the estimator network and the discriminator network to learn parameters to optimize the minimax objective function by repeating the following training operations for a plurality of iterations;

with the parameters of the estimator network fixed, learning the parameters of the discriminator network to optimize the minimax objective function; and

with the parameters of the discriminator network fixed, learning the parameters of the estimator network to optimize the minimax objective function.

15. The non-transitory computer readable medium of claim 14, wherein:

learning the parameters of the discriminator network to optimize the minimax objective function comprises adjusting the parameters of the discriminator network to maximize probability scores computed by the discriminator network for the ground truth output models and to minimize probability scores computed by the discriminator for the estimated output models estimated by the estimator network from the input training images over a set of training samples; and

learning the parameters of the estimator network to optimize the minimax objective function comprises adjusting the parameters of the estimator network to minimize the error between the ground truth output models and the estimated output models estimated by the estimator network and to maximize the probability scores computed by the discriminator for the estimated models estimated by the estimator network over the set of training samples.