Deep Image-to-Image Network Learning for Medical Image Analysis

• US 20170277981A1
• Filed: 06/09/2017
• Published: 09/28/2017
• Est. Priority Date: 01/08/2016
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Zhou, S. Kevin, Comaniciu, Dorin, Georgescu, Bogdan, Zheng, Yefeng, Liu, David, Xu, Daguang
• Granted Patent: US 10,062,014 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/295 Markov models or related models, e.g. semi-Markov models; Markov random fields; Networks embedding Markov models
• CPC Class Codes: G06T 2207/10072 Tomographic images
• CPC Class Codes: G06T 2207/10136 3D ultrasound image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 2207/30056 Liver; Hepatic
• CPC Class Codes: G06T 2207/30096 Tumor; Lesion
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06T 7/174 involving the use of two or more images
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning

Abstract

A method and apparatus for automatically performing medical image analysis tasks using deep image-to-image network (DI2IN) learning. An input medical image of a patient is received. An output image that provides a result of a target medical image analysis task on the input medical image is automatically generated using a trained deep image-to-image network (DI2IN). The trained DI2IN uses a conditional random field (CRF) energy function to estimate the output image based on the input medical image and uses a trained deep learning network to model unary and pairwise terms of the CRF energy function. The DI2IN may be trained on an image with multiple resolutions. The input image may be split into multiple parts and a separate DI2IN may be trained for each part. Furthermore, the multi-scale and multi-part schemes can be combined to train a multi-scale multi-part DI2IN.

Claims

1. A method for automatically performing a medical image analysis task on a medical image of a patient, comprising:

receiving an input medical image of a patient; and

automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN), wherein the DI2IN uses a conditional random field (CRF) energy function to estimate the output image based on the input medical image and uses a trained deep learning network to model unary and pairwise terms of the CRF energy function.

2. The method of claim 1, wherein the target medical image analysis task is detection of one or more anatomical landmarks in the input medical image, and the estimated output image is one of a mask image in which only locations of the one or more anatomical landmarks have non-zero pixel or voxel values or an image with a Gaussian-like circle defined surrounding locations of the one or more anatomical landmarks.

3. The method of claim 1, wherein the target medical image analysis task is detection of an anatomy of interest in the input medical image, and the estimated output image is a mask image in which only pixels or voxels located within a bounding box of the anatomy of interest have non-zero values.

4. The method of claim 1, wherein the target medical image analysis task is segmentation of one or more anatomies of interest in the input medical image, and the estimated output image is one of a mask image in which only pixels or voxels located within boundaries of the one or more anatomies of interest have non-zero values or an image with a Gaussian-like band defined surrounding boundaries of the one or more anatomies of interest.

5. The method of claim 1, wherein the target medical image analysis task is lesion detection, segmentation, and characterization, and the estimated output image is a multi-label mask image in which only pixels or voxels within lesion boundaries of one or more lesions have non-zero values assigned to each of the one or more lesions corresponding to a lesion type for each lesion.

6. The method of claim 1, wherein the target medical image analysis task is an image denoising task, and the estimated output image is a reduced noise image of the input medical image.

7. The method of claim 1, wherein the input medical image is a medical image in a source domain, the target medical image analysis task is cross-domain image synthesis, and the estimated output image is a synthesized medical image in a target domain corresponding to the input medical image.

8. The method of claim 1, wherein receiving the input medical image includes receiving the input medical image in a pair of input medical images acquired using different imaging modalities, the target medical image analysis task is registration of the pair of input medical images, and the estimated output image is a deformation field that provides the registration between the pair of input medical images.

9. The method of claim 1, wherein receiving the input medical image includes receiving the input medical image in a set of input medical images, the target medical image analysis task is a quantitative parametric mapping task, and the estimated output image is a set of quantitative parameters that generate the set of input medical images given a pointwise generative model.

10. The method of claim 1, further comprising, in a training stage prior to receiving the input medical image of the patient:

defining a type of output image that provides the result of the target medical image analysis task;

receiving a plurality of input training images;

receiving or generating corresponding output training images for the plurality of input training images, resulting in a training set of paired input and output training images; and

training the DI2IN by learning weight parameters of a deep learning network that models the unary and pairwise terms of the CRF energy function that result in a maximum likelihood for paired input and output training images over the training set of paired input and output training images.

11. The method of claim 1, wherein automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN) comprises:

estimating an output image that maximizes a likelihood of the CRF energy function given the input medical image and a set of learned weight parameters of the trained deep learning network, wherein the trained deep learning network calculates the unary and pairwise terms of the CRF energy function based on the input medical image, the estimated output image, and the set of learned weight parameters of the trained deep learning network.

12. The method of claim 1, wherein automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN) comprises:

generating an image pyramid with a plurality of reduced resolution images of the input medical image;

generating a respective output image that provides a result of the target medical image analysis task on each of the reduced resolution images of the input medical image using a sequence of trained DI2INs.

13. The method of claim 12, wherein the respective output image generated for each of the plurality of reduced resolution images of the input medical image defines a region of interest that is cropped from an image at a subsequent higher resolution of the input medical image.

14. The method of claim 1, wherein automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN) comprises:

dividing the input medical image into a plurality of parts;

automatically generating a respective output image that provides a result of the target medical image analysis task on each of the plurality of parts of the input medical image using a respective trained DI2IN for each of the plurality of parts; and

aggregating the output images that provide the results of the target medical image analysis task on each of the plurality of parts to generate a final output image that provides the result of the target medical image analysis task on the input medical image.

15. An apparatus for automatically performing a medical image analysis task on a medical image of a patient, comprising:

means for receiving an input medical image of a patient; and

means for automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN), wherein the DI2IN uses a conditional random field (CRF) energy function to estimate the output image based on the input medical image and uses a trained deep learning network to model unary and pairwise terms of the CRF energy function.

16. The apparatus of claim 15, wherein the target medical image analysis task is one of anatomic landmark detection, anatomic structure detection, anatomic structure segmentation, lesion detection, segmentation or characterization, image denoising, cross-domain image synthesis, cross-modality image registration, or quantitative parameter mapping.

17. The apparatus of claim 15, further comprising means for training the DI2IN comprising:

means for defining a type of output image that provides the result of the target medical image analysis task;

means for receiving a plurality of input training images;

means for generating corresponding output training images for the plurality of input training images, resulting in a training set of paired input and output training images; and

means for training the DI2IN by learning weight parameters of a deep learning network that models the unary and pairwise terms of the CRF energy function that result in a maximum likelihood for paired input and output training images over the training set of paired input and output training images.

18. The apparatus of claim 15, wherein the means for automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN) comprises:

means for estimating an output image that maximizes a likelihood of the CRF energy function given the input medical image and a set of learned weight parameters of the trained deep learning network, wherein the trained deep learning network calculates the unary and pairwise terms of the CRF energy function based on the input medical image, the estimated output image, and the set of learned weight parameters of the trained deep learning network.

19. The apparatus of claim 15, wherein the means for automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN) comprises:

means for generating an image pyramid with a plurality of reduced resolution images of the input medical image;

means for automatically generating a respective output image that provides a result of the target medical image analysis task on each of the reduced resolution images of the input medical image using a sequence of trained DI2INs.

20. The apparatus of claim 19, wherein the respective output image generated for each of the plurality of reduced resolution images of the input medical image defines a region of interest that is cropped from an image at a subsequent higher resolution of the input medical image.

21. The apparatus of claim 15, wherein the means for automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN) comprises:

means for dividing the input medical image into a plurality of parts;

means for automatically generating a respective output image that provides a result of the target medical image analysis task on each of the plurality of parts of the input medical image using a respective trained DI2IN for each of the plurality of parts; and

means for aggregating the output images that provide the results of the target medical image analysis task on each of the plurality of parts to generate a final output image that provides the result of the target medical image analysis task on the input medical image.

22. A non-transitory computer readable medium storing computer program instructions for automatically performing a medical image analysis task on a medical image of a patient, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

receiving an input medical image of a patient; and

automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN), wherein the DI2IN uses a conditional random field (CRF) energy function to estimate the output image based on the input medical image and uses a trained deep learning network to model unary and pairwise terms of the CRF energy function.

23. The non-transitory computer readable medium of claim 22, wherein the target medical image analysis task is one of anatomic landmark detection, anatomic structure detection, anatomic structure segmentation, lesion detection, segmentation or characterization, image denoising, cross-domain image synthesis, cross-modality image registration, or quantitative parameter mapping.

24. The non-transitory computer readable medium of claim 22, wherein the operations further comprise, in a training stage prior to receiving the input medical image of the patient:

defining a type of output image that provides the result of the target medical image analysis task;

receiving a plurality of input training images;

receiving or generating corresponding output training images for the plurality of input training images, resulting in a training set of paired input and output training images; and

training the DI2IN by learning weight parameters of a deep learning network that models the unary and pairwise terms of the CRF energy function that result in a maximum likelihood for paired input and output training images over the training set of paired input and output training images.

25. The non-transitory computer readable medium of claim 22, wherein automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN) comprises:

estimating an output image that maximizes a likelihood of the CRF energy function given the input medical image and a set of learned weight parameters of the trained deep learning network, wherein the trained deep learning network calculates the unary and pairwise terms of the CRF energy function based on the input medical image, the estimated output image, and the set of learned weight parameters of the trained deep learning network.

26. The non-transitory computer readable medium of claim 22, wherein automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN) comprises:

generating an image pyramid with a plurality of reduced resolution images of the input medical image;

automatically generating a respective output image that provides a result of the target medical image analysis task on each of the reduced resolution images of the input medical image using a sequence of trained DI2INs.

27. The non-transitory computer readable medium of claim 26, wherein the respective output image generated for each of the plurality of reduced resolution images of the input medical image defines a region of interest that is cropped from an image at a subsequent higher resolution of the input medical image.

28. The non-transitory computer readable medium of claim 22, wherein automatically generating an output image that provides a result of a target medical image analysis task on the input medical image using a trained deep image-to-image network (DI2IN) comprises:

dividing the input medical image into a plurality of parts;

automatically generating a respective output image that provides a result of the target medical image analysis task on each of the plurality of parts of the input medical image using a respective trained DI2IN for each of the plurality of parts; and

aggregating the output images that provide the results of the target medical image analysis task on each of the plurality of parts to generate a final output image that provides the result of the target medical image analysis task on the input medical image.