DEEP SIMILARITY LEARNING FOR MULTIMODAL MEDICAL IMAGES

• US 20160093048A1
• Filed: 09/25/2015
• Published: 03/31/2016
• Est. Priority Date: 09/25/2014
• Status: Active Grant

• Current Assignee: Siemens Healthcare GMBH (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Zhang, Li, Zheng, Yefeng, Cheng, Xi
• Granted Patent: US 9,922,272 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06T 2207/10072 Tomographic images
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/30 Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G06V 10/761 Proximity, similarity or dissimilarity measures
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 10/82 using neural networks

Abstract

The present embodiments relate to machine learning for multimodal image data. By way of introduction, the present embodiments described below include apparatuses and methods for learning a similarity metric using deep learning based techniques for multimodal medical images. A novel similarity metric for multi-modal images is provided using the corresponding states of pairs of image patches to generate a classification setting for each pair. The classification settings are used to train a deep neural network via supervised learning. A multi-modal stacked denoising auto encoder (SDAE) is used to pre-train the neural network. A continuous and smooth similarity metric is constructed based on the output of the neural network before activation in the last layer. The trained similarity metric may be used to improve the results of image fusion.

Claims

1. A method for similarity metric learning for multimodal medical image data, the method comprising:

receiving a first set of image data of a volume, wherein the first set of image data is captured with a first imaging modality;

receiving a second set of image data of the volume, wherein the second set of image data is captured with a second imaging modality;

aligning the first set of image data and the second set of image data;

training a first set of parameters with a multimodal stacked denoising auto encoder to generate a shared feature representation of the first set of image data and the second set of image data;

training a second set of parameters with a denoising auto encoder to generate a transformation of the shared feature representation;

initializing, using the first set of parameters and the second set of parameters, a neural network classifier; and

training, using training data from the aligned first set of image data and the second set of image data, the neural network classifier to generate a similarity metric for the first and second imaging modalities.

2. The method of claim 1 wherein the first imaging modality is computed tomography and the second imaging modality is magnetic resonance.

3. The method of claim 1 wherein the aligning comprises rigidly aligning the first set of image data and the second set of image data.

4. The method of claim 1 wherein the aligning comprises sampling the first set of image data and the second set of image data to generate a plurality of positive training data sets and a plurality of negative training data sets.

5. The method of claim 1 wherein the multimodal stacked denoising auto encoder comprises:

a first layer comprising a first denoising auto encoder and a second denosing auto encoder; and

a second layer comprising a third denoising auto encoder.

6. The method of claim 5 wherein training the multimodal stacked denoising auto encoder comprises:

training the first denoising auto encoder to generate a first feature vector from the first set of image data;

training the second denoising auto encoder to generate a second feature vector from the second set of image data; and

training the third denoising auto encoder to generate the shared feature representation from the first feature vector and the second feature vector.

7. The method of claim 1 wherein the neural network classifier is a five layer deep neural network classifier.

8. The method of claim 7 wherein initializing the neural network classifier comprises:

initializing parameters in a first three layers of the neural network classifier with parameters from the multimodal stacked denoising auto encoder; and

initializing parameters in a fourth layer of the neural network classifier with parameters from the denoising auto encoder.

9. The method of claim 8 wherein initializing the neural network classifier further comprises:

initializing missing parameters in the neural network classifier with zeros.

10. A system comprising:

a first scanner configured to capture a first set of image data of a volume with a first imaging modality;

a second scanner configured to capture a second set of image data of the volume with a second imaging modality; and

a processor configured to;

receive, from the first scanner and the second scanner over a network, the first set of image data and the second set of image data;

rigidly align the first set of image data and the second set of image data;

train a first set of parameters with a multimodal stacked denoising auto encoder to generate a shared feature representation of the first set of image data and the second set of image data;

train a second set of parameters with a denoising auto encoder to generate a transformation of the shared feature representation;

initialize, using the first set of parameters and the second set of parameters, a deep neural network classifier; and

train, using training data from the aligned first set of image data and the second set of image data, the deep neural network classifier to generate a similarity metric for the first and second imaging modalities.

11. The system of claim 10 wherein the first imaging modality is computed tomography and the second imaging modality is magnetic resonance.

12. The system of claim 10 wherein the rigidly aligning comprises sampling the first set of image data and the second set of image data to generate a plurality of positive training data sets and a plurality of negative training data sets.

13. The system of claim 10 wherein the multimodal stacked denoising auto encoder comprises:

a first layer comprising a first denoising auto encoder and a second denosing auto encoder; and

a second layer comprising a third denoising auto encoder.

14. The system of claim 13 wherein training the multimodal stacked denoising auto encoder comprises:

training the first denoising auto encoder to generate a first feature vector from the first set of image data;

training the second denoising auto encoder to generate a second feature vector from the second set of image data; and

training the third denoising auto encoder to generate the shared feature representation from the first feature vector and the second feature vector.

15. The system of claim 10 wherein the deep neural network classifier is a five layer deep neural network classifier.

16. The system of claim 15 wherein initializing the deep neural network classifier comprises:

initializing parameters in a first three layers of the deep neural network classifier with parameters from the multimodal stacked denoising auto encoder; and

initializing parameters in a fourth layer of the deep neural network classifier with parameters from the denoising auto encoder.

17. The system of claim 14 wherein initializing the deep neural network classifier further comprises:

initializing missing parameters in the deep neural network classifier with zeros.

18. A method comprising:

receiving, from a first scanner, a first set of image data captured of a volume using a first imaging modality;

receiving, from a second scanner, a second set of image data captured of the volume using a second imaging modality;

identifying, by the processor using a trained similarity metric for multimodal image data, which voxels from the first set of image data that correspond to the same position in the volume as voxels from the second set of image data; and

performing image fusion on the first set of image data and the second set of image data using the identified voxels.

19. The method of claim 18, wherein the similarity metric for multimodal image data is trained by a neural network classifier, wherein the neural network classifier is initialized by parameters from a multimodal stacked denoising auto encoder.

20. The method of claim 19, wherein the neural network classifier is a five layer deep neural network classifier.