Unsupervised deep representation learning for fine-grained body part recognition

• US 10,452,899 B2
• Filed: 08/29/2017
• Issued: 10/22/2019
• Est. Priority Date: 08/31/2016
• Status: Expired due to Fees

• Current Assignee: Siemens Healthcare GMBH (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Zhang, Pengyue, Zheng, Yefeng
• Primary Examiner(s): Allison, Andrae S
• Application Number:
  US15/689,046
• Publication Number:
  US 20180060652A1
• Time in Patent Office: 784 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2148 characterised by the process organisation or structure, e.g. boosting cascade
• CPC Class Codes: G06F 18/2155 characterised by the incorporation of unlabelled data, e.g. multiple instance learning [MIL], semi-supervised techniques using expectation-maximisation [EM] or naïve labelling
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/24133 Distances to prototypes
• CPC Class Codes: G06F 18/25 Fusion techniques
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10072 Tomographic images
• CPC Class Codes: G06T 2207/10081 Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/10088 Magnetic resonance imaging [MRI]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/60 Analysis of geometric attributes
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A method and apparatus for deep learning based fine-grained body part recognition in medical imaging data is disclosed. A paired convolutional neural network (P-CNN) for slice ordering is trained based on unlabeled training medical image volumes. A convolutional neural network (CNN) for fine-grained body part recognition is trained by fine-tuning learned weights of the trained P-CNN for slice ordering. The CNN for fine-grained body part recognition is trained to calculate, for an input transversal slice of a medical imaging volume, a normalized height score indicating a normalized height of the input transversal slice in the human body.

Claims

1. A method for deep learning based fine-grained body part recognition in medical imaging data, comprising:

training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes;

wherein training the P-CNN for slice ordering based on the unlabeled training medical image volumes comprises;

randomly sampling transversal slice pairs from the unlabeled training medical image volumes, wherein each transversal slice pair is randomly sampled from the same training volume, and

training the P-CNN to predict a relative order of a pair of transversal slices of a medical imaging volume based on the randomly sampled transversal slice pairs, wherein the P-CNN includes two identical sub-networks for a first plurality of layers, each to extract feature from a respective slice of the pair of transversal slices, and global final layers to fuse outputs of the sub-networks and calculate a binary classification result regarding the relative order of the pair of transversal slices; and

training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering.

2. The method of claim 1, wherein the CNN for fine-grained body part recognition includes a first plurality of layers for extracting features from an input transversal slice of a medical imaging volume and a set of final regression layers for calculating a normalized height score for the input transversal slice corresponding to a normalized height of the input transversal slice in a human body.

3. The method of claim 2, wherein training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering comprises:

initializing weights of the first plurality of layers of the CNN for fine-grained body part recognition using learned weights of the first plurality of layers of one of the sub-networks in the trained P-CNN for slice ordering;

randomly initializing weights of the final regression layers of the CNN; and

fine-tuning the weights of all of the layers of the CNN based on annotated training medical image volumes with a reduced learning rate of the first plurality of layers.

4. A method for deep learning based fine-grained body part recognition in medical imaging data, comprising:

training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes; and

training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering, wherein training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering comprises;

generating a normalized body height model based on locations of a set of anatomical landmarks in at least a subset of annotated medical imaging volumes,

calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model, and

training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering based on the transversal slices of the annotated medical imaging volumes and the normalized height scores calculated for the transversal slices.

5. The method of claim 4, wherein the normalized body height model normalizes a height of a human body between values of 0 and 1 and includes average normalized height values for each of the set of anatomical landmarks in at least the subset of the annotated medical imaging volumes.

6. The method of claim 5, wherein calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model comprises:

for each of the annotated medical imaging volumes;

assigning a respective normalized height value for the location of each of the set of anatomical landmarks as the average normalized height value of that anatomical landmark in the normalized body height model; and

interpolating between the anatomical landmarks in the set of anatomical landmarks to determine normalized height values for the transversal slices between the anatomical landmark locations.

7. The method of claim 6, wherein the set of anatomical landmarks comprises top of a head, neck, lung, spine, knee, and foot landmarks.

8. An apparatus for deep learning based fine-grained body part recognition in medical imaging data, comprising:

means for training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes, wherein the means for training the P-CNN for slice ordering based on the unlabeled training medical image volumes comprises;

means for randomly sampling transversal slice pairs from the unlabeled training medical image volumes, wherein each transversal slice pair is randomly sampled from the same training volume, and

means for training the P-CNN to predict a relative order of a pair of transversal slices of a medical imaging volume based on the randomly sampled transversal slice pairs, wherein the P-CNN includes two identical sub-networks for a first plurality of layers, each to extract feature from a respective slice of the pair of transversal slices, and global final layers to fuse outputs of the sub-networks and calculate a binary classification result regarding the relative order of the pair of transversal slices; and

means for training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering.

9. The apparatus of claim 8, wherein the CNN for fine-grained body part recognition includes a first plurality of layers for extracting features from an input transversal slice of a medical imaging volume and a set of final regression layers for calculating a normalized height score for the input transversal slice corresponding to a normalized height of the input transversal slice in a human body.

10. The apparatus of claim 9, wherein the means for training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering comprises:

means for initializing weights of the first plurality of layers of the CNN for fine-grained body part recognition using learned weights of the first plurality of layers of one of the sub-networks in the trained P-CNN for slice ordering;

means for randomly initializing weights of the final regression layers of the CNN; and

means for fine-tuning the weights of all of the layers of the CNN based on annotated training medical image volumes with a reduced learning rate of the first plurality of layers.

11. An apparatus for deep learning based fine-grained body part recognition in medical imaging data, comprising:

means for training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes; and

means for training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering, wherein the means for training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering comprises;

means for generating a normalized body height model based on locations of a set of anatomical landmarks in at least a subset of annotated medical imaging volumes,

means for calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model, and

means for training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering based on the transversal slices of the annotated medical imaging volumes and the normalized height scores calculated for the transversal slices.

12. The apparatus of claim 11, wherein the normalized body height model normalizes a height of a human body between values of 0 and 1 and includes average normalized height values for each of the set of anatomical landmarks in at least the subset of the annotated medical imaging volumes.

13. The apparatus of claim 12, wherein the means for calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model comprises:

means for assigning a respective normalized height value for the location of each of the set of anatomical landmarks in each of the annotated medical imaging volumes as the average normalized height value of that anatomical landmark in the normalized body height model; and

means for interpolating between the anatomical landmarks in the set of anatomical landmarks to determine normalized height values for the transversal slices between the anatomical landmark locations in each of the annotated medical imaging volumes.

14. A non-transitory computer readable medium storing computer program instructions for deep learning based fine-grained body part recognition in medical imaging data, the computer program instructions when executed on a processor cause the processor to perform operations comprising:

training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes, wherein training the P-CNN for slice ordering based on the unlabeled training medical image volumes comprises;

randomly sampling transversal slice pairs from the unlabeled training medical image volumes, wherein each transversal slice pair is randomly sampled from the same training volume, and

training the P-CNN to predict a relative order of a pair of transversal slices of a medical imaging volume based on the randomly sampled transversal slice pairs, wherein the P-CNN includes two identical sub-networks for a first plurality of layers, each to extract feature from a respective slice of the pair of transversal slices, and global final layers to fuse outputs of the sub-networks and calculate a binary classification result regarding the relative order of the pair of transversal slices; and

training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering.

15. The non-transitory computer readable medium of claim 14, wherein the CNN for fine-grained body part recognition includes a first plurality of layers for extracting features from an input transversal slice of a medical imaging volume and a set of final regression layers for calculating a normalized height score for the input transversal slice corresponding to a normalized height of the input transversal slice in a human body.

16. The non-transitory computer readable medium of claim 15, wherein training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering comprises:

initializing weights of the first plurality of layers of the CNN for fine-grained body part recognition using learned weights of the first plurality of layers of one of the sub-networks in the trained P-CNN for slice ordering;

randomly initializing weights of the final regression layers of the CNN; and

fine-tuning the weights of all of the layers of the CNN based on annotated training medical image volumes with a reduced learning rate of the first plurality of layers.

17. A non-transitory computer readable medium storing computer program instructions for deep learning based fine-grained body part recognition in medical imaging data, the computer program instructions when executed on a processor cause the processor to perform operations comprising:

training a paired convolutional neural network (P-CNN) for slice ordering based on unlabeled training medical image volumes; and

training a convolutional neural network (CNN) for fine-grained body part recognition by fine-tuning learned weights of the trained P-CNN for slice ordering, wherein training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering comprises;

generating a normalized body height model based on locations of a set of anatomical landmarks in at least a subset of annotated medical imaging volumes,

calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model, and

training the CNN for fine-grained body part recognition by fine-tuning the learned weights of the trained P-CNN for slice ordering based on the transversal slices of the annotated medical imaging volumes and the normalized height scores calculated for the transversal slices.

18. The non-transitory computer readable medium of claim 17, wherein the normalized body height model normalizes a height of a human body between values of 0 and 1 and includes average normalized height values for each of the set of anatomical landmarks in at least the subset of the annotated medical imaging volumes.

19. The non-transitory computer readable medium of claim 18, wherein calculating normalized height scores for transversal slices of the annotated medical imaging volumes based on the normalized body height model comprises:

for each of the annotated medical imaging volumes;

assigning a respective normalized height value for the location of each of the set of anatomical landmarks as the average normalized height value of that anatomical landmark in the normalized body height model; and

interpolating between the anatomical landmarks in the set of anatomical landmarks to determine normalized height values for the transversal slices between the anatomical landmark locations.