Method and System for Anatomical Object Detection Using Marginal Space Deep Neural Networks

• US 20160174902A1
• Filed: 02/26/2016
• Published: 06/23/2016
• Est. Priority Date: 10/17/2013
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens AG
• Inventors: Hguyen, Hien, Georgescu, Bogdan, Zheng, Yefeng, Singh, Vivek Kumar, Comaniciu, Dorin, Liu, David
• Granted Patent: US 9,730,643 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 5/7267 involving training the classification device
• CPC Class Codes: A61B 6/503 for diagnosis of the heart
• CPC Class Codes: A61B 6/507 for determination of haemodynamic parameters, e.g. perfusion CT
• CPC Class Codes: A61B 6/5217 extracting a diagnostic or physiological parameter from medical diagnostic data
• CPC Class Codes: A61B 6/563 involving image data transmission via a network
• CPC Class Codes: A61B 8/0883 for diagnosis of the heart
• CPC Class Codes: A61B 8/0891 for diagnosis of blood vessels
• CPC Class Codes: A61B 8/5223 for extracting a diagnostic or physiological parameter from medical diagnostic data for algorithms to analyse biomedical images G06T7/0012
• CPC Class Codes: A61B 8/565 involving data transmission via a network
• CPC Class Codes: G06F 18/28 Determining representative reference patterns, e.g. by averaging or distorting; Generating dictionaries
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2207/10072 Tomographic images
• CPC Class Codes: G06T 2207/10132 Ultrasound image
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20024 Filtering details
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30048 Heart; Cardiac
• CPC Class Codes: G06T 2207/30101 : Blood vessel; Artery; Vein; Vascular
• CPC Class Codes: G06T 7/0012 : Biomedical image inspection
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 10/25 : Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/454 : Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/772 : Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 2201/031 : of internal organs
• CPC Class Codes: G16H 50/30 : for calculating health indices; for individual health risk assessment

Abstract

A method and system for anatomical object detection using marginal space deep neural networks is disclosed. The pose parameter space for an anatomical object is divided into a series of marginal search spaces with increasing dimensionality. A respective sparse deep neural network is trained for each of the marginal search spaces, resulting in a series of trained sparse deep neural networks. Each of the trained sparse deep neural networks is trained by injecting sparsity into a deep neural network by removing filter weights of the deep neural network.

Claims

1. A method for anatomical object detection in a 3D medical image of a patient, comprising:

receiving a 3D medical image of a patient including a target anatomical object; and

detecting a 3D pose of the target anatomical object in the 3D medical image in a series of marginal parameter spaces of increasing dimensionality using a respective trained sparse deep neural network for each of the marginal search spaces.

2. The method of claim 1, wherein each respective trained sparse deep neural network inputs hypothesis image patches in the respective marginal parameter space, samples voxels in each hypothesis image patch in a sparse sampling pattern adaptively learned during training of the respective trained sparse deep neural network, and calculates, for each hypothesis image patch, a probability that the hypothesis image patch is an image patch of the target anatomical object in the respective marginal parameter space based on the voxels sampled in the hypothesis image patch in the sparse sampling pattern.

3. The method of claim 1, wherein at least one of the respective trained sparse deep neural networks is trained by:

training a deep neural network by learning parameters of a respective filter for each of a plurality of neurons in each of a plurality of layers of the deep neural network, wherein the parameters of each respective filter include a bias and a plurality of weights that weight node outputs of the nodes of a previous one of the plurality of layers; and

sparsifying the weights of the filters for at least one layer of the trained deep neural network.

4. The method of claim 3, wherein sparsifying the weights of the filters for at least one layer of the trained deep neural network comprises:

reducing a number of non-zero weights in each filter for the at least one layer in the trained deep neural network by setting a predetermined percentage of non-zero weights with lowest magnitudes in each filter equal to zero; and

retraining the deep neural network with only the remaining non-zero weights in each filter for the at least one layer to refine the bias and the remaining non-zero weights in each filter for the at least one layer to recover from an effect of reducing the number of non-zero weights.

5. The method of claim 4, wherein sparsifying the weights of the filters for at least one layer of the trained deep neural network further comprises:

repeating the reducing and retraining steps in each of a plurality of iterations.

6. The method of claim 5, wherein sparsifying the weights of the filters for at least one layer of the trained deep neural network further comprises:

performing, in each iteration, a normalization to preserve an L₁-norm for each filter for the at least one layer.

7. The method of claim 5, wherein repeating the reducing and retraining steps in each of a plurality of iterations comprises:

repeating the reducing and retraining steps until a target sparsity degree is reached.

8. The method of claim 3, wherein the at least one layer is a first hidden layer of the deep neural network.

9. The method of claim 3, wherein sparsifying the weights of the filters for at least one layer of the trained deep neural network comprises:

performing re-weighted L1-norm regularization on the weights of the filters for the at least one layer of the trained deep neural network, wherein the re-weighted L1-norm regularization drives a plurality of non-zero weights of the filters for the at least one layer to zero; and

training a sparse deep neural with only the remaining non-zero weights in the filters for the at least one layer.

10. The method of claim 1, wherein detecting a 3D pose of the target anatomical object in the 3D medical image in a series of marginal parameter spaces of increasing dimensionality using a respective trained sparse deep neural network for each of the marginal search spaces comprises, for each of the marginal parameter spaces:

filtering hypothesis image patches in that marginal parameter space using a respective cascade of trained shallow sparse neural networks to eliminate a plurality of negative hypothesis image patches; and

evaluating each of the remaining hypothesis image patches in that marginal parameter space using the respective trained sparse deep neural network to determine a classification score for each of the remaining hypothesis image patches in that marginal parameter space.

11. The method of claim 10, wherein the respective cascade of trained shallow sparse neural networks for each marginal parameter space is trained by sequentially adding shallow sparse neural networks to the cascaded until a balanced set of positive and negative training samples is achieved for training the respective sparse deep neural network.

12. The method of claim 1, further comprising:

rigidly registering a mean model of the target anatomical object to the 3D medical mage using the detected 3D pose of the target anatomical object in the 3D medical image;

refining the model of the target anatomical structure in the 3D medical image using a trained sparse deep neural network boundary detector applied to image patches corresponding to sampling points along a line normal to a surface of the model at each point on the surface of the model; and

projecting the refined model of the target anatomical object in the 3D medical image to a learned shape space of the target anatomical object.

13. The method of claim 1, wherein the series of marginal parameter spaces comprises a position parameter space, a position-orientation parameter space, and a position-orientation-scale parameter space.

14. The method of claim 13, wherein detecting a 3D pose of the target anatomical object in the 3D medical image in a series of marginal parameter spaces of increasing dimensionality using a respective trained sparse deep neural network for each of the marginal search spaces comprises:

detecting position candidates in the 3D medical image using a first cascade of trained sparse shallow neural networks to filter image patches centered at each of a plurality of voxels in the medical image and a first trained sparse deep neural network to evaluate image patches not eliminated by the first cascade;

generating position-orientation hypotheses based on the detected position candidates;

detecting position-orientation candidates in the 3D medical image using a second cascade of trained sparse shallow neural networks to filter image patches corresponding to the position-orientation hypotheses and a second trained sparse deep neural network to evaluate image patches corresponding position-orientation hypotheses not eliminated by the second cascade;

generating position-orientation-scale hypotheses based on the detected position-orientation candidates; and

detecting the pose of the target anatomical object in the 3D medical image using a third cascade of trained sparse shallow neural networks to filter image patches corresponding to the position-orientation-scale hypotheses and a third trained sparse deep neural network to evaluate image patches corresponding position-orientation-scale hypotheses not eliminated by the third cascade.

15. A method of automated anatomical landmark detection in a 3D medical image, comprising:

detecting a plurality of landmark candidates for a target anatomical image in the 3D medical image using an initial shallow neural network detector;

calculating deeply learned features for each of the plurality of landmark candidates using a trained deep neural network; and

detecting the target anatomical landmark in the 3D medical image from the plurality of landmark candidates based on the deeply learned features for each of the plurality of landmark candidates and other image-based features extracted from the 3D medical image using a trained classifier.

16. The method of claim 15, wherein the trained classifier is a probabilistic boosting tree (PBT) classifier, and the other image based-features are Haar wavelet features.

17. An apparatus for anatomical object detection in a 3D medical image of a patient, comprising:

means for receiving a 3D medical image of a patient including a target anatomical object; and

means for detecting a 3D pose of the target anatomical object in the 3D medical image in a series of marginal parameter spaces of increasing dimensionality using a respective trained sparse deep neural network for each of the marginal search spaces.

18. The apparatus of claim 17, wherein each respective trained sparse deep neural network inputs hypothesis image patches in the respective marginal parameter space, samples voxels in each hypothesis image patch in a sparse sampling pattern adaptively learned during training of the respective trained sparse deep neural network, and calculates, for each hypothesis image patch, a probability that the hypothesis image patch is an image patch of the target anatomical object in the respective marginal parameter space based on the voxels sampled in the hypothesis image patch in the sparse sampling pattern.

19. The apparatus of claim 17, further comprising means for training at least one of the respective trained sparse deep neural networks, comprising:

means for training a deep neural network by learning parameters of a respective filter for each of a plurality of neurons in each of a plurality of layers of the deep neural network, wherein the parameters of each respective filter include a bias and a plurality of weights that weight node outputs of the nodes of a previous one of the plurality of layers; and

means for sparsifying the weights of the filters for at least one layer of the trained deep neural network.

20. The apparatus of claim 19, wherein the means for sparsifying the weights of the filters for at least one layer of the trained deep neural network comprises:

means for reducing a number of non-zero weights in each filter for the at least one layer in the trained deep neural network by setting a predetermined percentage of non-zero weights with lowest magnitudes in each filter equal to zero; and

means for retraining the deep neural network with only the remaining non-zero weights in each filter for the at least one layer to refine the bias and the remaining non-zero weights in each filter for the at least one layer to recover from an effect of reducing the number of non-zero weights.

21. The apparatus of claim 20, wherein the means for sparsifying the weights of the filters for at least one layer of the trained deep neural network further comprises:

means for performing a normalization to preserve an L₁-norm for each filter for the at least one layer.

22. The apparatus of claim 19, wherein the at least one layer is a first hidden layer of the deep neural network.

23. The apparatus of claim 19, wherein the means for sparsifying the weights of the filters for at least one layer of the trained deep neural network comprises:

means for performing re-weighted L1-norm regularization on the weights of the filters for the at least one layer of the trained deep neural network, wherein the re-weighted L1-norm regularization drives a plurality of non-zero weights of the filters for the at least one layer to zero; and

means for training a sparse deep neural with only the remaining non-zero weights in the filters for the at least one layer.

24. The apparatus of claim 17, wherein the means for detecting a 3D pose of the target anatomical object in the 3D medical image in a series of marginal parameter spaces of increasing dimensionality using a respective trained sparse deep neural network for each of the marginal search spaces comprises:

means for filtering hypothesis image patches in each of the marginal parameter spaces using a respective cascade of trained shallow sparse neural networks to eliminate a plurality of negative hypothesis image patches; and

evaluating each of the remaining hypothesis image patches in each of the marginal parameter spaces using the respective trained sparse deep neural network to determine a classification score for each of the remaining hypothesis image patches in that marginal parameter space.

25. The apparatus of claim 17, further comprising:

means for rigidly registering a mean model of the target anatomical object to the 3D medical mage using the detected 3D pose of the target anatomical object in the 3D medical image;

means for refining the model of the target anatomical structure in the 3D medical image using a trained sparse deep neural network boundary detector applied to image patches corresponding to sampling points along a line normal to a surface of the model at each point on the surface of the model; and

means for projecting the refined model of the target anatomical object in the 3D medical image to a learned shape space of the target anatomical object.

26. The apparatus of claim 17, wherein the series of marginal parameter spaces comprises a position parameter space, a position-orientation parameter space, and a position-orientation-scale parameter space.

27. The apparatus of claim 26, wherein the means for detecting a 3D pose of the target anatomical object in the 3D medical image in a series of marginal parameter spaces of increasing dimensionality using a respective trained sparse deep neural network for each of the marginal search spaces comprises:

means for detecting position candidates in the 3D medical image using a first cascade of trained sparse shallow neural networks to filter image patches centered at each of a plurality of voxels in the medical image and a first trained sparse deep neural network to evaluate image patches not eliminated by the first cascade;

means for generating position-orientation hypotheses based on the detected position candidates;

means for detecting position-orientation candidates in the 3D medical image using a second cascade of trained sparse shallow neural networks to filter image patches corresponding to the position-orientation hypotheses and a second trained sparse deep neural network to evaluate image patches corresponding position-orientation hypotheses not eliminated by the second cascade;

means for generating position-orientation-scale hypotheses based on the detected position-orientation candidates; and

means for detecting the pose of the target anatomical object in the 3D medical image using a third cascade of trained sparse shallow neural networks to filter image patches corresponding to the position-orientation-scale hypotheses and a third trained sparse deep neural network to evaluate image patches corresponding position-orientation-scale hypotheses not eliminated by the third cascade.

28. An apparatus for automated anatomical landmark detection in a 3D medical image, comprising:

means for detecting a plurality of landmark candidates for a target anatomical image in the 3D medical image using an initial shallow neural network detector;

means for calculating deeply learned features for each of the plurality of landmark candidates using a trained deep neural network; and

means for detecting the target anatomical landmark in the 3D medical image from the plurality of landmark candidates based on the deeply learned features for each of the plurality of landmark candidates and other image-based features extracted from the 3D medical image using a trained classifier.

29. The apparatus of claim 28, wherein the trained classifier is a probabilistic boosting tree (PBT) classifier, and the other image based-features are Haar wavelet features.

30. A non-transitory computer readable medium storing computer program instructions for anatomical object detection in a 3D medical image of a patient, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

receiving a 3D medical image of a patient including a target anatomical object; and

detecting a 3D pose of the target anatomical object in the 3D medical image in a series of marginal parameter spaces of increasing dimensionality using a respective trained sparse deep neural network for each of the marginal search spaces.

31. The non-transitory computer readable medium of claim 30, wherein each respective trained sparse deep neural network inputs hypothesis image patches in the respective marginal parameter space, samples voxels in each hypothesis image patch in a sparse sampling pattern adaptively learned during training of the respective trained sparse deep neural network, and calculates, for each hypothesis image patch, a probability that the hypothesis image patch is an image patch of the target anatomical object in the respective marginal parameter space based on the voxels sampled in the hypothesis image patch in the sparse sampling pattern.

32. The non-transitory computer readable medium of claim 30, wherein at least one of the respective trained sparse deep neural networks is trained by:

training a deep neural network by learning parameters of a respective filter for each of a plurality of neurons in each of a plurality of layers of the deep neural network, wherein the parameters of each respective filter include a bias and a plurality of weights that weight node outputs of the nodes of a previous one of the plurality of layers; and

sparsifying the weights of the filters for at least one layer of the trained deep neural network.

33. The non-transitory computer readable medium of claim 32, wherein sparsifying the weights of the filters for at least one layer of the trained deep neural network comprises:

reducing a number of non-zero weights in each filter for the at least one layer in the trained deep neural network by setting a predetermined percentage of non-zero weights with lowest magnitudes in each filter equal to zero; and

retraining the deep neural network with only the remaining non-zero weights in each filter for the at least one layer to refine the bias and the remaining non-zero weights in each filter for the at least one layer to recover from an effect of reducing the number of non-zero weights.

34. The non-transitory computer readable medium of claim 33, wherein sparsifying the weights of the filters for at least one layer of the trained deep neural network further comprises:

repeating the reducing and retraining steps in each of a plurality of iterations.

35. The non-transitory computer readable medium of claim 34, wherein sparsifying the weights of the filters for at least one layer of the trained deep neural network further comprises:

performing, in each iteration, a normalization to preserve an L₁-norm for each filter for the at least one layer.

36. The non-transitory computer readable medium of claim 34, wherein repeating the reducing and retraining steps in each of a plurality of iterations comprises:

repeating the reducing and retraining steps until a target sparsity degree is reached.

37. The non-transitory computer readable medium of claim 32, wherein the at least one layer is a first hidden layer of the deep neural network.

38. The non-transitory computer readable medium of claim 32, wherein sparsifying the weights of the filters for at least one layer of the trained deep neural network comprises:

performing re-weighted L1-norm regularization on the weights of the filters for the at least one layer of the trained deep neural network, wherein the re-weighted L1-norm regularization drives a plurality of non-zero weights of the filters for the at least one layer to zero; and

training a sparse deep neural with only the remaining non-zero weights in the filters for the at least one layer.

39. The non-transitory computer readable medium of claim 30, wherein detecting a 3D pose of the target anatomical object in the 3D medical image in a series of marginal parameter spaces of increasing dimensionality using a respective trained sparse deep neural network for each of the marginal search spaces comprises, for each of the marginal parameter spaces:

filtering hypothesis image patches in that marginal parameter space using a respective cascade of trained shallow sparse neural networks to eliminate a plurality of negative hypothesis image patches; and

evaluating each of the remaining hypothesis image patches in that marginal parameter space using the respective trained sparse deep neural network to determine a classification score for each of the remaining hypothesis image patches in that marginal parameter space.

40. The non-transitory computer readable medium of claim 39, wherein the respective cascade of trained shallow sparse neural networks for each marginal parameter space is trained by sequentially adding shallow sparse neural networks to the cascaded until a balanced set of positive and negative training samples is achieved for training the respective sparse deep neural network.

41. The non-transitory computer readable medium of claim 30, further comprising:

rigidly registering a mean model of the target anatomical object to the 3D medical mage using the detected 3D pose of the target anatomical object in the 3D medical image;

refining the model of the target anatomical structure in the 3D medical image using a trained sparse deep neural network boundary detector applied to image patches corresponding to sampling points along a line normal to a surface of the model at each point on the surface of the model; and

projecting the refined model of the target anatomical object in the 3D medical image to a learned shape space of the target anatomical object.

42. The non-transitory computer readable medium of claim 30, wherein the series of marginal parameter spaces comprises a position parameter space, a position-orientation parameter space, and a position-orientation-scale parameter space.

43. The non-transitory computer readable medium of claim 42, wherein detecting a 3D pose of the target anatomical object in the 3D medical image in a series of marginal parameter spaces of increasing dimensionality using a respective trained sparse deep neural network for each of the marginal search spaces comprises:

detecting position candidates in the 3D medical image using a first cascade of trained sparse shallow neural networks to filter image patches centered at each of a plurality of voxels in the medical image and a first trained sparse deep neural network to evaluate image patches not eliminated by the first cascade;

generating position-orientation hypotheses based on the detected position candidates;

detecting position-orientation candidates in the 3D medical image using a second cascade of trained sparse shallow neural networks to filter image patches corresponding to the position-orientation hypotheses and a second trained sparse deep neural network to evaluate image patches corresponding position-orientation hypotheses not eliminated by the second cascade;

generating position-orientation-scale hypotheses based on the detected position-orientation candidates; and

detecting the pose of the target anatomical object in the 3D medical image using a third cascade of trained sparse shallow neural networks to filter image patches corresponding to the position-orientation-scale hypotheses and a third trained sparse deep neural network to evaluate image patches corresponding position-orientation-scale hypotheses not eliminated by the third cascade.

44. A non-transitory computer readable medium storing computer program instruction for automated anatomical landmark detection in a 3D medical image, the computer program instructions when executed by a processor cause the processor to perform operations comprising:

detecting a plurality of landmark candidates for a target anatomical image in the 3D medical image using an initial shallow neural network detector;

calculating deeply learned features for each of the plurality of landmark candidates using a trained deep neural network; and

detecting the target anatomical landmark in the 3D medical image from the plurality of landmark candidates based on the deeply learned features for each of the plurality of landmark candidates and other image-based features extracted from the 3D medical image using a trained classifier.

45. The non-transitory computer readable medium of claim 44, wherein the trained classifier is a probabilistic boosting tree (PBT) classifier, and the other image based-features are Haar wavelet features.