Method and System for Approximating Deep Neural Networks for Anatomical Object Detection

• US 20160328643A1
• Filed: 05/07/2015
• Published: 11/10/2016
• Est. Priority Date: 05/07/2015
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens AG
• Inventors: Kretschmer, Jan, Liu, David, Lay, Nathan, Zhou, Shaohua Kevin, Nguyen, Hien, Singh, Vivek Kumar, Zheng, Yefeng, Georgescu, Bogdan, Comaniciu, Dorin
• Granted Patent: US 9,633,306 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/082 modifying the architecture, e.g. adding, deleting or silencing nodes or connections
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06V 10/446 using Haar-like filters, e.g. using integral image techniques
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G06V 40/10 Human or animal bodies, e.g. vehicle occupants or pedestrians; Body parts, e.g. hands

Abstract

A method and system for approximating a deep neural network for anatomical object detection is discloses. A deep neural network is trained to detect an anatomical object in medical images. An approximation of the trained deep neural network is calculated that reduces the computational complexity of the trained deep neural network. The anatomical object is detected in an input medical image of a patient using the approximation of the trained deep neural network.

Claims

1. A method for anatomical object detection in a medical image comprising:

training a deep neural network to detect the anatomical object in medical images;

calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network; and

detecting the anatomical object in a received medical image of a patient using the approximation of the trained deep neural network.

2. The method of claim 1, wherein training a deep neural network to detect the anatomical object in medical images comprises training a respective filter for each of a plurality of nodes in each of a plurality of layers of the deep neural network, wherein each respective filter is a weight matrix comprising a plurality of weights that weight node outputs of the nodes of a previous one of the plurality of layers, and calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

sparsifying the weights of the filters for each of the plurality of layers of the trained deep neural network.

3. The method of claim 2, wherein sparsifying the weights of the filters for each of the plurality of layers of the trained deep neural network comprises:

reducing a number of non-zero weights in each filter for each of the plurality of layers in the trained deep neural network by setting a predetermined percentage of non-zero weights with lowest magnitudes in each filter equal to zero; and

refining the remaining non-zero weights in each filter for each of the plurality of layers to alleviate an effect of reducing the number of non-zero weights in each filter.

4. The method of claim 3, wherein refining the remaining non-zero weights in each filter for each of the plurality of layers to alleviate an effect of reducing the number of non-zero weights in each filter comprises:

performing one or more iterations of back-propagation on the approximation of the trained deep neural network resulting from reducing the number of non-zero weights in each filter to refine the remaining non-zero weights in each filter to reduce a cost function that measures an error between predicted anatomical object locations using the approximation of the trained deep neural network and ground truth anatomical object locations in a set of training data.

5. The method of claim 2, wherein sparsifying the weights of the filters for each of the plurality of layers of the trained deep neural network comprises:

performing re-weighted L1-norm regularization on the weights of the filters for each of the plurality layers of the trained deep neural network, wherein the re-weighted L1-norm regularization drives a plurality of non-zero weights of the filters to zero; and

refining the remaining non-zero weights in the filters for each of the plurality of layers to alleviate an effect of driving the plurality of non-zero weights to zero.

6. The method of claim 5, wherein performing re-weighted L1-norm regularization on the weights of the filters for each of the plurality layers of the trained deep neural network, wherein the re-weighted L1-norm regularization drives a plurality of non-zero weights of the filters to zero comprises:

adding a term that re-weights the L1-norm to a cost function that measures an error between predicted anatomical object locations and ground truth anatomical object locations in a set of training data; and

performing back-propagation on the trained deep neural network to refine the weights in the filters for each of the plurality of layers of the trained deep neural network to reduce the cost function with the added term that re-weights the L1-norm.

7. The method of claim 6, wherein refining the remaining non-zero weights in the filters for each of the plurality of layers to alleviate an effect of driving the plurality of non-zero weights to zero comprises:

performing one or more iterations of back-propagation on the approximation of the trained deep neural network resulting from driving the plurality of non-zero weights to zero to refine the remaining non-zero weights in the filters to reduce the cost function that measures an error between predicted anatomical object locations and ground truth anatomical object locations in the set of training data, without the added term that re-weights the L1-norm

8. The method of claim 1, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

determining a subset of nodes of a plurality nodes in a current layer of the trained deep neural network that linearly approximate the plurality of nodes in the current layer of the trained deep neural network and removing the plurality of nodes in the current layer that are not in subset of nodes from the trained deep neural network; and

updating weights for a next layer of the trained deep neural network based on the subset of nodes remaining in the current layer of the trained deep neural network.

9. The method of claim 8, wherein determining a subset of nodes of a plurality nodes in a current layer of the trained deep neural network that linearly approximate the plurality of nodes in the current layer of the trained deep neural network and removing the plurality of nodes in the current layer that are not in subset of nodes from the trained deep neural network comprises:

determining the subset of nodes in the current layer and a mixing matrix that best minimizes an error between each of the plurality of nodes in the current layer and a respective approximation for each of the plurality of nodes in the currently layer calculated by linearly combining the subset of nodes using the mixing matrix, subject to a constraint on a size of the subset of nodes.

10. The method of claim 9, wherein updating weights for a next layer of the trained deep neural network based on the subset of nodes remaining in the current layer of the trained deep neural network comprises:

removing filters for the next layer of the trained deep neural network whose indices are not in the subset of nodes in the current layer; and

updating the remaining filters for the next layer of the trained deep neural network with weights generated by linearly combining weights of the subset of nodes in the current layer using the mixing matrix to approximate weighted inputs to the next layer from the removed ones of the plurality of nodes in the current layer.

11. The method of claim 8, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network further comprises:

repeating the steps of determining a subset of nodes of a plurality nodes in a current layer of the trained deep neural network that linearly approximate the plurality of nodes in the current layer of the trained deep neural network and removing the plurality of nodes in the current layer that are not in subset of nodes from the trained deep neural network and updating weights for a next layer of the trained deep neural network based on the subset of nodes remaining in the current layer of the trained deep neural network, for each of a plurality of layers in the trained deep neural network, resulting in an initial approximation of the trained deep neural network; and

refining the initial approximation of the trained deep neural network by performing one or more iterations of back-propagation on the initial approximation of the trained deep neural network to reduce a cost function that measures an error between predicted anatomical object locations and ground truth anatomical object locations in a set of training data.

12. The method of claim 1, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

reducing a number of nodes in each of a plurality of layers of the trained deep neural network by determining a subset of nodes in each layer that linearly approximates all of the nodes in that layers, resulting in a first approximation of the trained deep neural network; and

reducing a number of non-zero weights in a respective filter for each of the nodes in each of the plurality of layers of the first approximation of the trained deep neural network, resulting in a second approximation of the trained deep neural network.

13. The method of claim 1, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

for each of a plurality of nodes in each of a plurality of layers of the trained deep neural network, reconstructing a trained weight matrix for the node using 1-D Haar wavelet bases and wavelet coefficients.

14. The method of claim 13, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network further comprises:

for each of the plurality of nodes in each of the plurality of layers of the trained deep neural network, reducing a number of wavelet coefficients used to reconstruct the trained weight matrix for the node.

15. The method of claim 14, wherein detecting the anatomical object in a received medical image of a patient using the approximation of the trained deep neural network comprises:

storing an integral image of the received medical image in a look-up table;

calculating, for each of a plurality of image patches in the received medical image, a respective multiplication result of multiplying the image patch by the 1-D Haar wavelet bases and the transposed Haar 1-D wavelets using look-up operations from the integral image stored in the look-up table; and

for each node of a first hidden layer in the approximation of the trained deep neural network, calculating a Frobenius inner product of the wavelet coefficients for that node and the respective multiplication result calculated for each of the plurality of image patches.

16. The method of claim 14, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network further comprises:

for each of the plurality of layers of the trained deep neural network, applying principal component analysis (PCA) to the space of the wavelet coefficients over all of the plurality of nodes for the layer.

17. The method of claim 1, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

reconstructing a respective trained weight matrix for each of a plurality of nodes in a current layer of the trained deep neural network using 1-D Haar wavelet bases and respective wavelet coefficients and reducing a number of wavelet coefficients used to reconstruct each respective trained weight matrix; and

re-training the approximation of the trained deep neural network resulting from reconstructing the respective trained weight matrix for each of the plurality of nodes in the current layer of the trained deep neural network and reducing the number of the wavelet coefficients used to reconstruct each respective trained weight matrix.

18. The method of claim 1, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

for each of a plurality of layers in the trained deep neural network, applying principal component analysis (PCA) to a space of trained weight matrices over all of a plurality of nodes in that layer.

19. An apparatus for anatomical object detection in a medical image comprising:

means for training a deep neural network to detect the anatomical object in medical images;

means for calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network; and

means for detecting the anatomical object in a received medical image of a patient using the approximation of the trained deep neural network.

20. The apparatus of claim 19, wherein the means for training a deep neural network to detect the anatomical object in medical images comprises means for training a respective filter for each of a plurality of nodes in each of a plurality of layers of the deep neural network, wherein each respective filter is a weight matrix comprising a plurality of weights that weight node outputs of the nodes of a previous one of the plurality of layers, and the means for calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

means for sparsifying the weights of the filters for each of the plurality of layers of the trained deep neural network.

21. The apparatus of claim 20, wherein the means for sparsifying the weights of the filters for each of the plurality of layers of the trained deep neural network comprises:

means for reducing a number of non-zero weights in each filter for each of the plurality of layers in the trained deep neural network by setting a predetermined percentage of non-zero weights with lowest magnitudes in each filter equal to zero; and

means for refining the remaining non-zero weights in each filter for each of the plurality of layers to alleviate an effect of reducing the number of non-zero weights in each filter.

22. The apparatus of claim 20, wherein the means for sparsifying the weights of the filters for each of the plurality of layers of the trained deep neural network comprises:

means for performing re-weighted L1-norm regularization on the weights of the filters for each of the plurality layers of the trained deep neural network, wherein the re-weighted L1-norm regularization drives a plurality of non-zero weights of the filters to zero; and

means for refining the remaining non-zero weights in the filters for each of the plurality of layers to alleviate an effect of driving the plurality of non-zero weights to zero.

23. The apparatus of claim 19, wherein the means for calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

means for determining a subset of nodes of a plurality nodes in each layer of the trained deep neural network that linearly approximate the plurality of nodes in that layer of the trained deep neural network and removing the plurality of nodes in each layer that are not in subset of nodes from the trained deep neural network; and

means for updating weights for each layer of the trained deep neural network based on the subset of nodes remaining in a preceding layer of the trained deep neural network.

24. The apparatus of claim 23, wherein the means for calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network further comprises:

means for refining the approximation of the trained deep neural network by performing one or more iterations of back-propagation on the approximation of the trained deep neural network to reduce a cost function that measures an error between predicted anatomical object locations and ground truth anatomical object locations in a set of training data.

25. The apparatus of claim 19, wherein the means for calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

means for reducing a number of nodes in each of a plurality of layers of the trained deep neural network by determining a subset of nodes in each layer that linearly approximates all of the nodes in that layers, resulting in a first approximation of the trained deep neural network; and

means for reducing a number of non-zero weights in a respective filter for each of the nodes in each of the plurality of layers of the first approximation of the trained deep neural network, resulting in a second approximation of the trained deep neural network.

26. The apparatus of claim 19, wherein the means for calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

means for reconstructing a respective trained weight matrix for each of a plurality of nodes in each of a plurality of layers of the trained deep neural network using 1-D Haar wavelet bases and wavelet coefficients.

27. The apparatus of claim 26, wherein the means for calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network further comprises:

means for reducing a number of wavelet coefficients used to reconstruct the respective trained weight matrix for each of the plurality of nodes in each of the plurality of layers of the trained deep neural network.

28. A non-transitory computer readable medium storing computer program instructions for anatomical object detection in a medical image, the computer program instructions defining operations comprising:

training a deep neural network to detect the anatomical object in medical images;

calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network; and

detecting the anatomical object in a received medical image of a patient using the approximation of the trained deep neural network.

29. The non-transitory computer readable medium of claim 28, wherein training a deep neural network to detect the anatomical object in medical images comprises training a respective filter for each of a plurality of nodes in each of a plurality of layers of the deep neural network, wherein each respective filter is a weight matrix comprising a plurality of weights that weight node outputs of the nodes of a previous one of the plurality of layers, and calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

sparsifying the weights of the filters for each of the plurality of layers of the trained deep neural network.

30. The non-transitory computer readable medium of claim 29, wherein sparsifying the weights of the filters for each of the plurality of layers of the trained deep neural network comprises:

reducing a number of non-zero weights in each filter for each of the plurality of layers in the trained deep neural network by setting a predetermined percentage of non-zero weights with lowest magnitudes in each filter equal to zero; and

refining the remaining non-zero weights in each filter for each of the plurality of layers to alleviate an effect of reducing the number of non-zero weights in each filter.

31. The non-transitory computer readable medium of claim 29, wherein sparsifying the weights of the filters for each of the plurality of layers of the trained deep neural network comprises:

performing re-weighted L1-norm regularization on the weights of the filters for each of the plurality layers of the trained deep neural network, wherein the re-weighted L1-norm regularization drives a plurality of non-zero weights of the filters to zero; and

refining the remaining non-zero weights in the filters for each of the plurality of layers to alleviate an effect of driving the plurality of non-zero weights to zero.

32. The non-transitory computer readable medium of claim 28, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

determining a subset of nodes of a plurality nodes in a current layer of the trained deep neural network that linearly approximate the plurality of nodes in the current layer of the trained deep neural network and removing the plurality of nodes in the current layer that are not in subset of nodes from the trained deep neural network; and

updating weights for a next layer of the trained deep neural network based on the subset of nodes remaining in the current layer of the trained deep neural network.

33. The non-transitory computer readable medium of claim 32, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network further comprises:

repeating the steps of determining a subset of nodes of a plurality nodes in a current layer of the trained deep neural network that linearly approximate the plurality of nodes in the current layer of the trained deep neural network and removing the plurality of nodes in the current layer that are not in subset of nodes from the trained deep neural network and updating weights for a next layer of the trained deep neural network based on the subset of nodes remaining in the current layer of the trained deep neural network, for each of a plurality of layers in the trained deep neural network, resulting in an initial approximation of the trained deep neural network; and

refining the initial approximation of the trained deep neural network by performing one or more iterations of back-propagation on the initial approximation of the trained deep neural network to reduce a cost function that measures an error between predicted anatomical object locations and ground truth anatomical object locations in a set of training data.

34. The non-transitory computer readable medium of claim 28, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

reducing a number of nodes in each of a plurality of layers of the trained deep neural network by determining a subset of nodes in each layer that linearly approximates all of the nodes in that layers, resulting in a first approximation of the trained deep neural network; and

reducing a number of non-zero weights in a respective filter for each of the nodes in each of the plurality of layers of the first approximation of the trained deep neural network, resulting in a second approximation of the trained deep neural network.

35. The non-transitory computer readable medium of claim 28, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

for each of a plurality of nodes in each of a plurality of layers of the trained deep neural network, reconstructing a trained weight matrix for the node using 1-D Haar wavelet bases and wavelet coefficients.

36. The non-transitory computer readable medium of claim 35, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network further comprises:

for each of the plurality of nodes in each of the plurality of layers of the trained deep neural network, reducing a number of wavelet coefficients used to reconstruct the trained weight matrix for the node.

37. The non-transitory computer readable medium of claim 36, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network further comprises:

for each of the plurality of layers of the trained deep neural network, applying principal component analysis (PCA) to the space of the wavelet coefficients over all of the plurality of nodes for the layer.

38. The non-transitory computer readable medium of claim 28, wherein calculating an approximation of the trained deep neural network that reduces the computational complexity of the trained deep neural network comprises:

for each of a plurality of layers in the trained deep neural network, applying principal component analysis (PCA) to a space of trained weight matrices over all of a plurality of nodes in that layer.