MULTIPLE LANDMARK DETECTION IN MEDICAL IMAGES BASED ON HIERARCHICAL FEATURE LEARNING AND END-TO-END TRAINING

• US 20170330319A1
• Filed: 05/10/2017
• Published: 11/16/2017
• Est. Priority Date: 05/12/2016
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Xu, Daguang, Xiong, Tao, Liu, David, Zhou, Shaohua Kevin, Chen, Mingqing, Comaniciu, Dorin
• Granted Patent: US 10,210,613 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 5/4887 Locating particular structures in or on the body
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods

Abstract

The present embodiments relate to detecting multiple landmarks in medical images. By way of introduction, the present embodiments described below include apparatuses and methods for detecting landmarks using hierarchical feature learning with end-to-end training. Multiple neural networks are provided with convolutional layers for extracting features from medical images and with a convolutional layer for learning spatial relationships between the extracted features. Each neural network is trained to detect different landmarks using a different resolution of the medical images, and the convolutional layers of each neural network are trained together with end-to-end training to learn appearance and spatial configuration simultaneously. The trained neural networks detect multiple landmarks in a test image iteratively by detecting landmarks at different resolutions, using landmarks detected a lesser resolutions to detect additional landmarks at higher resolutions.

Claims

1. A method of deep learning for multiple landmark detection, the method comprising:

receiving a plurality of training images;

training a first deep neural network at a first resolution of the training images, the training comprising;

learning locations of a first plurality of landmarks; and

learning the spatial relationships between the locations of the first plurality of landmarks;

training a second deep neural network at a second resolution of the training images, the training comprising;

learning locations of a second plurality of landmarks; and

learning spatial relationships between the locations of the second plurality of landmarks.

2. The method of claim 1, wherein the locations of the second plurality of landmarks are learnt at the second resolution based on the locations of the first plurality of landmarks at the first resolution.

3. The method of claim 1, wherein training the first deep neural network comprises simultaneously learning a plurality of landmark probability maps for the first plurality of landmarks and training the second deep neural network comprises simultaneously learning a plurality of landmark probability maps for the second plurality of landmarks.

4. The method of claim 3, wherein a different landmark probability map is learned for each landmark.

5. The method of claim 3, wherein simultaneously learning the plurality of landmark probability maps for the first plurality of landmarks and simultaneously learning the plurality of landmark probability maps for the second plurality of landmarks comprises message passing between landmark probability maps of neighboring landmarks.

6. The method of claim 5, wherein message passing between landmark probability maps of neighboring landmarks is configured to compensate for missing response information in each of the landmark probability maps.

7. The method of claim 1, wherein the first deep neural network and the second deep neural network each comprise:

one or more convolutional layers configured to extract features from the training images; and

a convolutional layer configured to map the spatial configuration of the features,

wherein the convolutional layers are trained simultaneously.

8. The method of claim 7, wherein learning the locations of the first plurality of landmarks comprises learning an appearance of each of the first plurality of landmarks.

9. The method of claim 1, further comprising:

receiving a test image;

detecting, using the first trained deep neural network, the locations of the first plurality of landmarks at a first resolution of the test image; and

detecting, using the second trained deep neural network, the locations of the second plurality of landmarks at a second resolution of the test image.

10. A system for detecting multiple landmarks in medical image data, the system comprising:

a scanner configured to capture medical image data;

at least one processor; and

at least one memory including computer program code for one or more programs, the at least one memory and the computer program code configured to, with the at least one processor, cause the system to;

receive the medical image data captured by the scanner;

detect, using a trained first artificial agent and a trained second artificial agent, multiple landmarks in the medical image data at different resolutions of the medical image data.

11. The system of claim 10, wherein the trained first artificial agent and the trained second artificial agent are deep neural networks.

12. The system of claim 11, training the first artificial agent and the second artificial agent comprises:

learning convolutional layers of the artificial agents to extract features from the plurality of training images; and

learning convolutional layers of the artificial agents to map spatial relationships between features of the plurality of training images.

13. The system of claim 12, wherein the convolutional layers are trained simultaneously.

14. The system of claim 10, wherein the at least one memory and the computer program code is further configured to, with the at least one processor, cause the system to:

receive a plurality of a plurality of training images; and

train, based on the plurality of training images, the first artificial agent to detect multiple landmarks at a first resolution and the second artificial agent to detect multiple landmarks at a second resolution.

15. A method for multiple landmark detection, the method comprising:

receiving, from a medical scanner, medical image data;

identifying, using a first learned deep neural network, a first subset of a plurality of landmarks from the medical image data at a first resolution;

identifying, using a second learned deep neural network, a second subset of the plurality of landmarks from the medical image data at a second resolution; and

displaying a medical image from the medical image data identifying the identified first subset of landmarks and the identified second subset of landmarks.

16. The method of claim 15, wherein the medical image data is two-dimensional image data, and wherein each pixel is one landmark of the plurality of landmarks.

17. The method of claim 15, wherein the medical image data is three-dimensional image data, and wherein each voxel is one landmark of the plurality of landmarks.

18. The method of claim 15, wherein locations of the first subset of landmarks is used to identify the second subset of landmarks.

19. The method of claim 15, wherein the first resolution is lower than the second resolution, and wherein the first subset of landmarks includes fewer landmarks than the second subset of landmarks.

20. The method of claim 19, wherein a first number landmarks in the first subset of landmarks and a second number of landmarks in the second subset of landmarks are selected such that identifying the first subset of landmarks and identifying the second subset of landmarks requires substantially the same computational complexity.