Intelligent Multi-scale Medical Image Landmark Detection

• US 20170116497A1
• Filed: 01/03/2017
• Published: 04/27/2017
• Est. Priority Date: 09/16/2015
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Georgescu, Bogdan, Ghesu, Florin Cristian, Zheng, Yefeng, Neumann, Dominik, Mansi, Tommaso, Comaniciu, Dorin, Liu, Wen, Zhou, Shaohua Kevin
• Granted Patent: US 9,792,531 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 5/0044 for the heart
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 6/5217 extracting a diagnostic or physiological parameter from medical diagnostic data
• CPC Class Codes: A61B 8/0883 for diagnosis of the heart
• CPC Class Codes: A61B 8/4416 related to combined acquisition of different diagnostic modalities, e.g. combination of ultrasound and X-ray acquisitions
• CPC Class Codes: A61B 8/483 involving the acquisition of a 3D volume of data
• CPC Class Codes: A61B 8/5223 for extracting a diagnostic or physiological parameter from medical diagnostic data for algorithms to analyse biomedical images G06T7/0012
• CPC Class Codes: G06F 18/214 Generating training patterns; Bootstrap methods, e.g. bagging or boosting
• CPC Class Codes: G06F 18/24 Classification techniques
• CPC Class Codes: G06F 18/2413 based on distances to training or reference patterns
• CPC Class Codes: G06N 3/006 based on simulated virtual individual or collective life forms, e.g. social simulations or particle swarm optimisation [PSO]
• CPC Class Codes: G06N 3/08 Learning methods
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06N 7/01 Probabilistic graphical models, e.g. probabilistic networks
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10088 Magnetic resonance imaging [MRI]
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30048 : Heart; Cardiac
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 7/0012 : Biomedical image inspection
• CPC Class Codes: G06T 7/70 : Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 : using feature-based methods
• CPC Class Codes: G06V 10/225 : based on a marking or identifier characterising the area
• CPC Class Codes: G06V 10/235 : based on user input or interaction
• CPC Class Codes: G06V 10/454 : Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/764 : using classification, e.g. of video objects
• CPC Class Codes: G06V 10/82 : using neural networks
• CPC Class Codes: G06V 2201/031 : of internal organs
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 50/70 : for mining of medical data, e.g. analysing previous cases of other patients

Abstract

Intelligent multi-scale image parsing determines the optimal size of each observation by an artificial agent at a given point in time while searching for the anatomical landmark. The artificial agent begins searching image data with a coarse field-of-view and iteratively decreases the field-of-view to locate the anatomical landmark. After searching at a coarse field-of view, the artificial agent increases resolution to a finer field-of-view to analyze context and appearance factors to converge on the anatomical landmark. The artificial agent determines applicable context and appearance factors at each effective scale.

Claims

1. A method for intelligent multi-scale image parsing, the method comprising:

specifying a state space of an artificial agent for discrete portions of a training image, the state space specified by a parametric space and a scale space for the discrete portions of the training image;

determining a set of actions, the set of actions comprising parametric actions specifying a possible change in the parametric space with respect to the training image and scale actions specifying a possible change in the scale space with respect to the training image;

establishing a reward system based on applying each action of the set of actions and based on at least one target location of the training image; and

learning, by the artificial agent, an optimal action-value function approximator specifying the behavior of the artificial agent to maximize a cumulative future reward value of the reward system, wherein the behavior of the artificial agent is a sequence of actions moving the agent towards the at least one target location of the training image, the sequence of actions comprising at least one scale action.

2. The method of claim 1, further comprising:

applying the learned artificial agent on a test image to automatically parse image content for a landmark location, the applying comprising;

evaluating the optimal action-value function approximator for a current state space;

simultaneously obtaining the optimal action-value function approximator for all possible actions at each current state space; and

applying a reward policy of the optimal action-value function approximator.

3. The method of claim 2, wherein applying the reward policy of the optimal action-value function approximator comprises determining a next action of the artificial agent based on balancing maximization of the cumulative future reward value by actions changing the parametric space and by actions changing the scale space.

4. The method of claim 1, wherein the at least one target location is an anatomical landmark location, and wherein the target location is defined by the position parameters of the landmark anatomical landmark, and wherein a reward value is indicative of a proximity of the state space to the at least one target state.

5. The method of claim 1, wherein learning the optimal action-value function approximator further comprises:

generating an experience memory database including a predefined number of last evaluated state spaces for the training image;

sampling the experience memory database; and

updating parameters of the optimal action-value function approximator based on the experience memory.

6. The method of claim 1, wherein learning the optimal action-value function approximator further comprises:

parameterizing the action-value function approximator using a deep neural network; and

optimizing the action-value function using an episodic trajectory for the training image based on discrete portions of the training image via the state space, wherein the episodic trajectory is indicative of a series actions of the set of actions of the artificial agent.

7. The method of claim 1, wherein the parametric actions change the parametric space of the state space in an upward, a downward, a left, a right, a forward or a backward direction with respect to the training image, and wherein the scale actions change the scale space of the state space by increasing or decreasing a resolution with respect to the training image.

8. The method of claim 7, wherein the set of actions further includes an action in which the state space is unchanged.

9. A method of machine learning for intelligent multi-scale image parsing, the method comprising:

receiving a plurality of training images; and

training, based on the plurality of training images, an artificial agent to parse a test image to identify a landmark location in the test image, wherein training simultaneously trains;

a search strategy model to search for the landmark location by parsing the test image by performing a series of actions, the series of actions comprises changing the position and the scale of a patch of the test image, wherein parsing the test image searches less than the entire test image; and

an appearance model to identify the landmark location in the patch of the test image.

10. The method of claim 9, wherein the search strategy model comprises an optimal action-value function, and wherein searching for the landmark location comprises changing the position of the patch at a first scale, changing the patch scale from the first scale to a second scale, and changing the position of the patch at the second scale.

11. The method of claim 9, wherein the search strategy model comprises an optimal action-value function, and wherein simultaneously training the search strategy model and the appearance model comprises encoding parameters of search strategy model and parameters of the appearance model in a multilayer data representation.

12. The method of claim 11, wherein the multilayer data representation is a deep neural network.

13. The method of claim 12, wherein the search strategy model comprises an optimal action-value function, and wherein training the artificial agent comprises using experience memory from previously parsed patches at different scales to solve for parameters of the deep neural network.

14. The method of claim 9, wherein the search strategy model comprises an optimal action-value function, and wherein training the search strategy model comprises maximizing a future reward using a reward system based on reward values for each position and scale change of the patch, and wherein training the appearance model is based on an annotated landmark location indexed in each of the training images.

15. A method for intelligent multi-scale landmark identification in an image, the method comprising:

receiving image data representing the image; and

automatically parsing, by a learned artificial agent comprising an optimal action-value function, the received image data to identify a landmark location in the image, wherein the learned artificial agent is configured to;

parameterize a patch of the image data in a trained hierarchical data representation, the hierarchical data representation trained by maximizing a future reward of a reward system of the action-value function for each a plurality of available actions to reposition the patch of the image;

determine a sequence of actions from the plurality of available actions to reposition and rescale the patch based on the parameterized patch of the image data; and

identify the landmark location in the repositioned and rescaled patch of the image.

16. The method of claim 15, wherein automatically parsing the received image data comprises performing the sequence of actions to move a location of the patch toward a location of a target patch and to increase the resolution of the patch, the target patch including the landmark location in the image.

17. The method of claim 16, wherein the sequence of actions comprises a path converging on the landmark location by parsing less than the entire image.

18. The method of claim 17, wherein determining the sequence of actions comprises parameterizing at least one previous patch in the hierarchical data representation after repositioning and rescaling the patch.

19. The method of claim 18, wherein the hierarchical data representation is a deep neural network.

20. The method of claim 15, wherein each of the sequence of actions comprises repositioning the patch in an upward, downward, left or right direction in the received image and rescaling the patch to increase the resolution of the patch.