METHOD AND SYSTEM FOR ARTIFICIAL INTELLIGENCE BASED MEDICAL IMAGE SEGMENTATION

• US 20190205606A1
• Filed: 07/19/2017
• Published: 07/04/2019
• Est. Priority Date: 07/21/2016
• Status: Active Grant

• Current Assignee: Siemens Healthineers AG (Siemens AG)
• Original Assignee: Siemens Healthcare GMBH (Siemens AG)
• Inventors: Zhou, Shaohua Kevin, Chen, Mingqing, Ding, Hui, Georgescu, Bogdan, Gulsun, Mehmet Akif, Kim, Tae Soo, Kiraly, Atilla Peter, Lu, Xiaoguang, Park, Jin-hyeong, Sharma, Puneet, Sun, Shanhui, Xu, Daguang, Xu, Zhoubing, Zheng, Yefeng
• Granted Patent: US 10,878,219 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/285 Selection of pattern recognition techniques, e.g. of classifiers in a multi-classifier system
• CPC Class Codes: G06N 3/044 Recurrent networks, e.g. Hopfield networks
• CPC Class Codes: G06N 3/045 Combinations of networks
• CPC Class Codes: G06N 3/084 Backpropagation, e.g. using gradient descent
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/7553 based on shape, e.g. active shape models [ASM]
• CPC Class Codes: G06V 20/695 Preprocessing, e.g. image segmentation
• CPC Class Codes: G06V 2201/031 of internal organs
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing

Abstract

Methods and systems for artificial intelligence based medical image segmentation are disclosed. In a method for autonomous artificial intelligence based medical image segmentation, a medical image of a patient is received. A current segmentation context is automatically determined based on the medical image and at least one segmentation algorithm is automatically selected from a plurality of segmentation algorithms based on the current segmentation context. A target anatomical structure is segmented in the medical image using the selected at least one segmentation algorithm.

Claims

1. A method for autonomous artificial intelligence based medical image segmentation, comprising:

receiving a medical image of a patient;

automatically determining a current segmentation context based on the medical image and selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context; and

segmenting a target anatomical structure in the medical image using the selected at least one segmentation algorithm.

2. The method of claim 1, wherein automatically determining a current segmentation context based on the medical image and selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context comprises:

automatically determining the current segmentation context and selecting the at least one segmentation algorithm by a trained master segmentation artificial agent.

3. The method of claim 1, wherein automatically determining a current segmentation context based on the medical image and selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context comprises:

automatically identifying the target anatomical structure based on a view of the medical image currently being visualized on a display device; and

automatically selecting the at least one segmentation algorithm based on the automatically identified target anatomical structure.

4. The method of claim 3, wherein automatically identifying the target anatomical structure based on a view of the medical image currently being visualized on a display device comprises:

automatically detecting one or more anatomical structures in the view of the medical image currently being visualized on the display device using a trained machine learning based classifier.

5. The method of claim 3, wherein automatically selecting the at least one segmentation algorithm based on the automatically identified target anatomical structure comprises:

selecting at least one segmentation algorithm specialized for segmentation of the automatically identified target anatomical structure from the plurality of segmentation algorithms.

6. The method of claim 5, wherein selecting at least one segmentation algorithm specialized for segmentation of the automatically identified target anatomical structure from the plurality of segmentation algorithms comprises:

selecting the at least one segmentation algorithm from multiple segmentation algorithms specialized for segmentation of the automatically identified target anatomical structure based on at least one or an imaging modality of the medical image or image characteristics of the medical image.

7. The method of claim 1, wherein automatically determining a current segmentation context based on the medical image and selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context comprises:

automatically extracting image characteristics of the medical image and selecting the at least one segmentation algorithm from the plurality of segmentation algorithms based on the extracted image characteristics.

8. The method of claim 7, wherein automatically extracting image characteristics of the medical image and selecting the at least one segmentation algorithm from the plurality of segmentation algorithms based on the extracted image characteristics comprises:

inputting the medical image to a trained deep neural network that automatically extracts learned image characteristics from the medical image and predicts a best segmentation algorithm or combination of segmentation algorithms from the plurality of segmentation algorithms for performing segmentation of the target anatomical structure in the medical image based on the extracted image characteristics.

9. The method of claim 8, wherein the trained deep neural network is trained based on medical image training samples and synthetic training samples generated at a plurality if noise levels from the medical image training samples.

10. The method of claim 1, wherein selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context comprises selecting multiple segmentation algorithms from the plurality of segmentation algorithms based on the current segmentation context, and segmenting a target anatomical structure in the medical image using the selected at least one segmentation algorithm comprises:

segmenting the target anatomical structure in the medical image using each of the selected multiple segmentation algorithms; and

fusing segmentation results from the selected multiple segmentation algorithms to generate final segmentation result for the target anatomical structure in the medical image.

11. The method of claim 1, wherein the plurality of segmentation algorithms include a plurality of deep learning based segmentation algorithms each with a respective deep network architecture trained for a specific segmentation task.

12. The method of claim 1, wherein segmenting a target anatomical structure in the medical image using the selected at least one segmentation algorithm comprises:

segmenting the target anatomical structure in the medical image using a trained deep neural network with one or more integrated priors.

13. The method of claim 1, wherein the selected at least one segmentation algorithm comprises a deep reinforcement learning based segmentation algorithm, and segmenting a target anatomical structure in the medical image using the selected at least one segmentation algorithm comprises:

iteratively adjusting an statistical shape model representing a shape of the target anatomical structure in the medical image by selecting, at each iteration, an action corresponding to an adjustment of a parameter of the statistical shape model based on action values calculated using deep neural network trained using deep reinforcement learning.

14. The method of claim 1, wherein segmenting a target anatomical structure in the medical image using the selected at least one segmentation algorithm comprises:

generating a plurality of Gaussian distributed probability maps at different scales from the medical image using a trained deep image-to-image network;

combining the plurality of Gaussian probability maps at the different scales into a combined probability map; and

extracting a segmented boundary of the target anatomical structure from the medical image based on the combined probability map.

15. The method of claim 1, wherein the selected at least one segmentation algorithm comprises a deep learning partial inference based segmentation algorithm, and segmenting a target anatomical structure in the medical image using the selected at least one segmentation algorithm comprises:

generating an initial segmentation mask for the target anatomical structure from the medical image using a trained initial convolutional encoder decoder (CED);

constructing a unified tensor combining a current segmentation mask and the medical image; and

generating an updated segmentation mask for the target anatomical structure using a trained partial inference CED.

16. The method of claim 1, wherein segmenting a target anatomical structure in the medical image using the selected at least one segmentation algorithm comprises:

fitting an active shape model to a shape of the target anatomical structure in the medical image by using a recurrent neural network (RNN) to adjust locations of points on the active shape model.

17. An apparatus for autonomous artificial intelligence based medical image segmentation, comprising:

means for receiving a medical image of a patient;

means for automatically determining a current segmentation context based on the medical image and selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context; and

means for segmenting a target anatomical structure in the medical image using the selected at least one segmentation algorithm.

18. The apparatus of claim 17, wherein the means for automatically determining a current segmentation context based on the medical image and selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context comprises:

means for automatically identifying the target anatomical structure based on a view of the medical image currently being visualized on a display device; and

means for automatically selecting the at least one segmentation algorithm based on the automatically identified target anatomical structure.

19. The apparatus of claim 18, wherein the means for automatically selecting the at least one segmentation algorithm based on the automatically identified target anatomical structure comprises:

means for selecting at least one segmentation algorithm specialized for segmentation of the automatically identified target anatomical structure from the plurality of segmentation algorithms.

20. The apparatus of claim 19, wherein the means for selecting at least one segmentation algorithm specialized for segmentation of the automatically identified target anatomical structure from the plurality of segmentation algorithms comprises:

means for selecting the at least one segmentation algorithm from multiple segmentation algorithms specialized for segmentation of the automatically identified target anatomical structure based on at least one or an imaging modality of the medical image or image characteristics of the medical image.

21. The apparatus of claim 17, wherein the means for automatically determining a current segmentation context based on the medical image and selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context comprises:

means for automatically extracting image characteristics of the medical image and selecting the at least one segmentation algorithm from the plurality of segmentation algorithms based on the extracted image characteristics.

22. The apparatus of claim 17, wherein the means for selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context comprises means for selecting multiple segmentation algorithms from the plurality of segmentation algorithms based on the current segmentation context, and the means for segmenting a target anatomical structure in the medical image using the selected at least one segmentation algorithm comprises:

means for segmenting the target anatomical structure in the medical image using each of the selected multiple segmentation algorithms; and

means for fusing segmentation results from the selected multiple segmentation algorithms to generate final segmentation result for the target anatomical structure in the medical image.

23. A non-transitory computer readable medium storing computer program instructions for autonomous artificial intelligence based medical image segmentation, the computer program instructions when executed by a processor perform operations comprising:

receiving a medical image of a patient;

automatically determining a current segmentation context based on the medical image and selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context; and

segmenting a target anatomical structure in the medical image using the selected at least one segmentation algorithm.

24. The non-transitory computer readable medium of claim 23, wherein automatically determining a current segmentation context based on the medical image and selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context comprises:

automatically identifying the target anatomical structure based on a view of the medical image currently being visualized on a display device; and

automatically selecting the at least one segmentation algorithm based on the automatically identified target anatomical structure.

25. The non-transitory computer readable medium of claim 24, wherein automatically selecting the at least one segmentation algorithm based on the automatically identified target anatomical structure comprises:

selecting at least one segmentation algorithm specialized for segmentation of the automatically identified target anatomical structure from the plurality of segmentation algorithms.

26. The non-transitory computer readable medium of claim 25, wherein selecting at least one segmentation algorithm specialized for segmentation of the automatically identified target anatomical structure from the plurality of segmentation algorithms comprises:

selecting the at least one segmentation algorithm from multiple segmentation algorithms specialized for segmentation of the automatically identified target anatomical structure based on at least one or an imaging modality of the medical image or image characteristics of the medical image.

27. The non-transitory computer readable medium of claim 23, wherein automatically determining a current segmentation context based on the medical image and selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context comprises:

automatically extracting image characteristics of the medical image and selecting the at least one segmentation algorithm from the plurality of segmentation algorithms based on the extracted image characteristics.

28. The non-transitory computer readable medium of claim 27, wherein automatically extracting image characteristics of the medical image and selecting the at least one segmentation algorithm from the plurality of segmentation algorithms based on the extracted image characteristics comprises:

inputting the medical image to a trained deep neural network that automatically extracts learned image characteristics from the medical image and predicts a best segmentation algorithm or combination of segmentation algorithms from the plurality of segmentation algorithms for performing segmentation of the target anatomical structure in the medical image based on the extracted image characteristics.

29. The non-transitory computer readable medium of claim 23, wherein selecting at least one segmentation algorithm from a plurality of segmentation algorithms based on the current segmentation context comprises selecting multiple segmentation algorithms from the plurality of segmentation algorithms based on the current segmentation context, and segmenting a target anatomical structure in the medical image using the selected at least one segmentation algorithm comprises:

segmenting the target anatomical structure in the medical image using each of the selected multiple segmentation algorithms; and

fusing segmentation results from the selected multiple segmentation algorithms to generate final segmentation result for the target anatomical structure in the medical image.

30. The non-transitory computer readable medium of claim 23, wherein the plurality of segmentation algorithms include a plurality of deep learning based segmentation algorithms each with a respective deep network architecture trained for a specific segmentation task.