System and Method for Automatic Detection, Localization, and Semantic Segmentation of Anatomical Objects

• US 20190311478A1
• Filed: 06/29/2017
• Published: 10/10/2019
• Est. Priority Date: 07/08/2016
• Status: Active Grant

• Current Assignee: Avent Incorporated (Avanos Medical, Inc.)
• Original Assignee: Avent Incorporated (Avanos Medical, Inc.)
• Inventors: Avendi, Michael R., Hsu, Kenneth C., Duffy, Shane A., Fantasia, Dominique J., Khalaj, Steve S., Somji, Hasnain, Bui, Aimee T., Shahriari, Shirzad, Avila, Ambar A., Mulders, Joost L.
• Granted Patent: US 11,151,721 B2
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/251 of input or preprocessed data
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/75 involving models
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/803 of input or preprocessed data
• CPC Class Codes: G06V 10/82 using neural networks
• CPC Class Codes: G06V 2201/031 of internal organs
• CPC Class Codes: G06V 30/274 Syntactic or semantic context, e.g. balancing

Abstract

The present invention is directed to a system and method for automatic detection, localization, and semantic segmentation of at least one anatomical object in a parameter space of an image generated by an imaging system. The method includes generating the image via the imaging system and providing the image of the anatomical object and surrounding tissue to a processor. Further, the method includes developing and training a parameter space deep learning network comprising convolutional neural networks to automatically detect the anatomical object and the surrounding tissue of the parameter space of the image. The method also includes automatically locating and segmenting, via additional convolutional neural networks, the anatomical object and surrounding tissue of the parameter space of the image. Moreover, the method includes automatically labeling the identified anatomical object and surrounding tissue on the image. Thus, the method also includes displaying the labeled image to a user in real time.

Claims

1. A method for automatic detection, localization, and segmentation of at least one anatomical object in a parameter space of an image generated by an imaging system, the method comprising:

providing the image of the anatomical object and surrounding tissue to a processor;

developing and training a parameter space deep learning network comprising one or more convolutional neural networks to automatically detect the anatomical object and the surrounding tissue of the parameter space of the image;

automatically locating and segmenting, via an additional convolutional neural network, the anatomical object and the surrounding tissue of the parameter space of the image;

automatically labeling the anatomical object and the surrounding tissue on the image; and

displaying the labeled image to a user.

3. The method of claim 1, wherein the image is a normalized image, wherein the normalized image is pre-processed to adjust for size, intensity, contrast, texture, or a combination thereof.

5. The method of claim 1, wherein the image is provided from a frame in a video stream, wherein the frame is stored locally or on a cloud service.

7. The method of claim 1, wherein automatically detecting the anatomical object and automatically locating and segmenting the anatomical object occur simultaneously in a single step and utilize a central processing unit (CPU)-based computer, a tablet, or a mobile device.

9. The method of claim 1, wherein automatically locating and segmenting the anatomical object involves estimating a shape of the anatomical object as an ellipse, and wherein the one or more convolutional neural networks, the additional convolutional neural network, or both utilize 8-bit integer or 1-bit binary number weights.

11. The method of claim 1, further comprising training the deep learning network to automatically detect the parameter space of the image containing the anatomical object and surrounding tissue via ground truth data,

wherein developing and training the deep learning network to automatically detect the anatomical object and surrounding tissue of the parameter space of the image further comprises:

scanning and collecting a dataset of images of the anatomical object and surrounding tissue from a plurality of patient,

annotating the dataset of images based on user input to create the ground truth data,

dividing the dataset of images and the ground truth data into a training dataset and a validation dataset, and

utilizing the training dataset to train the deep learning network.

13. The method of claim 11, further comprising normalizing the dataset of images by pre-processing the dataset of images to adjust for size, intensity, contrast, texture, or a combination thereof.

14. The method of claim 11, wherein the dataset of images is collected from multiple data imaging systems.

15. The method of claim 11, wherein the dataset of images comprises frames collected from at least one video stream, wherein the frames are stored locally or on a cloud service.

17. The method of claim 15, wherein annotating the dataset of images based on user input to create the ground truth data comprises:

performing an optical flow calculation of frame to frame pixel movements,

facilitating unsupervised learning to cluster the frame to frame pixel movements across the at least one video stream into groups, and

providing the groups of pixel movements to the user for identification and labeling across a range of frames.

18. The method of claim 11, wherein utilizing the training dataset to train the deep learning network further comprises optimizing a cost function to minimize an error between an output of the deep learning network and the ground truth data, wherein optimizing the cost function to minimize the error further comprises utilizing a stochastic gradient descent (SGD) algorithm that iteratively processes portions of the ground truth data and adjusts one or more parameters of the deep learning network based on the error between the output of the deep learning network and the ground truth data.

20. The method of claim 18, further comprising, after optimizing the cost function, utilizing the deep learning network in real-time to automatically provide predictions on the validation data and comparing the predictions with the ground truth data.

21. The method of claim 11, wherein annotating the dataset of images based on user input to create the ground truth data further comprises manually identifying and annotating the anatomical object and surrounding tissue in each image of the dataset.

22. The method of claim 1, wherein automatically locating and segmenting the anatomical object and the surrounding tissue of the parameter space of the image further comprises segmenting the anatomical object and the surrounding tissue via semantic segmentation.

23. The method of claim 1, further comprising initially training the deep learning network to automatically detect the parameter space of the imaging system containing the anatomical object and the surrounding tissue offline, further comprising continuously training the deep learning network to automatically detect the parameter space of the imaging system containing the anatomical object and the surrounding tissue online.

25. The method of claim 1, wherein labeling the anatomical object and the surrounding tissue on the image further comprises at least one of outlining or shading at least one of the anatomical object and the surrounding tissue on the image or overlaying a descriptive label atop each of the at least one anatomical object and the surrounding tissue, wherein automatically locating and segmenting the anatomical object and the surrounding tissue of the parameter space of the image via the additional convolutional network comprises extracting an area of interest around the anatomical object from the image, and wherein the image of the anatomical object and surrounding tissue is automatically cropped prior to providing the image to the processor.

2. (canceled)

4. (canceled)

6. (canceled)

8. (canceled)

10. (canceled)

12. (canceled)

16. (canceled)

19. (canceled)

24. (canceled)

26. (canceled)

27. (canceled)

28. An imaging system, comprising:

at least one processor configured to perform one or more operations, the one or more operations comprising;

receiving an image of at least one anatomical object and surrounding tissue,

developing and training a parameter space deep learning network having one or more convolutional neural networks to automatically detect the anatomical object and the surrounding tissue of the parameter space of the image,

automatically locating and segmenting, via an additional deep learning network, the anatomical object and the surrounding tissue of the parameter space of the image, and

automatically labeling the anatomical object and the surrounding tissue on the image; and

a user display configured to display the labeled image to a user.

29. The system of claim 28, wherein the imaging system comprises at least one of an ultrasound imaging system, a computer tomography (CT) scanner, or a magnetic resonance imaging (MRI) scanner, and wherein the image is a normalized image wherein the normalized image is pre-processed to adjust for size, intensity, contrast, texture, or a combination thereof.

32. The system of claim 28, wherein the image is provided from a frame in a video stream, wherein the frame is stored locally or on a cloud service.

34. The system of claim 28, wherein automatically detecting the anatomical object and automatically locating and segmenting the anatomical object occur simultaneously in a single step, wherein automatically detecting the anatomical object and automatically locating and segmenting the anatomical object utilize a central processing unit (CPU)-based computer, a tablet, or a mobile device.

36. The system of claim 28, wherein automatically locating and segmenting the anatomical object involves estimating a shape of the anatomical object as an ellipse, and wherein the parameter space deep learning network, the additional deep learning network, or both utilize 8-bit integer or 1-bit binary number weights.

38. The system of claim 28, wherein automatically locating and segmenting the anatomical object and the surrounding tissue of the parameter space of the image via the additional deep learning network comprises extracting an area of interest around the anatomical object from the image, wherein the system is configured to automatically crop the image of the at east one anatomical object and surrounding tissue before the image is received by the at least one processor.

30. (canceled)

31. (canceled)

33. (canceled)

35. (canceled)

37. (canceled)

39. (canceled)

40. A method for automatic detection, localization, and semantic segmentation of at least one anatomical object in a parameter space of an image generated by an imaging system, the method comprising:

providing the image of the anatomical object and surrounding tissue to a processor;

developing and training a parameter space deep learning network to automatically detect the anatomical object and the surrounding tissue of the parameter space of the image;

automatically locating and semantically segmenting, via one or more deep learning algorithms, the anatomical object and the surrounding tissue of the parameter space of the image;

automatically labeling the anatomical object and the surrounding tissue on the image; and

displaying the labeled image to a user.

43. The method of claim 40, wherein the image is a normalized image, wherein the normalized image is pre-processed to adjust for size, intensity, contrast, texture, or a combination thereof.

45. The method of claim 40, wherein the image is provided from a frame in a video stream, wherein the frame is stored locally or on a cloud service.

47. The method of claim 40, wherein automatically detecting the anatomical object and automatically locating and segmenting the anatomical object occur simultaneously in a single step, wherein automatically detecting the anatomical object and automatically locating and segmenting the anatomical object utilize a central processing unit (CPU)-based computer, a tablet, or a mobile device, and wherein the parameter space deep learning network, the one or more deep learning algorithms, or both utilize 8-bit integer or 1-bit binary number weights.

49. The method of claim 40, wherein automatically locating and segmenting the anatomical object involves estimating a shape of the anatomical object as an ellipse.

51. The method of claim 40, further comprising training the parameter space deep learning network to automatically detect the parameter space of the image containing the anatomical object and surrounding tissue via ground truth data, wherein developing and training the parameter space deep learning network to automatically detect the anatomical object and surrounding tissue of the parameter space of the image further comprise:

scanning and collecting a dataset of images of the anatomical object and surrounding tissue from a plurality of patients,

annotating the dataset of images based on user input to create the ground truth data,

dividing the dataset of images and the ground truth data into a training dataset and a validation dataset, and

utilizing the training dataset to train the one or more deep learning algorithms.

53. The method of claim 51, further comprising normalizing the dataset of images by pre-processing the dataset of images to adjust for size, intensity, contrast, texture, or a combination thereof, wherein the dataset of images is collected from multiple data imaging systems, wherein the dataset of images comprises frames collected from at least one video stream, wherein the frames are stored locally or on a cloud service.

57. The method of claim 51, wherein annotating the dataset of images based on user input to create the ground truth data comprises:

performing an optical flow calculation of frame to frame pixel movements,

facilitating unsupervised learning to cluster the frame to frame pixel movements across the at least one video stream into groups, and

providing the groups of pixel movements to the user for identification and labeling across a range of frames, and

wherein utilizing the training dataset to train the parameter space deep learning network further comprises optimizing a cost function to minimize an error between an output of the one or more deep learning algorithms and the ground truth data.

59. The method of claim 40, wherein automatically locating and segmenting the anatomical object and the surrounding tissue of the parameter space of the image via one or more deep learning algorithms comprises extracting an area of interest around the anatomical object from the image, wherein the image of the anatomical object and surrounding tissue is automatically cropped prior to providing the image to the processor.

41. (canceled)

42. (canceled)

44. (canceled)

46. (canceled)

48. (canceled)

50. (canceled)

52. (canceled)

54. (canceled)

55. (canceled)

56. (canceled)

58. (canceled)

60. (canceled)