Apparatus and method for visualizing anatomical elements in a medical image

• US 10,383,602 B2
• Filed: 03/17/2015
• Issued: 08/20/2019
• Est. Priority Date: 03/18/2014
• Status: Expired due to Fees

• Current Assignee: Samsung Electronics Co. Ltd.
• Original Assignee: Samsung Electronics Co. Ltd.
• Inventors: Cho, Baek Hwan, Seong, Yeong Kyeong, Kim, Ye Hoon, Kim, Ha Young, Jeon, Joo Hyuk
• Primary Examiner(s): Beg, Samah A
• Application Number:
  US14/660,533
• Publication Number:
  US 20150265251A1
• Time in Patent Office: 1,617 Days
• Field of Search: None
• CPC Class Codes: A61B 8/0825 for diagnosis of the breast, e.g. mammography
• CPC Class Codes: A61B 8/085 for locating body or organic structures, e.g. tumours, calculi, blood vessels, nodules
• CPC Class Codes: A61B 8/483 involving the acquisition of a 3D volume of data
• CPC Class Codes: A61B 8/5215 involving processing of medical diagnostic data
• CPC Class Codes: A61B 8/5292 using additional data, e.g. patient information, image labeling, acquisition parameters
• CPC Class Codes: G06T 2207/10132 Ultrasound image
• CPC Class Codes: G06T 2207/20092 Interactive image processing based on input by user
• CPC Class Codes: G06T 2207/20128 Atlas-based segmentation
• CPC Class Codes: G06T 2207/30068 Mammography; Breast
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling

Abstract

A method of visualizing anatomical elements in a medical includes receiving a medical image; detecting a plurality of anatomical elements from the medical image; verifying a location of each of the plurality of anatomical elements based on anatomical context information including location relationships between the plurality of anatomical elements; adjusting the location relationships between the plurality of anatomical elements; and combining the verified and adjusted information of the plurality of anatomical elements with the medical image.

Claims

1. An apparatus for visualizing anatomical elements in a medical image, the apparatus comprising:

a memory configured to store instructions therein; and

at least one processor, upon execution of the instructions, configured to;

receive a medical image,

detect a plurality of anatomical elements from the medical image,

verify a location of each of the plurality of anatomical elements based on anatomical context information comprising location relationships between the plurality of anatomical elements, and adjust the location of at least one of the plurality of anatomical elements based on a result of the verification,

combine at least one piece of information related to the plurality of verified anatomical elements and the adjusted at least one anatomical element with the medical image, and

display the medical image combined with the at least one piece of information,

wherein the at least one piece of information to be combined with the medical image comprises confidence level information indicating a confidence level that each of the plurality of anatomical elements is actually an anatomical element,

wherein the at least one information to be combined with the medical image is selected by a user from the information related to the plurality of verified anatomical elements and the adjusted at least one anatomical element,

wherein the medical image is one of a plurality of continuous frames or one of a plurality of three-dimensional (3D) images,

wherein the anatomical context information further comprises location information of anatomical elements detected from an adjacent frame of the medical image or an adjacent cross-section of the medical image,

wherein the at least one processor is further configured to;

verify the location of each of the plurality of anatomical elements detected from the medical image based on the location information of the anatomical elements detected from the adjacent frame or the adjacent cross-section,

wherein the medical image comprises a breast ultrasound image of a human breast captured using ultrasonic waves,

wherein the at least one processor is further configured to detect, from the breast ultrasound image, any two or more of skin, fat, glandular tissue, muscle, or bone, and

wherein the anatomical context information comprises location relationships between the any two or more of skin, fat, glandular tissue, muscle, or bone.

2. The apparatus of claim 1,

wherein the at least one processor is further configured to perform the verification and the adjustment based on the anatomical context information comprising the location relationships between the any two or more of skin, fat, glandular tissue, muscle, and bone.

3. The apparatus of claim 1, wherein the at least one processor is further configured to individually detect a respective one of the plurality of anatomical elements from the medical image.

4. The apparatus of claim 3, wherein the at least one processor is further configured to detect the respective one of the plurality of anatomical elements using any one of a deep learning technique, a sliding window technique, or a superpixel technique.

5. The apparatus of claim 1, wherein the at least one processor is further configured to detect the plurality of anatomical elements from the medical image simultaneously.

6. The apparatus of claim 5, wherein the at least one processor is further configured to detect the plurality of anatomical elements from the medical image simultaneously using any one of a deep learning technique, a sliding window technique, or a superpixel technique.

7. The apparatus of claim 5, wherein the at least one processor is further configured to detect the plurality of anatomical elements from the medical image simultaneously by extracting a plurality of feature maps from the medical image, allocating the plurality of feature maps to corresponding respective anatomical elements, and labeling a location in the medical image of a feature map allocated to the specific anatomical element as the specific anatomical element.

8. The apparatus of claim 1, wherein the at least one processor is further configured to adjust a detection result of the plurality of anatomical elements detected from the medical image using a conditional random field technique or a Markov random field technique.

9. The apparatus of claim 1,

wherein the anatomical context information further comprises information on a probability distribution of a location at which each of the plurality of anatomical elements is located in the medical image, and

wherein the probability distribution is acquired from pre-established learning data.

10. The apparatus of claim 1, wherein the at least one piece of information related to the plurality of anatomical elements to be combined with the medical image comprises area information indicating an area of each of the plurality of anatomical elements in the medical image.

11. The apparatus of claim 10, wherein the area information comprises any one or any combination of a contour of an area of each of the plurality of anatomical elements, text representing a name of each of the plurality of anatomical elements, or a color distinguishing an area of each of the plurality of anatomical elements.

12. The apparatus of claim 10, wherein the confidence level information comprises any one or any combination of text displayed within an area of each of the plurality of anatomical elements, a color of a contour of an area of each of the plurality of anatomical elements, or a transparency of a contour of an area of each of the plurality of anatomical elements.

13. The apparatus of claim 1, wherein the at least one piece of information related to the plurality of anatomical elements to be combined with the medical image comprises any one or any combination of the following:

area information about an area of each of the plurality of anatomical elements in the medical image,

a color or a transparency of a contour of an area of each of the plurality of anatomical elements,

a color or a transparency of text representing a name of each of the plurality of anatomical elements, or

a transparency of a color layer distinguishing an area of each of the plurality of anatomical elements.

14. The apparatus of claim 1, wherein the at least one processor is further configured to verify whether a region of interest (ROI) detected from the medical image is a lesion based on a lesion detection probability of an anatomical element in which the ROI is located.

15. A method of visualizing anatomical elements in a medical image, the method comprising:

receiving a medical image;

detecting a plurality of anatomical elements from the medical image;

verifying a location of each of the plurality of anatomical elements based on anatomical context information comprising location relationships between the plurality of anatomical elements;

adjusting the location of at least one of the plurality of anatomical elements based on a result of the verification;

combining at least one piece of information related to the plurality of verified anatomical elements and the adjusted at least one anatomical element with the medical image; and

displaying the medical image combined with the at least one piece of information,

wherein the at least one piece of information to be combined with the medical image comprises confidence level information indicating a confidence level that each of the plurality of anatomical elements is actually an anatomical element,

wherein the at least one information to be combined with the medical image is selected by a user from the information related to the plurality of verified anatomical elements and the adjusted at least one anatomical element,

wherein the medical image is one of a plurality of continuous frames or one of a plurality of three-dimensional (3D) images,

wherein the anatomical context information further comprises location information of anatomical elements detected from an adjacent frame of the medical image or an adjacent cross-section of the medical image,

wherein the verifying of the location of each of the plurality of anatomical elements comprises;

verifying the location of each of the plurality of anatomical elements detected from the medical image based on the location information of the anatomical elements detected from the adjacent frame or the adjacent cross-section,

wherein the medical image comprises a breast ultrasound image of a human breast captured using ultrasonic waves,

wherein the detecting of the plurality of anatomical elements comprises detecting, from the breast ultrasound image, any two or more of skin, fat, glandular tissue, muscle, or bone, and

wherein the anatomical context information comprises location relationships between the any two or more of skin, fat, glandular tissue, muscle, and bone.

16. The method of claim 15,

wherein the verifying and the adjusting comprise performing the verifying and the adjusting based on the anatomical context information comprising the location relationships between the any two or more of skin, fat, glandular tissue, muscle, or bone.

17. The method of claim 15, wherein the detecting of the plurality of anatomical elements comprises individually detecting respective ones of the plurality of anatomical elements from the medical image.

18. The method of claim 17, wherein the individually detecting of the respective ones of the plurality of anatomical elements comprises individually detecting the respective ones of the plurality of anatomical elements from the medical image using any one of a deep learning technique, a sliding window technique, or a superpixel technique.

19. The method of claim 15, wherein the detecting of the plurality of anatomical elements comprises simultaneously detecting the plurality of anatomical elements from the medical image.

20. The method of claim 19, wherein the simultaneously detecting of the plurality of anatomical elements comprises simultaneously detecting the plurality of anatomical elements using any one of a deep learning technique, a sliding window technique, or a superpixel technique.

21. The method of claim 19, wherein the simultaneously detecting of the plurality of anatomical elements comprises:

extracting a plurality of feature maps from the medical image,

allocating the plurality of feature maps to corresponding respective anatomical elements, and

labeling a location in the medical image of a feature map allocated to the specific anatomical element as the specific anatomical element.

22. The method of claim 15, wherein the adjusting comprises adjusting a detection result of the plurality of anatomical elements detected from the medical image using a conditional random field technique or a Markov random field technique.

23. The method of claim 15,

wherein the anatomical context information comprises information on a probability distribution of a location at which each of the plurality of anatomical elements is located in the medical image, and

wherein the probability distribution is acquired from pre-established learning data.

24. The method of claim 15, wherein the at least one piece of information related to the plurality of anatomical elements to be combined with the medical image comprises area information indicating an area of each of the plurality of anatomical elements in the medical image.

25. The method of claim 24, wherein the area information comprises any one or any combination of a contour of an area of each of the plurality of anatomical elements, text representing a name of each of the plurality of anatomical elements, or a color of an area distinguishing each of the plurality of anatomical elements.

26. The method of claim 15, wherein the confidence level information comprises any one or any combination of text within an area of each of the plurality of anatomical elements, a color of a contour of an area of each of the plurality of anatomical elements, or a transparency of a contour of an area of each of the plurality of anatomical elements.

27. The method of claim 15, wherein the at least one piece of information related to the plurality of anatomical elements to be combined with the medical image comprises any one or any combination of the following:

area information about an area of each of the plurality of anatomical elements in the medical image,

a color or a transparency of an area of each of the plurality of anatomical elements,

a color or a transparency of text representing a name of each of the plurality of anatomical elements, or

a transparency of a color layer distinguishing an area of each of the plurality of anatomical elements.

28. The method of claim 15, further comprising verifying whether a region of interest (ROI) detected from the medical image is a lesion based on a lesion detection probability of an anatomical element in which the detected ROI is located.

29. A method of visualizing anatomical elements in a medical image, the method comprising:

detecting a plurality of anatomical elements from a medical image;

correcting a boundary of at least one of the plurality of anatomical elements to match an actual boundary of at least one of the plurality of anatomical elements in the medical image based on anatomical context information comprising location relationships between the plurality of anatomical elements;

combining at least one piece of anatomical element information of the at least one anatomical element having the corrected boundary with the medical image; and

displaying the medical image combined with the at least one piece of anatomical element information,

wherein the at least one piece of anatomical element information to be combined with the medical image comprises confidence level information indicating a confidence level that each of the plurality of anatomical elements is actually an anatomical element,

wherein the at least one piece of anatomical element information to be combined with the medical image is selected by a user from anatomical element information related to the plurality of anatomical elements,

wherein the medical image is one of a plurality of continuous frames or one of a plurality of three-dimensional (3D) images,

wherein the anatomical context information further comprises location information of anatomical elements detected from an adjacent frame of the medical image or an adjacent cross-section of the medical image,

wherein the correcting of the boundary of the at least one of the plurality of anatomical elements comprises;

correcting the boundary of the at least one of the plurality of anatomical elements detected from the medical image based on the location information of the anatomical elements detected from the adjacent frame or the adjacent cross-section,

wherein the medical image comprises a breast ultrasound image of a human breast captured using ultrasonic waves,

wherein the detecting of the plurality of anatomical elements comprises detecting, from the breast ultrasound image, any two or more of skin, fat, glandular tissue, muscle, or bone, and

wherein the anatomical context information comprises location relationships between the any two or more of skin, fat, glandular tissue, muscle, and bone.

30. The method of claim 29, further comprising ignoring any detected anatomical element that does not correspond to an actual anatomical element in the medical image based on the anatomical context information.