Methods for improving the accuracy in differential diagnosis on radiologic examinations

• US 6,058,322 A
• Filed: 07/25/1997
• Issued: 05/02/2000
• Est. Priority Date: 07/25/1997
• Status: Expired due to Fees

• Current Assignee: ARCH Development Corporation (University of Chicago)
• Original Assignee: ARCH Development Corporation (University of Chicago)
• Inventors: Nishikawa, Robert M., Jiang, Yulei, Ashizawa, Kazuto, Doi, Kunio
• Primary Examiner(s): Casler, Brian L.
• Application Number:
  US08/900,361
• Time in Patent Office: 1,012 Days
• Field of Search: 600/407, 600/408, 128/920, 128/922, 128/925, 382/128, 382/130, 382/133
• US Class Current: 600/408
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: Y10S 128/925 Neural network

Abstract

A computer-aided method for detecting, classifying, and displaying candidate abnormalities, such as microcalcifications and interstitial lung disease in digitized medical images, such as mammograms and chest radiographs, a computer programmed to implement the method, and a data structure for storing required parameters, wherein in the classifying method candidate abnormalities in a digitized medical image are located, regions are generated around one or more of the located candidate abnormalities, features are extracted from at least one of the located candidate abnormalities within the region and from the region itself, the extracted features are applied to a classification technique, such as an artificial neural network (ANN) to produce a classification result (i.e., probability of malignancy in the form of a number and a bar graph), and the classification result is displayed along with the digitized medical image annotated with the region and the candidate abnormalities within the region. In the detecting method candidate abnormalities in each of a plurality of digitized medical images are located, regions around one or more of the located candidate abnormalities in each of a plurality of digitized medical images are generated, the plurality of digitized medical images annotated with respective regions and candidate abnormalities within the regions are displayed, and a first indicator (e.g., blue arrow) is superimposed over candidate abnormalities comprising of clusters and a second indicator (e.g., red arrow) is superimposed over candidate abnormalities comprising of masses. In a user modification mode, during classification, a user modifies the located candidate abnormalities, the determined regions, and/or the extracted features, so as to modify the extracted features applied to the classification technique and the displayed results, and, during detection, a user modifies the located candidate abnormalities, the determined regions, and the extracted features, so as to modify the displayed results.

Claims

1. A computer-aided method for classifying a digitized medical image of interest, comprising:

a) locating at least one candidate abnormality in the digitized medical image of interest;

b) determining a region in which the at least one located candidate abnormality is located;

c) extracting features from at least one of

     1) the at least one located candidate abnormality and

     2) said region in which said at least one located candidate abnormality is located;

d) applying the extracted features to a neural network to produce a classification result;

e) calculating a likelihood of malignancy of said at least one candidate abnormality using at least one of the following equations, ##EQU16## wherein x is the classification result produced by the neural network, M(x) is the probability density function of the classification result x that said at least one candidate abnormality is actually malignant, B(x) is the analogous probability density function for actually benign cases, η

is the prevalence of malignant cases in a population studied; and

f) displaying the calculated likelihood of malignancy.

2. The method of claim 1, wherein step a) comprises:

using differential imaging techniques.

3. The method of claim 1, wherein step b) comprises:

using segmentation techniques.

4. The method of claim 1, wherein step c) comprises:

extracting from the region features including at least one of circularity of the region, area of the region, and a number of candidate abnormalities within the region; and

extracting from the at least one candidate abnormality within the region features including at least one of shape irregularity, area, and volume of said at least one candidate abnormality.

5. The method of claim 1, wherein step f) comprises:

displaying the likelihood of malignancy and the digitized medical image annotated with the region and the at least one candidate abnormality within the region.

6. The method of claim 1, wherein step d) comprises:

applying the extracted features to a neural network having eight input units, six hidden units, and one output unit.

7. The method of claim 1, further comprising:

obtaining and defining as features plural of the following;

a patient'"'"'s age, sex, duration of symptoms, severity of symptoms, temperature, immune status, underlying malignancies, smoking habits, dust exposure, and drug treatment; and

wherein step c) comprises;

extracting features from the region based on a location of candidate abnormalities within the region, and

extracting from the at least one candidate abnormality plural features selected from the group of features consisting of homogeneity, fineness, coarseness, nodularity, septal lines, honeycombing, and loss of lung volume, and a patient'"'"'s lymphadenopathy, pleural effusion, and heart; and

step d) comprises applying the obtained plural features and the extracted features to said neural network.

8. The method of claim 7, wherein:

said step e) comprises calculating a respective likelihood of malignancy of each of plural predetermined abnormalities; and

step f) comprises displaying the calculated likelihood of malignancy of each of the plural predetermined abnormalities.

9. The method of claim 7, wherein:

said step e) comprises applying the obtained plural features and the extracted features to a neural network having twenty-six input units, eighteen hidden units, and eleven output units.

10. The method of claim 9, wherein:

step e) comprises calculating a respective likelihood of malignancy of each of eleven abnormalities; and

step f) comprises displaying the calculated likelihood of malignancy of each of the eleven predetermined abnormalities.

11. The method of claim 7, wherein step f) comprises:

displaying at least one of the likelihood of malignancy and the extracted features in at least one of numerical and analog form;

displaying the region annotated with a line around a perimeter of the region superimposed on the digitized medical image; and

displaying the candidate abnormalities within the region in the digitized medical image of the case of interest with a location indicator superimposed on the candidate abnormalities.

12. The method of claim 1, wherein:

said step e) comprises calculating a respective likelihood of malignancy of each of plural predetermined abnormalities; and

step f) comprises displaying the calculated likelihood of malignancy of each of the plural predetermined abnormalities.

13. The method of claim 1, further comprising:

g) obtaining digitized image data from at least one medical image which includes a candidate abnormality known to be actually abnormal and which has a likelihood of malignancy obtained by means of steps a) through e) similar to the likelihood of malignancy of the candidate abnormality of the digital medical image of interest located in step a);

h) obtaining digitized image data from at least one medical image which includes a candidate abnormality known to be actually normal and which has a likelihood of malignancy obtained by means of steps a) through e) similar to the likelihood of malignancy of the candidate abnormality of the digital medical image of interest located in step a); and

i) displaying the digitized medical image of interest and at least one of the digitized image data obtained in steps g) and h), and the likelihood of malignancy of the displayed digitized image data obtained in steps g) and h).

14. The method of claim 13, wherein step a) comprises:

using differential imaging techniques.

15. The method of claim 13, wherein step b) comprises:

using segmentation techniques.

16. The method of claim 13, wherein step c) comprises:

extracting from the region features including at least one of circularity of the region, area of the region, and a number of candidate abnormalities within the region; and

extracting from the at least one candidate abnormality within the region features including at least one of shape irregularity, area, and volume of said at least one candidate abnormality.

17. The method of claim 13, wherein step i) comprises:

displaying at least one of the likelihood of malignancy and the extracted features in at least one of numerical and analog form for at least one of the digital image data of interest and the obtained image data;

displaying the region annotated with a line around a perimeter of the region superimposed on the digitized medical images of at least one of the digital image data of interest and the obtained image data; and

displaying the at least one candidate abnormality within the region superimposed on the medical images of at least one of the digitized image data of interest and the obtained image data.

18. The method of claim 13, further comprising:

obtaining and defining as features plural of the following;

a patient'"'"'s age, sex, duration of symptoms, severity of symptoms, temperature, immune status, underlying malignancies, smoking habits, dust exposure, and drug treatment; and

wherein step c) comprises,

extracting features from the region based on a location of candidate abnormalities within the region, and

extracting features from the candidate abnormalities within the region based on at least one of homogeneity, fineness, coarseness, nodularity, septal lines, honeycombing, and loss of lung volume, and a patient'"'"'s lymphadenopathy, pleural effusion, and heart size due to the candidate abnormalities within the region; and

step d) comprises applying the obtained plural features and the extracted features to said neural network.

19. The method of claim 18, wherein:

said step e) comprises calculating a respective likelihood of malignancy of each of plural predetermined abnormalities; and

step f) comprises displaying the calculated likelihood of malignancy of each of the plural predetermined abnormalities.

20. The method of claim 1, further comprising:

displaying plural regions each containing a located candidate abnormality derived from a digitized medical image, and in response to a user input selecting one of the displayed regions, displaying a full image of the digitized medical image from which the selected region was derived, and the location of the user selected region within the full image.

21. The method of claim 1, comprising:

displaying the digitized medical image of interest annotated with a respective region and said at least one candidate abnormality within the region; and

superimposing a distinct indicator over a respective candidate abnormality.

22. The method of claim 21, wherein step a) comprises:

using differential imagine techniques.

23. The method of claim 21, wherein step b) comprises:

using segmentation techniques.

24. The method of claim 21, wherein step f) comprises:

displaying plural digitized medical images annotated with respective regions and candidate abnormalities within the regions.

25. The method of claim 21, wherein step f) comprises:

superimposing a first indicator of a first color over a cluster and a second indicator of a second color over a mass.

27. The method of claim 21, further comprising:

displaying plural regions each containing a located candidate abnormality derived from a digitized medical image, and in response to a user input selecting one of the displayed regions, displaying a full image of the digitized medical image from which the selected region was derived, and the location of the user selected region within the full image.

26. The method of claim to 21, wherein step f) comprises:

displaying a detailed view of one of clusters and masses indicated by one of first and second indicators upon one of a user touching one of the first and second indicators on a touch screen display and a user pointing to one of the first and second indicators with a pointing device.

28. A computer program product comprising a computer storage medium and a computer program code mechanism embedded in the computer storage medium for causing a computer to classify, and display candidate abnormalities in digitized medical images, by performing the following steps:

a) locating at least one candidate abnormality in the digitized medical image of interest;

b) determining a region in which the at least one located candidate abnormality is located;

c) extracting features from at least one of

     1) the at least one located candidate abnormality and

     2) said region in which said at least one located candidate abnormality is located;

d) applying the extracted features to a neural network to produce a classification result;

e) calculating a likelihood of malignancy of said at least one candidate abnormality using at least one of the following equations, ##EQU17## wherein x is the classification result produced by the neural network, M(x) is the probability density function of the classification result x that said at least one candidate abnormality is actually malignant, B(x) is the analogous probability density function for actually benign cases, η

is the prevalence of malignant cases in a population studied; and

f) displaying the calculated likelihood of malignancy.

29. A system for classifying, a digitized medical image of interest, comprising:

a first mechanism configured to locate at least one candidate abnormality in the digitized medical image of interest;

a second mechanism configured to determine a region in which the at least one located candidate abnormality is located;

a third mechanism configured to extract features from at least one of

     1) the at least one located candidate abnormality and

     2) said region in which said at least one located candidate abnormality is located;

a fourth mechanism configured to apply the extracted features to a neural network to produce a classification result;

a fifth mechanism configured to calculate a likelihood of malignancy of said at least one candidate abnormality using at least one of the following equations, ##EQU18## wherein x is the classification result produced by the neural network, M(x) is the probability density function of the classification result x that said at least one candidate abnormality is actually malignant, B(x) is the analogous probability density function for actually benign cases, η

is the prevalence of malignant cases in a population studied; and

a sixth mechanism configured to display the calculated likelihood of malignancy.

30. The system of claim 29, further comprising a memory containing a data structure, comprising:

a field which stores a location of the located at least one candidate abnormality;

a field which stores a location of the region;

fields which store the extracted features from the at least one candidate abnormality and the extracted features from the region; and

a field which stores the calculated likelihood of malignancy.

31. The system of claim 30, wherein the sixth mechanism is further configured to display the digitized medical image of interest annotated with a respective region and said at least one candidate abnormality within the region, and superimpose a distinct indicator over a respective candidate abnormality, and

said data structure of said memory further comprises:

fields which store locations of the candidate abnormalities for the respective digitized medical images;

fields which store locations of the regions for the respective digitized medical images; and

fields which store locations of the distinct indicators.

32. The system of claim 31, wherein the first mechanism is further configured to use differential imaging techniques, and

said data structure of said memory further comprises:

fields which store parameters for the differential imaging techniques.

33. The system of claim 31, wherein the second mechanism is further configured to use segmentation techniques, and

said data structure of said memory further comprises:

fields which store parameters for the segmentation techniques.

34. The system of claim 31, wherein the sixth mechanism is further configured to superimpose a first indicator of a first color over a cluster and a second indicator of a second color over a mass, and

said data structure of said memory further comprises:

fields which store the locations and colors of the first and second indicators.

35. The system of claim 30, wherein the first mechanism is further configured to use differential imaging techniques, and

said data structure of said memory further comprises:

fields which store parameters for the differential imaging techniques.

36. The system of claim 30, wherein the second mechanism is further configured to use segmentation techniques, and

said data structure of said memory further comprises:

fields which store parameters for the segmentation techniques.

37. The system of claim 30, wherein the third mechanism is further configured to extract from the region features including at least one of circularity of the region, area of the region, and a number of candidate abnormalities within the region, and extract from the at least one candidate abnormality within the region features including at least one of shape irregularity, area, and volume of said at least one candidate abnormality, and

said data structure of said memory further comprises:

fields which store the circularities of the respective regions, the areas of the respective regions, and the number of candidate abnormalities in the respective regions for the respective digitized medical images.

38. The system of claim 30, wherein the fourth mechanism is further configured to apply the extracted features to a neural network having eight input units, six hidden units, and one output unit, and

said data structure of said memory further comprises:

field which store value of the input units, the hidden units, the output unit, and connection weights of the neutral network; and

field which store the calculated likelihood of malignancy.

39. The system of claim 30, wherein the fifth mechanism is further configured to calculate a respective likelihood of malignancy of each of plural predetermined abnormalities,

the sixth mechanism is further configured to display the calculated likelihood of malignancy of each of the plural predetermined abnormalities, and

said data structure of said memory further comprises:

fields which store the calculated likelihoods of malignancy of the plural predetermined abnormalities.

40. The system of claim 30, further comprising:

a seventh mechanism configured to obtain and define as features plural of the following;

a patient'"'"'s age, sex, duration of symptoms, severity of symptoms, temperature, immune status, underlying malignancies, smoking habits, dust exposure, and drug treatment,

wherein the third mechanism is further configured to extract features from the region based on a location of candidate abnormalities within the region, and extract from the at least one candidate abnormality plural features selected from the group of features consisting of homogeneity, fineness, coarseness, nodularity, septal lines, honeycombing, and loss of lung volume, and a patient'"'"'s lymphadenopathy, pleural effusion, and heart,

the fourth mechanism is further configured to apply the obtained plural features and the extracted features to said neural network, and

said data structure of said memory further comprises;

fields which store the patient'"'"'s age, sex, duration of symptoms, severity of symptoms, temperature, immune status, underlying malignancies, smoking habits, dust exposure, and drug treatment;

fields which store the features extracted from the region based on the location of the candidate abnormalities within the region; and

fields which store the homogeneity, the fineness, the coarseness, nodularity, septal lines, honeycombing, and loss of lung volume, and the patient'"'"'s lymphadenopathy, pleural effusion, and heart size, due to the candidate abnormalities within the region.

41. The system of claim 40, wherein the fifth mechanism is further configured to calculate a respective likelihood of malignancy of each of plural predetermined abnormalities,

the sixth mechanism is further configured to display the calculated likelihood of malignancy of each of the plural predetermined abnormalities, and

said data structure of said memory further comprises:

fields which store the calculated likelihoods of malignancy of the plural predetermined abnormalities.

42. The system of claim 40, wherein the fourth mechanism is further configured to apply the obtained plural features and the extracted features to a neural network having twenty-six input units, eighteen hidden units, and eleven output units,

the fifth mechanism is further configured to calculate a respective likelihood of malignancy of each of eleven abnormalities,

the sixth mechanism is further configured to display the calculated likelihood of malignancy of each of the eleven predetermined abnormalities, and

said data structure of said memory further comprises:

fields which store values of the input units, the hidden units, the output units, and connection weights of the neural network; and

fields which store the calculated likelihoods of malignancy of abnormalities.

43. The system of claim 40, wherein the sixth mechanism is further configured to display at least one of the likelihood of malignancy and the extracted features in at least one of numerical and analog form, display the region annotated with a line around a perimeter of the region superimposed on the digitized medical image, and display the candidate abnormalities within the region in the digitized medical image of the case of interest with a location indicator superimposed on the candidate abnormalities, and

said data structure of said memory further comprises:

fields which store parameters for annotating the region with a line around a perimeter of the region; and

fields which store parameters for displaying the location indicators superimposed on the candidate abnormalities.