SYSTEMS AND METHODS FOR MULTIPLE INSTANCE LEARNING FOR CLASSIFICATION AND LOCALIZATION IN BIOMEDICAL IMAGING

• US 20190295252A1
• Filed: 03/22/2019
• Published: 09/26/2019
• Est. Priority Date: 03/23/2018
• Status: Active Grant

• Current Assignee: Memorial Sloan-Kettering Cancer Center
• Original Assignee: Memorial Sloan-Kettering Cancer Center
• Inventors: Fuchs, Thomas, Campanella, Gabriele
• Granted Patent: US 10,445,879 B1
• Time in Patent Office: Days
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06F 18/217 Validation; Performance evaluation; Active pattern learning techniques
• CPC Class Codes: G06F 18/2323 based on graph theory, e.g. minimum spanning trees [MST] or graph cuts
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06T 2207/10056 Microscopic image
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30024 Cell structures in vitro; Tissue sections in vitro
• CPC Class Codes: G06T 2207/30096 Tumor; Lesion
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06V 10/7635 based on graphs, e.g. graph cuts or spectral clustering
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/98 Detection or correction of errors, e.g. by rescanning the pattern or by human intervention; Evaluation of the quality of the acquired patterns
• CPC Class Codes: G06V 2201/03 Recognition of patterns in medical or anatomical images
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 50/20 for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16H 50/30 : for calculating health indices; for individual health risk assessment
• CPC Class Codes: G16H 50/70 : for mining of medical data, e.g. analysing previous cases of other patients

Abstract

The present disclosure is directed to systems and methods for classifying biomedical images. A feature classifier may generate a plurality of tiles from a biomedical image. Each tile may correspond to a portion of the biomedical image. The feature classifier may select a subset of tiles from the plurality of tiles by applying an inference model. The subset of tiles may have highest scores. Each score may indicate a likelihood that the corresponding tile includes a feature indicative of the presence of the condition. The feature classifier may determine a classification result for the biomedical image by applying an aggregation model. The classification result may indicate whether the biomedical includes the presence or lack of the condition.

Claims

1. A method of training models for classifying biomedical images, comprising:

generating, by an image classifier executing on one or more processors, a plurality of tiles from each biomedical image of a plurality of biomedical images, the plurality of biomedical images including a first biomedical image having a first label indicating a presence of a first condition and a second biomedical image having a second label indicating a lack of presence of the first condition or a presence of a second condition;

establishing, by the image classifier, an inference system to determine, for each tile of the plurality of tiles in each biomedical image of the plurality of biomedical images, a score indicating a likelihood that the tile includes a feature indicative of the presence of the first condition;

for the first biomedical image;

selecting, by the image classifier, a first subset of tiles from the plurality of tiles having the highest scores;

comparing, by the image classifier, the scores of the tiles in the first subset to a first threshold value corresponding to the presence of the first condition; and

modifying, by the image classifier, the inference system responsive to determining that the scores of at least one tile of the first subset of tiles is below the first threshold value; and

for the second biomedical image;

selecting, by the image classifier, a second subset of tiles from the plurality of tiles having the highest scores;

comparing, by the image classifier, the scores of the tiles in the second subset to a second threshold value corresponding to the lack of the presence of the first condition or the presence of the second condition; and

modifying, by the image classifier, the inference system responsive to determining that the scores of at least one tile of the second subset of tiles is above the second threshold value.

2. The method of claim 1, further comprising:

determining, by the image classifier, for the at least one tile of the first subset, a first error metric between the score of the at least one tile to a first value corresponding to the presence of the first condition; and

wherein modifying the inference system further comprises modifying the inference system based on the first error metric of the at least one tile of the first subset;

determining, by the image classifier, for the at least one tile of the second subset, a second error metric between the score of the at least one tile to a second value corresponding to the lack of the presence of the first condition; and

wherein modifying the inference system further comprises modifying the inference system based on the second error metric of the at least one tile of the second subset.

3. The method of claim 1, further comprising:

maintaining, by the image classifier, the inference system responsive to determining that scores of none of a plurality of tiles for a third biomedical image of the plurality of biomedical images is below the first threshold, the third biomedical image having the first label indicating the presence of the first condition; and

maintaining, by the image classifier, the inference system responsive to determining that scores of none of a plurality of tiles for a fourth biomedical image of the plurality of biomedical images is below the second threshold, the fourth biomedical image having the first label indicating the lack of the presence of the first condition.

4. The method of claim 1, wherein selecting the first subset of tiles further comprises selecting a predefined first number of tiles from the plurality of tiles for the first biomedical image having the highest scores;

and

wherein selecting the second subset of tiles further comprises selecting a predefined second number of tiles from the plurality of tiles for the second biomedical image having the highest scores.

5. The method of claim 1, wherein establishing the inference system further comprises initializing the inference system comprising a convolutional neural network, the convolutional neural network having one or more parameters, each parameter of the one or more parameters set to a random value.

6. The method of claim 1, further comprising applying, by the image classifier, a third subset of tiles from a plurality of tiles for a third biomedical image of the plurality of biomedical images to an aggregation system to train the aggregation system based on a comparison on a label of the third biomedical image with a classification result from applying the aggregation system to third subset.

7. A method of training models for classifying features in biomedical images, comprising:

identifying, by an image classifier executing on one or more processors, a subset of tiles from a plurality of tiles of a biomedical image of a plurality of biomedical images, the biomedical image having a label indicating a presence of a condition;

establishing, by the image classifier, an aggregation system to determine classifications of biomedical images to indicate whether the corresponding biomedical image contains a feature indicative of the presence of the condition;

determining, by the image classifier, a classification result for the biomedical image by applying the aggregation system to the subset of tiles identified from the biomedical image, the classification result indicating one of the biomedical image as containing at least one feature corresponding to the presence of the condition or the biomedical image as lacking any features corresponding to the lack the of the condition;

comparing, by the image classifier, the classification result determined for the biomedical image with the label indicating the presence of the condition on the biomedical image; and

modifying, by the image classifier, the aggregation system responsive to determining that the classification result from the aggregation system does not match the label for the biomedical image.

8. The method of claim 7, further comprising determining, by the image classifier, an error metric between the classification result and the label, responsive to determining that the classification result does not match the label for the biomedical image;

and

wherein modifying the aggregation system further comprises modifying at least one parameter of the aggregation system based on the error metric.

9. The method of claim 7, wherein establishing the aggregation system further comprises initializing the aggregation system comprising a recurrent neural network, the recurrent neural network having one or more parameters, each parameter of the one or more parameters set to a random value.

10. The method of claim 7, further comprising maintaining, by the image classifier, the aggregation system responsive to determining that a second classification result from the aggregation system for a second subset of tiles from a second biomedical image matches a second label for the second biomedical image.

11. The method of claim 7, wherein applying the aggregation system to the subset of tiles further comprises applying the subset of tiles in one of a sequential order or random order from the plurality of tiles for the biomedical image.

12. The method of claim 7, wherein identifying the subset of tiles further comprises identifying the subset of tiles from the plurality of tiles for the biomedical image selected by an inference system based on scores, each score for a corresponding tile of the subset indicating a likelihood that the corresponding tile includes a feature indicative of the presence of the condition.

13. A system for classifying biomedical images, comprising:

a plurality of biomedical images maintainable on a database;

an inference system maintainable on one or more processors, configured to select subsets of tiles from the plurality of biomedical images including features indicative of a presence of a first condition;

an aggregation system maintainable on the one or more processors, configured to determine whether biomedical images are classified as one of including the presence of the first condition or a lack of the first condition or a presence of a second condition;

a feature classifier executable on the one or more processors, configured to;

generate a plurality of tiles from at least one biomedical image of the plurality of biomedical images, each tile corresponding to a portion of the biomedical image;

select a subset of tiles from the plurality of tiles for the biomedical image by applying the inference system to the plurality of tiles, the subset of tiles having highest scores, each score indicating a likelihood that the corresponding tile includes a feature indicative of the presence of the first condition; and

determine a classification result for the biomedical image by applying the aggregation system to the selected subset of tiles, the classification result indicating whether the biomedical includes the presence of the first condition or the lack of the condition or the presence of the second condition.

14. The system of claim 13, wherein the feature classifier is further configured to generate the plurality of tiles by using one of a plurality of defined magnification factors onto the biomedical image.

15. The system of claim 13, wherein the feature classifier is further configured to:

determine, for each tile of the plurality of tiles of the biomedical image, by applying the inference system to the tile, a score indicating the likelihood that the tile includes features indicative of the presence of the first condition; and

select a predefined number of tiles from the plurality of tiles having the highest scores to form the subset of tiles.

16. The system of claim 13, wherein the feature classifier is further configured to input the selected subset of tiles in sequential order or in random order in to the aggregation system to determine the classification result for the biomedical image.

17. The system of claim 13, further comprising a model trainer executable on the one or more processors, configured to:

generate a plurality of tiles from each biomedical image of the plurality of biomedical images, the plurality of biomedical images including a first biomedical image having a first label indicating the presence of a condition and a second biomedical image having a second label indicating a lack of the presence of the first condition;

select a first subset of tiles from the plurality of tiles of the first biomedical image having the highest scores among the plurality of tiles from the first biomedical image;

select a second subset of tiles from the plurality of tiles of the second biomedical image having the highest scores among the plurality of tiles from the second biomedical image; and

modify the inference system based on a first comparison between the scores of the first subset of tiles and a first value corresponding to the presence of the first condition and a second comparison between the scores of the second subset of tiles and a second value corresponding to the lack of the presence of the first condition.

18. The system of claim 17, further comprising a model trainer executable on the one or more processors, configured to:

determine a first error metric based on the first comparison between the scores of the first subset of tiles and a first value corresponding to the presence of the first condition;

determine a second error metric based on the second comparison between the scores of the second subset of tiles and a second value corresponding to the lack of the presence of the first condition; and

modify at least one parameter of the inference system based on the first error metric and the second error metric.

19. The system of claim 13, further comprising a model trainer executable on the one or more processors, configured to:

identify a subset of tiles from the plurality of tiles of a second biomedical image of the plurality of biomedical images, the second biomedical image having a label indicating the presence of the first condition;

determine a second classification result for the second biomedical image by applying the aggregation system to the subset of tiles identified from the second biomedical image, the classification result indicating one of the second biomedical image as containing at least one feature corresponding to the presence of the first condition or the second biomedical image as lacking any features corresponding to the lack the of the first condition or the presence of the second condition;

modify the aggregation system based on a comparison between the second classification result and the label for the second biomedical image.

20. The system of claim 13, further comprising a model trainer executable on the one or more processors, configured to:

determine, subsequent to modifying the inference system, that one or more parameters of the inference system have converged relative the one or more parameters prior to the modification of the inference system;

initiate training of the aggregation mode, responsive to the determination that the one or more parameters of the inference has converged.