Systems and methods for quantitative analysis of histopathology images using multiclassifier ensemble schemes

• US 10,055,551 B2
• Filed: 10/10/2014
• Issued: 08/21/2018
• Est. Priority Date: 10/10/2013
• Status: Active Grant

• Current Assignee: Board of Trustees of the Leland Stanford Junior University (Stanford Management Co.), Board of Regents of the University of Texas System (University of Texas System)
• Original Assignee: Board of Regents of the University of Texas System (University of Texas System)
• Inventors: Agaian, Sos, Mosquera-Lopez, Clara M., Greenblatt, Aaron
• Primary Examiner(s): Patel, Nirav G
• Application Number:
  US15/028,314
• Publication Number:
  US 20160253466A1
• Time in Patent Office: 1,411 Days
• Field of Search: None
• CPC Class Codes: G06F 18/2431 Multiple classes
• CPC Class Codes: G06F 18/253 of extracted features
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G06N 20/20 Ensemble learning
• CPC Class Codes: G06N 3/042 Knowledge-based neural networks; Logical representations of neural networks
• CPC Class Codes: G06N 5/043 Distributed expert systems; Blackboards
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10056 Microscopic image
• CPC Class Codes: G06T 2207/20036 Morphological image processing
• CPC Class Codes: G06T 2207/20064 Wavelet transform [DWT]
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/20084 Artificial neural networks [ANN]
• CPC Class Codes: G06T 2207/30024 Cell structures in vitro; Tissue sections in vitro
• CPC Class Codes: G06T 2207/30096 Tumor; Lesion
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/0014 using an image reference approach
• CPC Class Codes: G06T 7/246 using feature-based methods, e.g. the tracking of corners or segments
• CPC Class Codes: G06V 10/467 Encoded features or binary features, e.g. local binary patterns [LBP]
• CPC Class Codes: G06V 10/758 Involving statistics of pixels or of feature values, e.g. histogram matching
• CPC Class Codes: G06V 10/806 of extracted features
• CPC Class Codes: G06V 20/695 : Preprocessing, e.g. image segmentation
• CPC Class Codes: G16H 50/20 : for computer-aided diagnosis, e.g. based on medical expert systems
• CPC Class Codes: G16Z 99/00 : Subject matter not provided for in other main groups of this subclass

Abstract

Described herein are systems and methods for performing multi-stage detection and classification of cancer regions from digitized images of biopsy slides. Novel methods for processing the digitized images to improve feature extraction and structure identification are disclosed, including but not limited to the use of quaternions, logarithmic mappings of color channels, and application of wavelets to logarithmic color channel mappings. The extracted features are utilized in improved machine learning algorithms that are further optimized to analyze multiple color channels in multiple dimensions. The improved machine learning algorithms include techniques for accelerating the training of the algorithms, making their application to biopsy detection and classification practical for the first time. The performance of the described systems and methods are further improved by the disclosure of a novel multistage machine learning scheme, in which additional classifiers are utilized to choose among the classes proposed by other classifiers in close cases.

Claims

1. A method for performing multi-stage classification for detection of cancer regions from one or more digitized images of biopsy slides comprising:

pre-processing, from the digitized images, biopsy regions of interest or biopsy whole-slides, using color normalization methods and segmentation methods;

decomposing the pre-processed digitized images into several segments;

describing the image segments by forming feature vectors based on at least one of textural, color, or morphology tissue characteristics;

classifying the image segments using one or more classification methods in a first classification stage, wherein the classification methods utilize the feature vectors of the image segments, and further wherein the classification methods are able to produce scores and posterior conditional probabilities or equivalents for each possible class, and further wherein the first classification stage is a multi-class classifier;

classifying one or more of the image segments using a second classification stage if one or more conditions are met, wherein the second classification stage comprises a set of one or more specialized classifiers or refinement classifiers which are trained on a smaller group of classes with respect to the classifiers used in the first stage;

classifying, based upon either the first or second classification stage, a portion of the region of interest or biopsy whole slides, as cancerous or non-cancerous;

determining a cancer classification, tumor extent, tumor size, or tumor localization based at least in part upon the output of either of the first or second classification stage.

2. The method of claim 1, wherein the condition comprises when a maximum posterior probability produced in the first classification stage is less than a confidence threshold.

3. The method of claim 1, wherein the condition comprises when two or more of the classes outputted by the first classification stage have a difference in classification margin or when the two or more classes have similar posterior probabilities.

4. The method of claim 1, wherein the classification methods used in the second classification stage are binary.

5. The method of claim 1, wherein the digitized images comprise biopsy images of different body tissues.

6. The method of claim 1, wherein, for one or more of the classification methods used in either the first or second classification stage, data utilized by the classification methods for training or validation is separated into at least two disjoint groups using a cross-validation approach.

7. The method of claim 1, wherein image color normalization is performed by one or more of the following:

fuzzy local color transference, color deconvolution, the Reinhard method, histogram matching, or non-linear color mapping.

8. The method of claim 1, where the preprocessed digitized images may be decomposed to generate various images'"'"' segments using:

channel decomposition, bit plane decomposition, empirical mode decomposition, ensemble empirical mode decomposition, or human visual system-based image segmentation.

9. The method of claim 1, wherein at least one of one of the classifiers'"'"' parameters is adjusted using cross-validation based on the performance metrics obtained through several cross-validation rounds.

10. The method of claim 1, wherein at least one of the classification stages outputs posterior probabilities of Gleason grades, and the outputed posterior probabilities are used to estimate the distribution of grades within a tissue core.

11. A method for performing multi-stage classification for detection of cancer regions from one or more digitized images of biopsy slides comprising:

pre-processing, from the digitized images, biopsy regions of interest or biopsy whole-slides, using color normalization methods and segmentation methods;

decomposing the pre-processed digitized images into several segments;

describing the image segments by forming feature vectors based on at least one of textural, color, or morphology tissue characteristics;

classifying the image segments using one or more classification methods in a first classification stage, wherein the classification methods utilize the feature vectors of the image segments, and further wherein the classification methods are able to produce scores and posterior conditional probabilities or equivalents for each possible class;

classifying one or more of the image segments using a second classification stage if one or more conditions are met;

classifying, based upon either the first or second classification stage, a portion of the region of interest or biopsy whole slides, as cancerous or non-cancerous;

determining a cancer classification, tumor extent, tumor size, or tumor localization based at least in part upon the output of either of the first or second classification stage;

wherein at least one of the classification stages outputs posterior probabilities of Gleason grades, and the outputed posterior probabilities are used to estimate the distribution of grades within a tissue core.

12. The method of claim 11, wherein the condition comprises when the maximum posterior probability produced in the first classification stage is less than a confidence threshold.

13. The method of claim 11, wherein the condition comprises when two or more of the classes outputted by the first classification stage have a small difference in classification margin or when the two or more classes have similar posterior probabilities.

14. The method of claim 11, wherein the classification methods used in the second classification stage are binary.

15. The method of claim 11, wherein the digitized images comprise biopsy images of different body tissues.

16. The method of claim 11, wherein, for one or more of the classification methods used in either the first or second classification stage, data utilized by the classification methods for training or validation is separated into at least two disjoint groups using a cross-validation approach.

17. The method of claim 11, wherein image color normalization is performed by one or more of the following:

fuzzy local color transference, color deconvolution, the Reinhard method, histogram matching, or non-linear color mapping.

18. The method of claim 11, where the preprocessed digitized images may be decomposed to generate various images'"'"' segments using:

channel decomposition, bit plane decomposition, empirical mode decomposition, ensemble empirical mode decomposition, or human visual system-based image segmentation.

19. The method of claim 11, wherein at least one of one of the classifiers'"'"' parameters is adjusted using cross-validation based on the performance metrics obtained through several cross validation rounds.