ADAPTIVE CLASSIFICATION FOR WHOLE SLIDE TISSUE SEGMENTATION

• US 20160335478A1
• Filed: 07/28/2016
• Published: 11/17/2016
• Est. Priority Date: 01/28/2014
• Status: Active Grant

• Current Assignee: Ventana Medical Systems Incorporated (Roche Holding AG)
• Original Assignee: Ventana Medical Systems Incorporated (Roche Holding AG)
• Inventors: Chefd'hotel, Christophe, Chen, Ting, Nguyen, Kien, Bredno, Joerg, Chukka, Srinivas
• Granted Patent: US 10,102,418 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: A61B 1/018 for receiving instruments
• CPC Class Codes: A61B 1/126 provided with means for cleaning in-use
• CPC Class Codes: A61B 17/00234 for minimally invasive surgery A61B17/0218, A61B17/0469, A61B17/12013, A61B17/1285, A61B17/29, A61B17/320016 take precedence
• CPC Class Codes: A61B 17/3415 for introducing tubes or catheters, e.g. gastrostomy tubes, drain catheters A61B17/3417 takes precedence; body piercing catheter guide needles A61M25/06
• CPC Class Codes: A61B 17/3423 Access ports, e.g. toroid shape introducers for instruments or hands access sites for liquids A61M39/0247
• CPC Class Codes: A61B 2017/00261 Discectomy
• CPC Class Codes: A61B 2017/00477 Coupling A61B2017/0046 takes precedence
• CPC Class Codes: A61B 2017/00526 Methods of manufacturing
• CPC Class Codes: A61B 2017/3445 used as instrument channel for multiple instruments catheters A61B2017/22072
• CPC Class Codes: A61B 2034/715 Cable tensioning mechanisms for removing slack
• CPC Class Codes: A61B 2034/742 Joysticks
• CPC Class Codes: A61B 34/71 Manipulators operated by drive cable mechanisms
• CPC Class Codes: A61B 34/74 Manipulators with manual electric input means
• CPC Class Codes: B33Y 10/00 Processes of additive manufacturing
• CPC Class Codes: B33Y 80/00 Products made by additive manufacturing
• CPC Class Codes: G06F 18/2113 by ranking or filtering the set of features, e.g. using a measure of variance or of feature cross-correlation
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30024 Cell structures in vitro; Tissue sections in vitro
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/40 : Analysis of texture depth or shape recovery from texture G06T7/529
• CPC Class Codes: G06T 7/74 : involving reference images or patches
• CPC Class Codes: G06V 10/50 : by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/771 : Feature selection, e.g. selecting representative features from a multi-dimensional feature space
• CPC Class Codes: G06V 20/698 : Matching; Classification
• CPC Class Codes: G06V 30/194 : References adjustable by an adaptive method, e.g. learning

Abstract

A method of segmenting images of biological specimens using adaptive classification to segment a biological specimen into different types of tissue regions. The segmentation is performed by, first, extracting features from the neighborhood of a grid of points (GPs) sampled on the whole-slide (WS) image and classifying them into different tissue types. Secondly, an adaptive classification procedure is performed where some or all of the GPs in a WS image are classified using a pre-built training database, and classification confidence scores for the GPs are generated. The classified GPs with high confidence scores are utilized to generate an adaptive training database, which is then used to re-classify the low confidence GPs. The motivation of the method is that the strong variation of tissue appearance makes the classification problem more challenging, while good classification results are obtained when the training and test data origin from the same slide.

Claims

1. A tissue analysis system comprising:

an image region identifier module configured to receive image data representative of an at least two-dimensional image of a tissue sample and to output image region data that identifies a plurality of subsets of said image data, each image data subset being representative of a unique, spatially contiguous region of said image;

an image region classifier module that is configured to classify, for a plurality of said image regions, the respective image region as one of a plurality of tissue types using the respective image data subset for querying a database of tissue characteristics and to determine a confidence score indicative of a confidence of said classifying of the respective image region;

a database modifier module that, for a plurality of said image regions having a confidence score falling within a first range, is configured to effect modification of said database such that the resultant modified database comprises data representative of the tissue type classified to the respective image region and tissue characteristic data obtained from the respective image data subset, said first range being a range of confidence scores that includes a confidence score representative of certainty that the classification is correct; and

an image region reclassifier module that is configured to reclassify, for a plurality of said image regions having a confidence score falling within a second range distinct from said first range, the respective image region as one of said plurality of tissue types using the respective image data subset for querying said modified database and to output said one of said plurality of tissue types as an analysis result.

2. The tissue analysis system of claim 1, further comprising:

a tissue imaging device configured to image a tissue sample to obtain raw image data, wherein said received image data is obtained from said raw image data.

3. The tissue analysis system of claim 1, further comprising:

a tissue staining device configured to stain said tissue sample to obtain a stained tissue sample; and

a tissue imaging device configured to image said stained tissue sample to obtain said image data, wherein said received image data is obtained from said raw image data.

4. The tissue analysis system of claim 1, further comprising:

a data storage system configured to store said database, wherein said database comprises, for each of a plurality of tissue image regions, data representative of an at least two-dimensional image of tissue, data representative of at least one tissue feature, data representative of a tissue type and data representative of a confidence score, wherein the database is a pre-built first training database that is used for the image classification and confidence score determination by the image region classifier module, wherein the database modifier module is configured to generate a second training database that is constituted by the data representative of the tissue type classified to the image regions and the tissue characteristic data obtained from the image data subsets of the plurality of said image regions having a confidence score falling within the first range and to combine the first and the second training database to provide the modified database.

5. The tissue analysis system of claim 1, wherein said image region classifier module comprises a support vector machine and is configured to use an output of said support vector machine for determining said confidence score.

6. The tissue analysis system of any claim 1, wherein said classifying the respective image region comprises extracting at least one feature from the respective image region using the respective image data subset and said data obtained from said database, said feature belonging to the group consisting of textural features, biological features, intensity features, gradient features, Gabor features, co-occurrence features, and nuclei features.

7. The tissue analysis system of claim 1, wherein said reclassifying the respective image region comprises weighting data of the respective image data subset and the data obtained from said modified database using at least one of a spatial proximity value, a confidence score and feature similarity value.

8. The tissue analysis system of claim 1, wherein:

said image region classifier module comprises an image channel extractor; and

said classifying the respective image region comprises separating, using said image channel extractor, at least the respective region of said image into one or more component channels and extracting at least one feature from the respective image region using any of said component channels of the respective image region and said data obtained from said database, wherein said feature belongs to the group consisting of textural features, biological features, intensity features, gradient features, Gabor features, co-occurrence features, and nuclei features, and said component channel belongs to the group consisting of a hematoxylin channel, an eosin channel, and a luminance channel.

9. The tissue analysis system of claim 1, wherein:

said image region classifier module is configured to build classifier logic using data of said database and to apply, for each of said image regions, said classifier logic to said image data subset of the respective image region to determine the respective tissue type and the respective confidence score;

said database modifier module is configured to find those image regions having a confidence score falling within said first range and those image regions having a confidence score falling within said second range and to combine said database and the respective tissue types and the respective image data subset of said image regions found to have a confidence score falling within said first range to obtain said modified database;

said image region reclassifier module is configured to modify said classifier logic by means of machine learning using data of said modified database and to apply, for each of said image regions found to have a confidence score falling within said second range, said modified classifier logic to said image data subset of the respective image region to determine the respective reclassified tissue type;

said system is configured to output the respective tissue type of each of said image regions found to have a confidence score falling within said first range and the respective reclassified tissue type of each of said image regions found to have a confidence score falling within said second range as a classification result.

10. The tissue analysis system claim 1, wherein:

said database modifier module is configured to determine, for each of said plurality of tissue types and only for those image regions having a confidence score falling within said first range, the total number of image regions having the respective tissue type, and to effect said modification of said database only for those tissue types for which the total number of image regions having the respective tissue type exceeds a respective threshold number for the respective tissue type.

11. The tissue analysis system of claim 10, wherein, for those tissue types for which the total number of image regions having the respective tissue type exceeds a respective threshold number for the respective tissue type, said database modifier module is configured to effect said modification of said database such that said modified database contains solely tissue characteristic data obtained from the respective image data subsets.

12. A tissue analysis method, comprising:

receiving image data representative of an at least two-dimensional image of a tissue sample;

generating image region data that identifies a plurality of subsets of said image data, each image data subset being representative of a unique, spatially contiguous region of said image;

classifying, for a plurality of said image regions, the respective image region as one of a plurality of tissue types using the respective image data subset and using data obtained from a database of tissue characteristics;

determining a confidence score indicative of a confidence of said classifying of the respective image region;

effecting, for a plurality of said image regions having a confidence score falling within a first range, modification of said database such that the resultant modified database comprises data representative of the tissue type classified to the respective image region and tissue characteristic data obtained from the respective image data subset, said first range being a range of confidence scores that includes a confidence score representative of certainty that the classification is correct;

reclassifying, for a plurality of said image regions having a confidence score falling within a second range distinct from said first range, the respective image region as one of said plurality of tissue types using the respective image data subset for querying said modified database; and

outputting, for at least one of said image regions having a confidence score falling within a second range distinct from said first range, the reclassified tissue type of the respective image region.

13. The tissue analysis method of claim 12, further comprising:

imaging a tissue sample to obtain raw image data, wherein said received image data is obtained from said raw image data.

14. The tissue analysis method of claim 12, further comprising:

staining said tissue sample to obtain a stained tissue sample;

imaging said stained tissue sample to obtain said raw image data, wherein

said received image data is obtained from said raw image data.

15. The tissue analysis method of claim 12, wherein:

said database comprises, for each of a plurality of tissue image regions, data representative of an at least two-dimensional image of tissue, data representative of at least one tissue feature, data representative of a tissue type and data representative of a confidence score, wherein the database is a pre-built first training database that is used for the image classification and confidence score determination by the image region classifier module, wherein a database modifier module generates a second training database that is constituted by the data representative of the tissue type classified to the image regions and the tissue characteristic data obtained from the image data subsets of the plurality of said image regions having a confidence score falling within the first range and combines the first and the second training database to provide the modified database such as by copying some or all of the content of the second training database into the first database.

16. The tissue analysis method of claim 12, wherein said determining a confidence score comprises a support vector machine operation.

17. The tissue analysis method of claim 12, wherein said classifying the respective image region comprises extracting at least one feature from the respective image region using the respective image data subset and said data obtained from said database, said feature belonging to the group consisting of textural features, biological features, intensity features, gradient features, Gabor features, co-occurrence features, and nuclei features.

18. The tissue analysis method of claim 12, wherein said reclassifying the respective image region comprises weighting data of the respective image data subset and the data obtained from said modified database using at least one of a spatial proximity value, a confidence score and feature similarity value.

19. The tissue analysis method of claim 12, wherein said classifying the respective image region comprises separating at least the respective region of said image into one or more component channels and extracting at least one feature from the respective image region using any of said component channels of the respective image region and said data obtained from said database, wherein

said feature belongs to the group consisting of textural features, biological features, intensity features, gradient features, Gabor features, co-occurrence features, and nuclei features, and said component channel belongs to the group consisting of a hematoxylin channel, an eosin channel and a luminance channel.

20. The tissue analysis method of claim 12, further comprising:

building classifier logic using data of said database;

applying, for each of said image regions, said classifier logic to said image data subset of the respective image region to determine the respective tissue type and the respective confidence score;

finding those image regions having a confidence score falling within said first range and those image regions having a confidence score falling within said second range;

combining said database and the respective tissue types and the respective image data subset of said image regions found to have a confidence score falling within said first range to obtain said modified database;

modifying said classifier logic by means of machine learning using data of said modified database;

applying, for each of said image regions found to have a confidence score falling within said second range, said modified classifier logic to said image data subset of the respective image region to determine the respective reclassified tissue type; and

outputting the respective tissue type of each of said image regions found to have a confidence score falling within said first range and the respective reclassified tissue type of each of said image regions found to have a confidence score falling within said second range as a classification result.

21. The tissue analysis method of claim 12, further comprising:

determining, for each of said plurality of tissue types and only for those image regions having a confidence score falling within said first range, the total number of image regions having the respective tissue type, wherein said modification of said database is only effected for those tissue types for which the total number of image regions having the respective tissue type exceeds a respective threshold number for the respective tissue type.

22. The tissue analysis method of claim 21, wherein, for those tissue types for which the total number of image regions having the respective tissue type exceeds a respective threshold number for the respective tissue type, said modification of said database is effected such that said modified database contains solely tissue characteristic data obtained from the respective image data subsets.

23. A method for segmentation of a tissue image being implemented by an electronic device, comprising:

identifying grid points in the tissue image;

classifying the grid points as one of a plurality of tissue types, and generating classified grid points based on a database of known characteristics of tissue types;

assigning the classified grid points at least one of a high confidence score and a low confidence score;

modifying the database of known characteristics of tissue types based on the grid points that were assigned a high confidence score, and generating a modified database; and

reclassifying the grid points that were assigned a low confidence score based on the modified database.

24. A non-transitory computer-readable medium storing instructions which, when executed by a processor, cause the processor to:

assign an image patch of a tissue sample with a confidence score based on a comparison with a database of known features associated with said tissue sample; and

refine the confidence score for the image patch based on a comparison of the image patch with one or more high-confidence image patches from the same tissue sample;

wherein the high-confidence image patches are stored in a database of high-confidence image patches associated with the tissue sample.