Method and system for hierarchical tissue analysis and classification

• US 8,600,143 B1
• Filed: 05/20/2011
• Issued: 12/03/2013
• Est. Priority Date: 05/20/2010
• Status: Active Grant

• Current Assignee: KLA-Tencor Corporation
• Original Assignee: KLA-Tencor Corporation
• Inventors: Kulkarni, Ashok V., Young, Scott, Rosengaus, Eliezer
• Primary Examiner(s): NAKHJAVAN, SHERVIN K
• Application Number:
  US13/112,977
• Time in Patent Office: 928 Days
• Field of Search: 382/128, 382/129, 382/133, 382159-161, 382/224, 382/225, 382/227, 435/1.1, 73/617.2, 700/47
• US Class Current: 382/133
• CPC Class Codes: G06F 18/231 Hierarchical techniques, i.e. dividing or merging pattern sets so as to obtain a dendrogram
• CPC Class Codes: G06T 2207/10004 Still image; Photographic image
• CPC Class Codes: G06T 2207/10056 Microscopic image
• CPC Class Codes: G06T 2207/20076 Probabilistic image processing
• CPC Class Codes: G06T 2207/20081 Training; Learning
• CPC Class Codes: G06T 2207/30024 Cell structures in vitro; Tissue sections in vitro
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/143 involving probabilistic approaches, e.g. Markov random field [MRF] modelling
• CPC Class Codes: G06V 10/26 Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
• CPC Class Codes: G06V 10/7625 Hierarchical techniques, i.e. dividing or merging patterns to obtain a tree-like representation; Dendograms
• CPC Class Codes: G06V 20/698 Matching; Classification

Abstract

The present invention may include segmenting an image at a first resolution level and a second resolution level, wherein one or more parameters of a segmentation algorithm are trainable via user classification feedback, extracting features from a first plurality of segment primitives and a second plurality of segment primitives, wherein one or more parameters of a segment feature extraction algorithm are trainable via user classification feedback, building a first and second segmentation hierarchy by generating one or more clusters of the first plurality of segment primitives and the second plurality of segment primitives, extracting one or more features from the first segmentation hierarchy and the second segmentation hierarchy utilizing a hierarchy feature extraction algorithm, determining an inter-level relationship between the clusters generated for the first resolution level and the second level, and automatically classifying one or more tissue elements of the tissue specimen via a user-trained classification algorithm.

Claims

1. A method for hierarchical segmentation and classification of image data from a tissue specimen, comprising:

receiving one or more images of a tissue specimen;

segmenting the one or more images at a first resolution level and at least a second resolution level utilizing a segmentation algorithm, wherein one or more parameters of the segmentation algorithm are trainable via user classification feedback, wherein the image at the first resolution level is segmented into a first plurality of segment primitives and the image at the at least a second resolution level is segmented into a second plurality of segment primitives;

extracting one or more features from some of the first plurality of segment primitives and some of the at least a second plurality of segment primitives utilizing a segment feature extraction algorithm, wherein one or more parameters of the segment feature extraction algorithm are trainable via user classification feedback;

building a first segmentation hierarchy by generating one or more clusters of some of the first plurality of segment primitives utilizing a clustering algorithm, wherein one or more of the clustering algorithm parameters are trainable via user classification feedback;

building at least a second segmentation hierarchy by generating one or more clusters of some of the at least a second plurality of segment primitives utilizing the clustering algorithm;

extracting one or more features from the first segmentation hierarchy and the at least a second segmentation hierarchy utilizing a hierarchy feature extraction algorithm, wherein one or more hierarchy feature extraction algorithm parameters are trainable via feedback from user classification feedback;

determining an inter-level relationship between one or more clusters generated for the first resolution level and one or more clusters generated for the at least a second level by comparing one or more characteristics of the one or more clusters of the first resolution level to one or more characteristics of the one or more clusters of the at least a second resolution level; and

autoclassifying one or more tissue elements of the tissue specimen via a user-trained classification algorithm utilizing at least one of the extracted features from some of the first plurality of segment primitives, the extracted features from some of the at least a second plurality of segment primitives, the extracted features from the first segmentation hierarchy, the extracted features from the at least a second segmentation hierarchy, the determined inter-level relationship between clusters of the first resolution level and the clusters of the at least a second resolution level, or user classification feedback.

2. The method of claim 1, wherein the receiving one or more images of a tissue specimen comprises:

receiving at least two images of a tissue specimen, wherein a first image is acquired via a first imaging mode and at a second image is acquired via at least a second imaging mode.

3. The method of claim 1, further comprising:

pre-analysis refinement processing of the one or more images.

4. The method of claim 1, further comprising:

displaying one or more regions of interest on a display device.

5. The method of claim 4, wherein the displaying one or more regions of interest on a display device comprises:

displaying one or more regions of interest on a display device in order to receive user classification feedback.

6. The method of claim 4, wherein the displaying one or more regions of interest on a display device comprises:

displaying one or more regions of interest on a display device, wherein the one or more regions of interest are identified utilizing the extracted features from some of the first plurality of segment primitives, the extracted features from some of the at least a second plurality of segment primitives, the extracted features from the first segmentation hierarchy, the extracted features from the at least a second segmentation hierarchy, the determined inter-level relationship between clusters of the first resolution level and the clusters of the at least a second resolution level, or user classification feedback.

7. The method of claim 1, further comprising:

displaying one or more regions of interest on a display device, wherein the one or more regions of interest are determined by the autoclassification of the one or more tissue elements.

8. The method of claim 1, wherein the user feedback is provided via a user interface, the user interface comprising at least one of a mouse, a trackpad, a trackball, a keyboard, a touch screen interface, and a voice input system.

9. The method of claim 1, wherein the user feedback is provided via a user interface, the user interface comprising an eye movement tracking system.

10. The method of claim 9, wherein the eye movement tracking system comprises:

an eye movement tracking system configured to monitor at least one of eye movement or eye dwell time.

11. The method of claim 1, wherein the one or more images are received from an image source, the image source comprising a microscope imagery output or an image database.

12. The method of claim 1, wherein the first resolution level is greater than the at least a second resolution level.

13. The method of claim 1, wherein the segmenting the one or more images comprises:

segmenting the one or more images into cells, cell clusters, and interconnective tissue.

14. The method of claim 1, wherein the features extracted from some of the first plurality of segment primitives and some of the at least a second plurality of segment primitives comprise at least one of color, size, shape, texture, average hue, intensity or saturation.

15. The method of claim 1, wherein the one or more clusters of at least one of the first plurality of segment primitives or the at least a second segment primitive is constructed based on the types of segment primitives or the spatial distribution of the segment primitives.

16. The method of claim 1, wherein the features extracted from the first segmentation hierarchy and the at least a second segmentation hierarchy comprise at least one of the relative proportion of constituent segment types, spatial arrangement of constituent segment types, number of hierarchy nodes, average degree of nodes, eccentricity, clustering coefficient, cluster diameter, hue, saturation, or intensity level.

17. The method of claim 1, wherein the one or more characteristics of the one or more clusters of the first resolution level and the one or more characteristics of the one or more clusters of the at least a second resolution level used to determine an inter-level relationship between one or more clusters generated for the first resolution level and one or more clusters generated for the at least a second level comprises at least one of the group including types of intersecting clusters, spatial arrangement of clusters, or number of each cluster type.

18. The method of claim 1, further comprising:

tagging an eye dwell position of a user in an image being viewed by the user utilizing one or more user input devices.

19. The method of claim 18, wherein the tagging an eye dwell position of a user in an image being viewed by the user utilizing one or more user input devices comprise:

tagging an eye dwell position of a user in an image being viewed by the user utilizing one or more user input devices in order to record a class label associated with a location in the image.

20. An apparatus for hierarchical segmentation and classification of image data from a tissue specimen, comprising:

one or more image sources;

one or more user interfaces;

one or more display devices; and

one or more computer systems configured to;

receive one or more images of a tissue specimen;

segment the one or more images at a first resolution level and at least a second resolution level utilizing a segmentation algorithm, wherein one or more parameters of the segmentation algorithm are trainable via user classification feedback, wherein the image at the first resolution level is segmented into a first plurality of segment primitives and the image at the at least a second resolution level is segmented into a second plurality of segment primitives;

extract one or more features from some of the first plurality of segment primitives and some of the at least a second plurality of segment primitives utilizing a segment feature extraction algorithm, wherein one or more parameters of the segment feature extraction algorithm are trainable via user classification feedback;

build a first segmentation hierarchy by generating one or more clusters of some of the first plurality of segment primitives utilizing a clustering algorithm, wherein one or more of the clustering algorithm parameters are trainable via user classification feedback;

build at least a second segmentation hierarchy by generating one or more clusters of some of the at least a second plurality of segment primitives utilizing the clustering algorithm;

extract one or more features from the first segmentation hierarchy and the at least a second segmentation hierarchy utilizing a hierarchy feature extraction algorithm, wherein one or more hierarchy feature extraction algorithm parameters are trainable via feedback from user classification feedback;

determine an inter-level relationship between one or more clusters generated for the first resolution level and one or more clusters generated for the at least a second level by comparing one or more characteristics of the one or more clusters of the first resolution level to one or more characteristics of the one or more clusters of the at least a second resolution level; and

autoclassify one or more tissue elements of the tissue specimen utilizing at least one of the extracted features from some of the first plurality of segment primitives, the extracted features from some of the at least a second plurality of segment primitives, the extracted features from the first segmentation hierarchy, the extracted features from the at least a second segmentation hierarchy, the determined inter-level relationship between clusters of the first resolution level and the clusters of the at least a second resolution level, or user classification feedback.

21. The apparatus of claim 20, wherein the user interface comprises:

a mouse, a keyboard, a touch screen interface, or a voice input system.

22. The apparatus of claim 20, wherein the user interface comprises:

an eye movement tracking system.

23. The apparatus of claim 20, wherein the eye movement tracking system comprises:

an eye movement tracking system configured to monitor at least one of eye movement or eye dwell time.

24. The apparatus of claim 23, wherein the eye movement tracking system configured to monitor at least one of eye movement or eye dwell time comprises:

an eye movement tracking system configured to execute an algorithm configured to record image locations of at least one of eye movement or dwell time.

25. The apparatus of claim 24, wherein the eye movement tracking system configured to execute an algorithm configured to record image locations of at least one of eye movement or dwell time comprises:

an eye movement tracking system configured to execute an algorithm configured to record image locations of at least one of eye movement or dwell time, wherein the eye movement tracking system is configured to provide training feedback to a segment classifier or a cluster classifier at one or more resolution levels of an image.