Combinational pixel-by-pixel and object-level classifying, segmenting, and agglomerating in performing quantitative image analysis that distinguishes between healthy non-cancerous and cancerous cell nuclei and delineates nuclear, cytoplasm, and stromal material objects from stained biological tissue materials

• US 8,488,863 B2
• Filed: 11/06/2008
• Issued: 07/16/2013
• Est. Priority Date: 11/06/2008
• Status: Expired due to Fees

• Current Assignee: TRIAD National Security, LLC
• Original Assignee: Los Alamos National Security LLC (Government of the United States of America)
• Inventors: Boucheron, Laura E.
• Primary Examiner(s): Koval, Melissa
• Assistant Examiner(s): FETZNER, TIFFANY A
• Application Number:
  US12/266,450
• Publication Number:
  US 20100111396A1
• Time in Patent Office: 1,713 Days
• Field of Search: 324300-322, 600407-435, 382128-132, 382/133
• US Class Current: 382/133
• CPC Class Codes: G06F 18/2115 by evaluating different subsets according to an optimisation criterion, e.g. class separability, forward selection or backward elimination
• CPC Class Codes: G06F 18/29 Graphical models, e.g. Bayesian networks
• CPC Class Codes: G06T 2207/10056 Microscopic image
• CPC Class Codes: G06T 2207/30024 Cell structures in vitro; Tissue sections in vitro
• CPC Class Codes: G06T 2207/30068 Mammography; Breast
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/11 Region-based segmentation
• CPC Class Codes: G06T 7/187 involving region growing; involving region merging; involving connected component labelling
• CPC Class Codes: G06V 10/426 Graphical representations
• CPC Class Codes: G06V 10/771 Feature selection, e.g. selecting representative features from a multi-dimensional feature space
• CPC Class Codes: G06V 10/84 using probabilistic graphical models from image or video features, e.g. Markov models or Bayesian networks
• CPC Class Codes: G06V 20/695 Preprocessing, e.g. image segmentation

Abstract

Quantitative object and spatial arrangement-level analysis of tissue are detailed using expert (pathologist) input to guide the classification process. A two-step method is disclosed for imaging tissue, by classifying one or more biological materials, e.g. nuclei, cytoplasm, and stroma, in the tissue into one or more identified classes on a pixel-by-pixel basis, and segmenting the identified classes to agglomerate one or more sets of identified pixels into segmented regions. Typically, the one or more biological materials comprises nuclear material, cytoplasm material, and stromal material. The method further allows a user to markup the image subsequent to the classification to re-classify said materials. The markup is performed via a graphic user interface to edit designated regions in the image.

Claims

1. A method of analyzing an image of stained biological tissue material, the method comprising:

(a) obtaining a first image of a stained biological tissue material using a microscope coupled to an image capture device;

(b) inputting said first image into a computer processor; and

(c) executing programming on the computer processor, said programming performing steps comprising;

(i) performing a first level classification of said stained biological tissue material within said first image on a pixel by pixel basis by assigning a classification to each pixel;

wherein said classification is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of color, or texture, or an anomaly compared to the rest of said image, or pixel location within said image, or topological relationship between pixels, or connectivity relationship between pixels; and

producing, on an output device, a second image comprising classified pixels;

(ii) segmenting said classified pixels within said second image by applying a segmentation algorithm;

wherein said segmenting is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of contour, or color, or texture, or an anomaly compared to the rest of said image, or pixel location within said image, shape, or topological relationship between said classified pixels, or connectivity relationship between said classified pixels;

agglomerating at least one set of said classified pixels whereby nuclear material objects, cytoplasm material objects, and stromal material objects in said stained biological tissue material is capable of being delineated; and

producing, on the output device, a third image of said biological tissue material from agglomerated sets of classified and segmented pixels; and

wherein said third image comprises at least one of delineated nuclear material objects, or delineated cytoplasm material objects, or delineated stromal material objects;

(iii) classifying at least one of said objects within said third image from step (ii) on a second level object basis by assigning a classification to at least one of said objects within said third image from step (ii);

wherein said classification is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of contour, color, texture, an anomaly compared to the rest of said image, object location within said image, shape, topological relationship between said objects within said third image from step (ii), or connectivity relationship between said objects within said third image from step (ii); and

producing, on the output device, a fourth image comprising at least one of classified delineated nuclear material objects, or classified delineated cytoplasm material objects, or classified delineated stromal material objects;

(iv) segmenting at least one of said objects within said fourth image from step (iii) by applying a segmentation algorithm;

wherein said segmenting is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of contour, or color, or texture, or an anomaly compared to the rest of said image, or object location within said image, shape, or topological relationship between said objects within said fourth image from step (iii), or connectivity relationship between said objects within said fourth image from step (iii);

agglomerating one or more sets of said objects within said fourth image from step (iii) in order to delineate cells or extracellular objects from said stained biological tissue material; and

producing, on an output device, a fifth image of said stained biological tissue material from agglomerated sets of said classified and segmented objects within said fourth image from step (iii);

wherein said fifth image comprises delineated cells or delineated extracellular objects or both;

(v) generating information on at least one of histology, or cytopathology, or histopathology based on at least one of said second image or an image produced in a subsequent step, and providing the information to the user, wherein said information is capable of providing information beyond just distinguishing cell nuclei as cancerous or non-cancerous; and

(vi) outputting said at least said generated information to an output device;

wherein said generated information is available to the user.

2. A method as recited in claim 1, further comprising:

optionally adding additional classifying and segmenting steps in order to produce a further image of said stained biological tissue material comprising additional delineated tissue.

3. A method as recited in claim 2, further comprising:

displaying at least one of said produced images on the output device;

allowing a said user to markup at least one of said images in order to re-classify said stained biological tissue material and produce an image markup; and

using said image markup subsequent to classification in order to develop one or more said classifiers for specific classifications of said stained biological tissue material, wherein said one or more classifiers are used in order to re-classify materials in subsequent images.

4. A method as recited in claim 3, wherein said image markup is used as an input for a classification algorithm of said subsequent images.

5. A method as recited in claim 3, further comprising:

inputting said image markup as training data into said computer processor; and

incorporating machine learning using said training data in order to develop one or more classifiers, wherein at least one of said developed classifiers are used in order to classify subsequent images.

6. A method as recited in claim 2, further comprising:

displaying segmented regions of at least one of said third image or images produced in a subsequent step following the generation of the third image;

allowing the user to markup at least one of said third image or images produced in a subsequent step following the generation of the third image in order to re-segment said stained biological tissue material and produce an image markup; and

using said image markup subsequent to re-segmentation in order to further develop said segmentation algorithms of steps (ii) and (iv).

7. A method as recited in claim 3, wherein said markup is performed via a graphic user interface in order to edit re-segmented regions in the images.

8. A method as recited in claim 6, wherein said markup is performed via a graphic user interface in order to edit re-segmented regions in the images.

9. A method as recited in claim 6, wherein each segmentation algorithm of steps (ii) and (iv) provides a segmentation output, the method further comprising:

applying a segmentation metric in order to quantitatively compare the outputs of the individual segmentation algorithms of steps (ii) and (iv); and

subsequently selecting an output based on said segmentation metric comparison.

10. A method as recited in claim 9, wherein the segmentation metric is a function of:

a number of segmented regions;

a size and shape of the region of pixels missed;

a size and shape of excess pixels;

a fraction of nuclei detected; and

a number of extra segmented regions.

11. A method as recited in claim 10, wherein the at least one said segmentation is a function of the equation:

12. A method as recited in claim 2, wherein classifying one or more biological tissue materials comprises

classifying cytoplasm material and stromal material by subtracting out pixels related to image background and nuclear material.

13. An apparatus configured for analyzing an image of stained biological tissue material, the apparatus comprising:

(a) a first image of a stained biological tissue material that is input into a computer processor;

(b) an output device connected to the computer processor that is capable of displaying images; and

(c) non-transitory programming executable on the computer processor for performing steps comprising;

(i) performing a first level classification of said stained biological tissue material within said first image on a pixel by pixel basis by assigning a classification to each pixel;

wherein said classification is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of color, or texture, or an anomaly compared to the rest of said image, or pixel location within said image, or topological relationship between pixels, or connectivity relationship between pixels; and

producing, on an output device, a second image comprising classified pixels;

(ii) segmenting said classified pixels within said second image by applying a segmentation algorithm;

wherein said segmenting is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of contour, or color, or texture, or an anomaly compared to the rest of said image, or pixel location within said image, shape, or topological relationship between said classified pixels, or connectivity relationship between said classified pixels;

agglomerating at least one set of said classified pixels whereby nuclear material objects, cytoplasm material objects, and stromal material objects in said stained biological tissue material is capable of being delineated; and

producing, on the output device, a third image of said biological tissue material from agglomerated sets of classified and segmented pixels; and

wherein said third image comprises at least one of delineated nuclear material objects, or delineated cytoplasm material objects, or delineated stromal material objects;

(iii) classifying at least one of said objects within said third image from step (ii) on a second level object basis by assigning a classification to at least one of said objects within said third image from step (ii);

wherein said classification is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of contour, color, texture, an anomaly compared to the rest of said image, object location within said image, shape, topological relationship between said objects within said third image from step (ii), or connectivity relationship between said objects within said third image from step (ii); and

producing, on the output device, a fourth image comprising at least one of classified delineated nuclear material objects, or classified delineated cytoplasm material objects, or classified delineated stromal material objects;

(iv) segmenting at least one of said objects within said fourth image from step (iii) by applying a segmentation algorithm;

wherein said segmenting is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of contour, or color, or texture, anomalies or an anomaly compared to the rest of said image, or object location within said image, shape or topological relationship between said objects within said fourth image from step (iii), or connectivity relationship between said objects within said fourth image from step (iii);

agglomerating one or more sets of said objects within said fourth image from step (iii) in order to delineate cells or extracellular objects from said stained biological tissue material; and

producing, on an output device, a fifth image of said stained biological tissue material from agglomerated sets of said classified and segmented objects within said fourth image from step (iii);

wherein said fifth image comprises delineated cells or delineated extracellular objects or both;

(v) generating information on at least one of histology, or cytopathology, or histopathology based on at least one of said second image or an image produced in a subsequent step, and providing the information to the user, wherein said information is capable of providing information beyond just distinguishing cell nuclei as cancerous or non-cancerous; and

(vi) outputting said at least said generated information to an output device;

wherein said generated information is available to the user.

14. An apparatus as recited in claim 13, further comprising:

non-transitory programming executable on the computer processor for performing steps comprising;

optional classifying and segmenting steps in order to produce a further image of stained biological tissue material comprising additional delineated tissue.

15. An apparatus as recited in claim 13, further comprising:

non-transitory programming executable on the computer processor for performing steps comprising;

extracting one or more parameters from said third image in order to classify at least one of said delineated nuclear material objects, said delineated cytoplasm material objects and said delineated stromal material objects.

16. An apparatus as recited in claim 13, further comprising:

non-transitory programming executable on the computer processor for performing steps comprising;

extracting one or more parameters from said fifth image in order to classify at least one of said delineated cells and extracellular objects.

17. An apparatus as recited in claim 14, further comprising:

non-transitory programming executable on the computer processor for performing steps comprising;

extracting one or more parameters from said further image in order to classify at least one of said delineated cells and extracellular objects.

18. An apparatus as recited in claim 13, further comprising:

non-transitory programming executable on the computer processor for performing steps comprising;

receiving an image markup, wherein said image markup is created using a user interface configured to allow said user to markup an image subsequent to classifying in order to re-classify, at least one of said pixels, said delineated nuclear material objects, said delineated cytoplasm material objects and said delineated stromal material objects, said delineated cells, said delineated extracellular objects and said delineated tissue.

19. An apparatus as recited in claim 18, further comprising:

non-transitory programming executable on the computer processor for performing steps comprising;

inputting said image markup as training data; and

incorporating machine learning using said training data in order to develop one or more classifiers, wherein any one or more of said developed classifiers are used in order to re-classify subsequent images.

20. An apparatus as recited in claim 13, further comprising:

non-transitory programming executable on the computer processor for performing steps comprising;

receiving an image markup, wherein said image markup is created using an user interface configured to allow said user to markup an image subsequent to segmenting in order to re-segment at least one of said classified pixels, said categorized delineated nuclear material objects, said classified delineated nuclear material objects and said classified delineated cells and extracellular objects.

21. An apparatus as recited in claim 20, further comprising:

non-transitory programming executable on the computer processor for performing steps comprising;

inputting said image markup as training data; and

incorporating machine learning using said training data in order to further develop at least one of said segmentation algorithms of steps ii and iv of claim 13, that is then used in order to re-segment subsequently produced images on said output device.

22. An apparatus as recited in claim 21, further comprising:

non-transitory programming executable on the computer processor for performing steps comprising;

comparing the outputs of each further developed segmentation algorithm quantitatively using a segmentation metric; and

subsequently selecting an output based on said segmentation metric comparison.

23. An apparatus as recited in claim 22, wherein the segmentation metric is a function of:

a number of segmented regions;

a size and shape of the region of pixels missed;

a size and shape of excess pixels;

a fraction of nuclei detected; and

a number of extra segmented regions.

24. A non-transitory computer readable media containing programming executable on a computer processor in order to analyze an image of stained biological tissue material by performing the steps comprising:

(i) performing a first level classification of said stained biological tissue material within said first image on a pixel by pixel basis by assigning a classification to each pixel;

wherein said classification is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of color, or texture, or an anomaly compared to the rest of said image, or pixel location within said image, or topological relationship between pixels, or connectivity relationship between pixels; and

producing, on an output device, a second image comprising classified pixels;

(ii) segmenting said classified pixels within said second image by applying a segmentation algorithm;

wherein said segmenting is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of contour, or color, or texture, or an anomaly compared to the rest of said image, or pixel location within said image, shape, or topological relationship between said classified pixels, or connectivity relationship between said classified pixels;

agglomerating at least one set of said classified pixels whereby nuclear material objects, cytoplasm material objects, and stromal material objects in said stained biological tissue material is capable of being delineated; and

producing, on the output device, a third image of said biological tissue material from agglomerated sets of classified and segmented pixels; and

wherein said third image comprises at least one of delineated nuclear material objects, or delineated cytoplasm material objects, or delineated stromal material objects;

(iii) classifying at least one of said objects within said third image from step (ii) on a second level object basis by assigning a classification to at least one of said objects within said third image from step (ii);

wherein said classification is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of contour, color, texture, an anomaly compared to the rest of said image, object location within said image, shape, topological relationship between said objects within said third image from step (ii), or connectivity relationship between said objects within said third image from step (ii); and

producing, on the output device, a fourth image comprising at least one classified delineated nuclear material objects, or classified delineated cytoplasm material objects, or classified delineated stromal material objects;

(iv) segmenting at least one of said objects within said fourth image from step (iii) by applying a segmentation algorithm;

wherein said segmenting is based on a desired staining property of said stained biological tissue material or a desired spatial property of said stained biological tissue material or both;

said staining and spatial properties comprising at least one of contour, or color, or texture, or an anomaly compared to the rest of said image, or object location within said image, shape, or topological relationship between said objects within said fourth image from step (iii), or connectivity relationship between said objects within said fourth image from step (iii);

agglomerating one or more sets of said objects within said fourth image from step (iii) in order to delineate cells or extracellular objects from said biological tissue material; and

producing, on an output device, a fifth image of said biological tissue material from agglomerated sets of said classified and segmented objects within said fourth image from step (iii);

wherein said fifth image comprises delineated cells or delineated extracellular objects or both;

(v) generating information on at least one of histology, or cytopathology, or histopathology based on at least one of said second image or an image produced in a subsequent step, and providing the information to the user, wherein said information is capable of providing information beyond just distinguishing cell nuclei as cancerous or non-cancerous; and

(vi) outputting at least said generated information to an output device;

wherein said generated information is available to the user.

25. A non-transitory computer readable media containing programming executable on a computer processor in order to analyze an image of stained biological tissue material by performing steps as recited in claim 24, further comprising:

optional additional classifying and segmenting steps in order to produce a further image of said stained biological tissue material comprising delineated tissue.

26. A non-transitory computer readable media containing programming executable on a computer processor in order to analyze an image of stained biological tissue material by performing steps as recited in claim 24, further comprising:

extracting one or more parameters from said third image in order to classify at least one of said delineated nuclear material objects, said delineated cytoplasm material objects and said delineated stromal material objects.

27. A non-transitory computer readable media containing programming executable on a computer processor in order to analyze an image of stained biological tissue material by performing steps as recited in claim 24, further comprising:

extracting one or more parameters from said fifth image in order to classify at least one of said delineated cells and extracellular objects.

28. A non-transitory computer readable media containing programming executable on a computer processor in order to analyze an image of stained biological tissue material by performing steps as recited in claim 25, further comprising:

extracting one or more parameters from said further image in order to classify at least one of said delineated cells and extracellular objects.

29. A non-transitory computer readable media containing programming executable on a computer processor in order to analyze an image of stained biological tissue material by performing steps as recited in claim 28, further comprising:

receiving an image markup, wherein said image markup is created using a user interface configured to allow said user to markup an image subsequent to classifying in order to re-classify, any one or more of said pixels, said delineated nuclear material objects, said delineated cytoplasm material objects and said delineated stromal material objects, said delineated cells, said delineated extracellular objects and said delineated tissue.

30. A non-transitory computer readable media containing programming executable on a computer processor in order to analyze an image of stained biological tissue material by performing steps as recited in claim 29, further comprising:

inputting said image markup as training data; and

incorporating machine learning using said training data in order to develop classifiers, wherein any one or more of said developed classifiers are used in order to re-classify subsequent images.

31. A non-transitory computer readable media containing programming executable on a computer processor in order to analyze an image of stained biological tissue material by performing steps as recited in claim 24, further comprising:

receiving an image markup, wherein said image markup is created using an user interface configured to allow said user to markup an image subsequent to segmenting in order to re-segment any one or more of;

said classified pixels, said classified delineated nuclear material objects, said classified delineated nuclear material objects and said classified delineated cells and extracellular objects.

32. A non-transitory computer readable media containing programming executable on a computer processor in order to analyze an image of biological tissue material by performing steps as recited in claim 31, further comprising:

inputting said image markup as training data; and

incorporating machine learning using said training data in order to further develop at least one of said segmentation algorithms of steps ii and iv of claim 24, that is then used in order to re-segment subsequently produced images on said output device.

33. A non-transitory computer readable media containing programming executable on a computer processor in order to analyze an image of stained biological tissue material by performing steps as recited in claim 24, further comprising:

comparing the outputs of each further developed segmentation algorithm quantitatively using a segmentation metric; and

subsequently selecting an output based on said segmentation metric comparison.

34. A non-transitory computer readable media containing programming executable on a computer processor in order to analyze an image of stained biological tissue material by performing steps as recited in claim 33, wherein the segmentation metric is a function of:

a number of segmented regions;

a size and shape of the region of pixels missed;

a size and shape of excess pixels;

a fraction of nuclei detected; and

a number of extra segmented regions.