Applying pixelwise descriptors to a target image that are generated by segmenting objects in other images

• US 10,474,874 B2
• Filed: 09/28/2017
• Issued: 11/12/2019
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: Definiens AG (Astrazeneca PLC)
• Original Assignee: Definiens AG (Astrazeneca PLC)
• Inventors: Brieu, Nicolas
• Primary Examiner(s): Bella, Matthew C
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US15/719,544
• Publication Number:
  US 20180053033A1
• Time in Patent Office: 775 Days
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/695 Preprocessing, e.g. image segmentation
• CPC Class Codes: G06V 20/698 Matching; Classification
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 40/1394 using acquisition arrangements

Abstract

Both pixel-oriented analysis and the more accurate yet slower object-oriented analysis are used to recognize patterns in images of stained cancer tissue. Images of tissue from other patients that are similar to tissue of a target patient are identified using the standard deviation of color in the images. Object-oriented segmentation is then used to segment small portions of the images of the other patients into object exhibiting object characteristics. Pixelwise descriptors associate each pixel in the remainder of the images with object characteristics based on the color of pixels at predetermined offsets from the characterized pixel. Pixels in the image of the target patient are assigned object characteristics without performing the slow segmentation of the image into objects. A pixel heat map is generated from the target image by assigning pixels the color corresponding to the object characteristic that the pixelwise descriptors indicate is most likely associated with each pixel.

Claims

1. A method comprising:

dividing images of tissue of cancer patients into tiles;

separating the tiles into clusters, wherein each of the clusters has tiles with similar colors;

identifying a matching cluster whose tiles have colors that most closely match the colors of a target image of tissue of a target patient;

segmenting the tiles of the matching cluster into objects;

classifying the objects into classes;

assigning a color to each object class;

generating pixelwise descriptors that associate an object class with each pixel of the tiles of the matching cluster based on colors of other pixels at predetermined offsets from the classified pixel; and

generating a pixel heat map by applying the pixelwise descriptors to each pixel of the target image without segmenting the target image into objects, wherein each pixel of the target image has the color assigned to the object class associated with that pixel.

2. The method of claim 1, wherein the segmenting the tiles of the matching cluster into objects involves generating a data network in which objects are linked to selected pixels of the tiles.

3. The method of claim 1, wherein the separating the tiles into clusters having similar colors involves successively separating the tiles into clusters until each cluster exhibits a standard deviation of color whose magnitude falls below a predetermined threshold.

4. The method of claim 1, wherein only learning portions of the tiles of the matching cluster are segmented.

5. The method of claim 1, wherein pixel characteristics are used to identify the matching cluster of tiles, and wherein the pixel characteristics are determined only from regions on the tiles that exhibit an intermediate degree of local contrast.

6. The method of claim 1, wherein the pixelwise descriptors are generated using a random forest of decision trees applied to random different pixels of the tiles of the matching cluster.

7. A method comprising:

dividing digital images of stained tissue of cancer patients into tiles;

identifying clusters of tiles whose pixel characteristics are similar;

identifying a matching cluster of tiles whose pixel characteristics most closely match the pixel characteristics of a target digital image of stained tissue of a target patient, wherein the pixel characteristics used to identify the matching cluster of tiles are determined only from regions on the tiles that exhibit an intermediate degree of local contrast;

segmenting the tiles of the matching cluster into objects;

determining an object class to which each of the objects belongs;

generating pixelwise descriptors that describe the object class to which a characterized pixel most probably belongs based on a quality of a second pixel at a predetermined offset from the characterized pixel; and

generating a pixel heat map by applying the pixelwise descriptors to each pixel of the target digital image without segmenting the target digital image into objects, wherein each pixel of the target digital image has a color assigned to the object class to which that pixel most probably belongs.

8. The method of claim 7, wherein the color assigned to the object class is taken from the group consisting of:

a color assigned to pixels that likely belong to an object class of round healthy cells, a color assigned to pixels that likely belong to an object class of irregularly shaped cancerous cells, a color assigned to pixels that likely belong to an object class of nuclei containing textured DNA, and a color assigned to pixels that likely belong to an object class of nuclei whose DNA is homogeneously disbursed.

9. The method of claim 7, wherein the object class is taken from the group consisting of:

a class of objects whose objects have a predetermined average number of concavities, a class of objects whose objects have a predetermined average size, a class of objects whose objects exhibit a predetermined variation in sizes, a class of objects whose objects exhibit a predetermined amount of intensity variation within the objects, a class of objects whose objects exhibit a predetermined elliptical fit, and a class of objects whose objects exhibit a predetermined average intensity.

10. The method of claim 7, wherein the stained tissue has been stained by hematoxylin and eosin (H&

E).

11. The method of claim 7, wherein the stained tissue has been immunohistochemically stained for the expression of a protein.

12. The method of claim 7, further comprising:

displaying the pixel heat map on a graphical user interface.

13. A method of identifying cancer tissue comprising:

dividing images of tissue of a plurality of cancer patients into tiles;

separating the tiles into clusters having similar colors by successively separating the tiles into clusters until each cluster exhibits a standard deviation of color whose magnitude falls below a predetermined threshold;

identifying a matching cluster of tiles whose colors most closely match the colors of a target image of tissue of a target cancer patient;

segmenting the tiles of the matching cluster into objects;

classifying the objects into classes;

assigning a color to each object class;

generating pixelwise descriptors that associate each pixel of the tiles of the matching cluster to an object class based on colors of other pixels at predetermined offsets from the classified pixel; and

generating a pixel heat map by applying the pixelwise descriptors to each pixel of the target image without segmenting the target image into objects.

14. The method of claim 13, further comprising:

determining that tissue of the target cancer patient matches particular tissue of the plurality of cancer patients.

15. The method of claim 13, wherein each pixel of the target image on which the pixelwise descriptors is applied has the color assigned to the object class associated with the characterized pixel.

16. The method of claim 13, wherein only learning portions of the tiles of the matching cluster are segmented.

17. The method of claim 13, wherein pixel characteristics are used to identify the matching cluster of tiles, and wherein the pixel characteristics are determined only from regions on the tiles that exhibit an intermediate degree of local contrast.

18. The method of claim 13, wherein the pixelwise descriptors are generated using a random forest of decision trees applied to random different pixels of the tiles of the matching cluster.

19. The method of claim 13, wherein the object class is taken from the group consisting of:

an object class of round healthy cells, an object class of irregularly shaped cancerous cells, an object class of nuclei containing textured DNA, and an object class of nuclei whose DNA is homogeneously disbursed.

20. The method of claim 13, wherein the objects are classified based on characteristics taken from the group consisting of:

an average number of concavities of the objects, an average size of the objects, a variation in size of the objects, an intensity variation among the objects, an elliptical fit of each of the objects, and an average intensity of the objects.