Applying pixelwise descriptors to a target image that are generated by segmenting objects in other images

• US 9,805,248 B2
• Filed: 11/26/2015
• Issued: 10/31/2017
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: Definiens AG (Astrazeneca PLC)
• Original Assignee: Definiens AG (Astrazeneca PLC)
• Inventors: Brieu, Nicolas
• Primary Examiner(s): Moyer, Andrew
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US14/952,982
• Publication Number:
  US 20160098589A1
• Time in Patent Office: 705 Days
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/695 Preprocessing, e.g. image segmentation
• CPC Class Codes: G06V 20/698 Matching; Classification
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 40/1394 using acquisition arrangements

Abstract

Both pixel-oriented analysis and the more accurate yet slower object-oriented analysis are used to recognize patterns in images of stained cancer tissue. Images of tissue from other patients that are similar to tissue of a target patient are identified using the standard deviation of color in the images. Object-oriented segmentation is then used to segment small portions of the images of the other patients into object exhibiting object characteristics. Pixelwise descriptors associate each pixel in the remainder of the images with object characteristics based on the color of pixels at predetermined offsets from the characterized pixel. Pixels in the image of the target patient are assigned object characteristics without performing the slow segmentation of the image into objects. A pixel heat map is generated from the target image by assigning pixels the color corresponding to the object characteristic that the pixelwise descriptors indicate is most likely associated with each pixel.

Claims

1. A method comprising:

dividing digital images of stained tissue of cancer patients into tiles;

separating the tiles into clusters whose tiles have pixel characteristics that are similar;

identifying a matching cluster of tiles whose pixel characteristics most closely match the pixel characteristics of a target digital image of stained tissue of a target patient;

segmenting the tiles of the matching cluster into objects;

determining object characteristics of the segmented objects;

assigning a color to each of the object characteristics;

generating pixelwise descriptors that describe which of the object characteristics a characterized pixel most probably exhibits based on a quality of a second pixel at a predetermined offset from the characterized pixel;

generating a pixel heat map by applying the pixelwise descriptors to each pixel of the target digital image without segmenting the target digital image into objects, wherein each pixel of the target digital image has the color assigned to the object characteristic most probably exhibited by that pixel; and

displaying the pixel heat map on a graphical user interface.

2. The method of claim 1, further comprising:

indicating a probability that each pixel of the target digital image exhibits a particular object characteristic by applying the pixelwise descriptors in a random forest decision tree.

3. The method of claim 1, wherein the pixel characteristics of only a predefined portion of each tile are used to identify the matching cluster of tiles.

4. The method of claim 1, wherein the pixel characteristics are taken from the group consisting of:

a mean color vector of all pixels in an intermediate contrast region of a tile, a mean of the mean color vectors of individual tiles of a cluster, a standard deviation of the color vectors of the tiles of a cluster, and an l^2 Euclidean norm of the standard deviation of the color vectors of the tiles of a cluster.

5. The method of claim 1, wherein the stained tissue has been immunohistochemically stained for the expression of a protein.

6. The method of claim 1, wherein objects that exhibit a particular object characteristic belong to a class, and wherein the object characteristics are taken from the group consisting of:

an average number of concavities of the objects of the class, an average size of the objects of the class, a variation in sizes of the objects of the class, an amount of intensity variation within the objects of the class, an elliptical fit of the objects of the class, and an average intensity of the objects of the class.

7. The method of claim 1, wherein a plurality of colors are assigned to each of the object characteristics depending on the degree of the object characteristic, and wherein each pixel of the target digital image has the color assigned to the degree to which that pixel exhibits the object characteristic.

8. The method of claim 1, wherein the quality of the second pixel at the predetermined offset from the characterized pixel is taken from the group consisting of:

a color value of the second pixel, a difference between a color value of the second pixel and a color value of the characterized pixel, and a difference between an intensity value of the second pixel and an intensity value of the characterized pixel.

9. The method of claim 1, wherein the segmenting the tiles of the matching cluster into objects involves generating a data network in which objects are linked to selected pixels of the tiles.

10. The method of claim 1, wherein the separating the tiles into clusters having similar pixel characteristics involves successively separating the tiles into clusters until each cluster exhibits a standard deviation of color whose magnitude falls below a predetermined threshold.

11. A method comprising:

dividing digital images of stained tissue of cancer patients into tiles;

separating the tiles into clusters whose tiles have similar pixel characteristics;

identifying a matching cluster of tiles whose pixel characteristics most closely match the pixel characteristics of target tiles of a digital image of stained tissue of a target patient;

segmenting the target tiles and the tiles of the matching cluster into objects;

determining object characteristics of the segmented objects;

assigning a color to each of the object characteristics;

generating pixelwise descriptors that describe which of the object characteristics a characterized pixel most probably exhibits based on a quality of a second pixel at a predetermined offset from the characterized pixel;

generating a pixel heat map by applying the pixelwise descriptors to each pixel of the target digital image without segmenting the target digital image into objects, wherein each pixel of the target digital image has the color assigned to the object characteristic most probably exhibited by that pixel; and

displaying the pixel heat map on a graphical user interface.

12. The method of claim 11, wherein the pixelwise descriptors are generated using a random forest of decision trees applied to random different pixels of the target tiles and the tiles of the matching cluster.

13. The method of claim 11, wherein the stained tissue has been stained by hematoxylin and eosin (H&

E).

14. The method of claim 11, wherein objects that exhibit a particular object characteristic belong to a class, and wherein the object characteristics are taken from the group consisting of:

an average number of concavities of the objects of the class, an average size of the objects of the class, a variation in sizes of the objects of the class, an amount of intensity variation within the objects of the class, an elliptical fit of the objects of the class, and an average intensity of the objects of the class.

15. The method of claim 11, wherein the quality of the second pixel at the predetermined offset from the characterized pixel is taken from the group consisting of:

a color value of the second pixel, a difference between a color value of the second pixel and a color value of the characterized pixel, and a difference between an intensity value of the second pixel and an intensity value of the characterized pixel.

16. A method comprising:

dividing digital images of stained tissue of cancer patients into tiles;

identifying clusters of tiles whose pixel characteristics are similar;

identifying a matching cluster of tiles whose pixel characteristics most closely match the pixel characteristics of a target digital image of stained tissue of a target patient, wherein the pixel characteristics used to identify the matching cluster of tiles are determined only from regions on the tiles that exhibit an intermediate degree of local contrast;

segmenting the tiles of the matching cluster into objects;

determining object characteristics of the objects;

generating pixelwise descriptors that describe which of the object characteristics a characterized pixel most probably exhibits based on a quality of a second pixel at a predetermined offset from the characterized pixel; and

generating a pixel heat map by applying the pixelwise descriptors to each pixel of the target digital image without segmenting the target digital image into objects, wherein each pixel of the target digital image has a color assigned to the object characteristic most probably exhibited by that pixel.

17. The method of claim 16, wherein the color assigned to the object characteristic is taken from the group consisting of:

a color assigned to pixels that likely belong to round healthy cells, a color assigned to pixels that likely belong to irregularly shaped cancerous cells, a color assigned to pixels that likely belong to nuclei containing textured DNA, and a color assigned to pixels that likely belong to nuclei whose DNA is homogeneously disbursed.

18. The method of claim 16, wherein the object characteristics are taken from the group consisting of:

an average number of concavities of the objects, an average size of the objects, a variation in sizes of the objects, an amount of intensity variation within the objects, an elliptical fit of the objects, and an average intensity of the objects.

19. The method of claim 16, wherein the stained tissue has been stained by hematoxylin and eosin (H&

E).

20. The method of claim 16, wherein the stained tissue has been immunohistochemically stained for the expression of a protein.