Applying Pixelwise Descriptors to a Target Image that are Generated by Segmenting Objects in Other Images

• US 20160098589A1
• Filed: 11/26/2015
• Published: 04/07/2016
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: Definiens AG (Astrazeneca PLC)
• Original Assignee: Definiens AG (Astrazeneca PLC)
• Inventors: Brieu, Nicolas
• Granted Patent: US 9,805,248 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/2415 based on parametric or probabilistic models, e.g. based on likelihood ratio or false acceptance rate versus a false rejection rate
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 20/695 Preprocessing, e.g. image segmentation
• CPC Class Codes: G06V 20/698 Matching; Classification
• CPC Class Codes: G06V 30/19173 Classification techniques
• CPC Class Codes: G06V 30/194 References adjustable by an adaptive method, e.g. learning
• CPC Class Codes: G06V 40/1394 using acquisition arrangements

Abstract

Both pixel-oriented analysis and the more accurate yet slower object-oriented analysis are used to recognize patterns in images of stained cancer tissue. Images of tissue from other patients that are similar to tissue of a target patient are identified using the standard deviation of color in the images. Object-oriented segmentation is then used to segment small portions of the images of the other patients into object exhibiting object characteristics. Pixelwise descriptors associate each pixel in the remainder of the images with object characteristics based on the color of pixels at predetermined offsets from the characterized pixel. Pixels in the image of the target patient are assigned object characteristics without performing the slow segmentation of the image into objects. A pixel heat map is generated from the target image by assigning pixels the color corresponding to the object characteristic that the pixelwise descriptors indicate is most likely associated with each pixel.

Claims

1-56. -56. (canceled)

57. A method comprising:

dividing digital images of stained tissue of cancer patients into tiles;

separating the tiles into clusters whose tiles have pixel characteristics that are similar;

identifying a matching cluster of tiles whose pixel characteristics most closely match the pixel characteristics of a target digital image of stained tissue of a target patient;

segmenting the tiles of the matching cluster into objects;

determining object characteristics of the segmented objects;

assigning a color to each of the object characteristics;

generating pixelwise descriptors that describe which of the object characteristics a characterized pixel most probably exhibits based on a quality of a second pixel at a predetermined offset from the characterized pixel;

generating a pixel heat map by applying the pixelwise descriptors to each pixel of the target digital image without segmenting the target digital image into objects, wherein each pixel of the target digital image has the color assigned to the object characteristic most probably exhibited by that pixel; and

displaying the pixel heat map on a graphical user interface.

58. The method of claim 57, wherein the pixelwise descriptors are applied in a random forest decision tree to indicate the probability that each pixel of the target digital image exhibits a particular object characteristic.

59. The method of claim 57, wherein the pixel characteristics of only a predefined portion of each tile are used to identify the matching cluster of tiles.

60. The method of claim 57, wherein the pixel characteristics are taken from the group consisting of:

a mean color vector of all pixels in an intermediate contrast region of a tile, a mean of the mean color vectors of individual tiles of a cluster, a standard deviation of the color vectors of the tiles of a cluster, and an l̂

2 Euclidean norm of the standard deviation of the color vectors of the tiles of a cluster

61. The method of claim 57, wherein the stained tissue has been immunohistochemically stained for the expression of a protein.

62. The method of claim 57, wherein objects that exhibit a particular object characteristic belong to a class, and wherein the object characteristics are taken from the group consisting of:

an average number of concavities of the objects of the class, an average size of the objects of the class, a variation in sizes of the objects of the class, an amount of intensity variation within the objects of the class, an elliptical fit of the objects of the class, and an average intensity of the objects of the class.

63. The method of claim 57, wherein a plurality of colors are assigned to each of the object characteristics depending on the degree of the object characteristic, and wherein each pixel of the target digital image has the color assigned to the degree to which that pixel exhibits the object characteristic.

64. The method of claim 57, wherein the quality of the second pixel at the predetermined offset from the characterized pixel is taken from the group consisting of:

a color value of the second pixel, a difference between a color value of the second pixel and a color value of the characterized pixel, and a difference between an intensity value of the second pixel and an intensity value of the characterized pixel.

65. The method of claim 57, wherein the segmenting the tiles of the matching cluster into objects involves generating a data network in which objects are linked to selected pixels of the tiles.

66. The method of claim 57, wherein the separating the tiles into clusters having similar pixel characteristics involves successively separating the tiles into clusters until each cluster exhibits a standard deviation of color whose magnitude falls below a predetermined threshold.

67. A method comprising:

dividing digital images of stained tissue of cancer patients into tiles;

separating the tiles into clusters whose tiles have similar pixel characteristics;

identifying a matching cluster of tiles whose pixel characteristics most closely match the pixel characteristics of target tiles of a digital image of stained tissue of a target patient;

segmenting the target tiles and the tiles of the matching cluster into objects;

determining object characteristics of the segmented objects;

assigning a color to each of the object characteristics;

generating pixelwise descriptors that describe which of the object characteristics a characterized pixel most probably exhibits based on a quality of a second pixel at a predetermined offset from the characterized pixel;

generating a pixel heat map by applying the pixelwise descriptors to each pixel of the target digital image without segmenting the target digital image into objects, wherein each pixel of the target digital image has the color assigned to the object characteristic most probably exhibited by that pixel; and

displaying the pixel heat map on a graphical user interface.

68. The method of claim 67, wherein the pixelwise descriptors are generated using a random forest of decision trees applied to random different pixels of the target tiles and the tiles of the matching cluster.

69. The method of claim 67, wherein the stained tissue has been stained by hematoxylin and eosin (H&

E).

70. The method of claim 67, wherein objects that exhibit a particular object characteristic belong to a class, and wherein the object characteristics are taken from the group consisting of:

an average number of concavities of the objects of the class, an average size of the objects of the class, a variation in sizes of the objects of the class, an amount of intensity variation within the objects of the class, an elliptical fit of the objects of the class, and an average intensity of the objects of the class.

71. The method of claim 67, wherein the quality of the second pixel at the predetermined offset from the characterized pixel is taken from the group consisting of:

a color value of the second pixel, a difference between a color value of the second pixel and a color value of the characterized pixel, and a difference between an intensity value of the second pixel and an intensity value of the characterized pixel.

72. A method comprising:

dividing images of tissue of cancer patients into tiles;

separating the tiles into clusters, wherein each of the clusters has tiles with similar colors;

identifying a matching cluster whose tiles have colors that most closely match the colors of a target image of tissue of a target patient;

segmenting the tiles of the matching cluster into objects;

classifying the objects into classes;

assigning a color to each object class;

generating pixelwise descriptors that associate an object class with each pixel of the tiles of the matching cluster based on colors of other pixels at predetermined offsets from the classified pixel; and

generating a pixel heat map by applying the pixelwise descriptors to each pixel of the target image without segmenting the target image into objects, wherein each pixel of the target image has the color assigned to the object class associated with that pixel.

73. The method of claim 72, wherein the segmenting the tiles of the matching cluster into objects involves generating a data network in which objects are linked to selected pixels of the tiles.

74. The method of claim 72, wherein the separating the tiles into clusters having similar colors involves successively separating the tiles into clusters until each cluster exhibits a standard deviation of color whose magnitude falls below a predetermined threshold.

75. The method of claim 72, wherein only learning portions of the tiles of the matching cluster are segmented.

76. The method of claim 72, wherein the pixel characteristics used to identify the matching cluster of tiles are determined only from regions on the tiles that exhibit an intermediate degree of local contrast.