Learning Pixel Visual Context from Object Characteristics to Generate Rich Semantic Images

• US 20160063308A1
• Filed: 08/29/2014
• Published: 03/03/2016
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: Definiens AG (Astrazeneca PLC)
• Original Assignee: Definiens AG (Astrazeneca PLC)
• Inventors: Brieu, Nicolas, Schmidt, Guenter, Binnig, Gerd, Pauly, Olivier, Zimmermann, Johannes
• Granted Patent: US 9,740,957 B2
• Time in Patent Office: Days
• US Class Current: 1/1
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 20/695 Preprocessing, e.g. image segmentation
• CPC Class Codes: G06V 20/698 Matching; Classification

Abstract

Both object-oriented analysis and the faster pixel-oriented analysis are used to recognize patterns in an image of stained tissue. Object-oriented image analysis is used to segment a small portion of the image into object classes. Then the object class to which each pixel in the remainder of the image most probably belongs is determined using decision trees with pixelwise descriptors. The pixels in the remaining image are assigned object classes without segmenting the remainder of the image into objects. After the small portion is segmented into object classes, characteristics of object classes are determined. The pixelwise descriptors describe which pixels are associated with particular object classes by matching the characteristics of object classes to the comparison between pixels at predetermined offsets. A pixel heat map is generated by giving each pixel the color assigned to the object class that the pixelwise descriptors indicate is most probably associated with that pixel.

Claims

1. A method comprising:

dividing a digital image into tiles;

determining a degree of local contrast in each of the tiles;

selecting a first plurality of the tiles that exhibits a greatest degree of local contrast;

determining an average color of each of the first plurality of tiles;

dividing the first plurality of tiles into clusters of tiles with similar colors;

selecting a learning tile from each cluster of tiles, wherein each learning tile has the greatest degree of local contrast from among the tiles of the cluster to which the learning tile belongs;

segmenting the learning tiles into objects;

classifying the objects into classes of objects;

associating a color with each class of objects;

determining characteristics of the objects that belong to distinct classes of objects;

generating pixelwise descriptors that indicate the class of objects to which each pixel most probably belongs;

generating a pixel heat map without again segmenting the digital image into objects by applying the pixelwise descriptors to each pixel of the digital image, wherein each pixel of the digital image is assigned the color associated with the class of objects to which that pixel most probably belongs; and

displaying the pixel heat map on a graphical user interface.

2. The method of claim 1, wherein the pixelwise descriptors are applied in a random forest decision tree to indicate the probability that each pixel of each learning tile belongs to a class of objects.

3. The method of claim 1, wherein the pixelwise descriptors indicate the probability that a characterized pixel belongs to a class of objects based on a characteristic of a second pixel at a predetermined offset from the characterized pixel.

4. The method of claim 1, wherein the determining the degree of local contrast in each tile is performed by determining a proportion of area of each tile that has maximally stable extremal regions.

5. The method of claim 1, wherein the segmenting the learning tiles into objects involves generating a data network in which objects are linked to selected pixels of the learning tiles.

6. The method of claim 1, wherein at least one of the objects of each learning tile is linked to a plurality of subobjects, and wherein each of the subobjects is linked to other subobjects or to pixels of the learning tile.

7. The method of claim 1, wherein the color associated with each class of objects is a shade of gray.

8. A method comprising:

segmenting a first portion of an image into objects that belong to classes;

determining class characteristics of each of the classes of objects;

assigning a color to each of the classes of objects;

generating pixelwise descriptors used to describe which pixels are associated with particular classes of objects;

generating a pixel heat map by applying the pixelwise descriptors to each pixel in a second portion of the image without segmenting the second portion of the image into objects, wherein each pixel of the second portion of the image has the color assigned to the class of objects that the pixelwise descriptors indicate is most probably associated with that pixel; and

displaying the pixel heat map on a graphical user interface.

9. The method of claim 8, wherein the segmenting the first portion of the image into objects involves generating a data network in which objects are linked to selected pixels of the first portion of the image.

10. The method of claim 9, wherein no pixels of the second portion of the image are linked to objects.

11. The method of claim 8, wherein the image depicts a tissue slice that has been immunohistochemically stained for the expression of a protein.

12. The method of claim 8, wherein one of the classes of objects represents nuclei that have been stained by hematoxylin and eosin (H&

E).

13. The method of claim 8, wherein the characteristics of a class of objects are taken from the group consisting of:

an average number of concavities of the objects of the class, an average size of the objects of the class, a variation in sizes of the objects of the class, an amount of intensity variation within the objects of the class, an elliptical fit of the objects of the class, and an average intensity of the objects of the class.

14. The method of claim 8, wherein the pixelwise descriptors for a characterized pixel are taken from the group consisting of:

a color value of a second pixel at a predetermined offset, a difference between a color value of the characterized pixel and of a second pixel at a predetermined offset, an average of the color values of pixels in a box of predetermined size at a predetermined offset from the characterized pixel, a standard deviation among the color values of pixels in a box of predetermined size at a predetermined offset from the characterized pixel, a sum of gradient magnitude of the color values of pixels in a box of predetermined size at a predetermined offset from the characterized pixel, and an orientation of a gradient edge at a predetermined offset.

15. The method of claim 8, wherein the pixelwise descriptors are used in a random forest decision tree to indicate the probability that a characterized pixel whose surrounding pixels are defined by the pixelwise descriptors belongs to a particular class of objects.

16. The method of claim 8, wherein each of the pixelwise descriptors indicates a probability that a characterized pixel belongs to a particular class of objects based on a characteristic of a second pixel at a predetermined offset from the characterized pixel.

17. The method of claim 8, wherein the color assigned to each class of objects is a shade of gray.

18. A method comprising:

segmenting a first portion of an image into objects;

determining object characteristics of the segmented objects;

assigning a color to each of the object characteristics;

generating pixelwise descriptors that describe which of the object characteristics a characterized pixel most probably exhibits based on a quality of a second pixel at a predetermined offset from the characterized pixel;

generating a pixel heat map by applying the pixelwise descriptors to each pixel in a second portion of the image without segmenting the second portion of the image into objects, wherein each pixel of the second portion of the image has the color assigned to the object characteristic most probably exhibited by that pixel; and

displaying the pixel heat map on a graphical user interface.

19. The method of claim 18, wherein a plurality of colors are assigned to each of the object characteristics depending on the degree of the object characteristic, and wherein each pixel of the second portion of the image has the color assigned to the degree to which that pixel exhibits the object characteristic.

20. The method of claim 18, wherein the quality of the second pixel at the predetermined offset from the characterized pixel is taken from the group consisting of:

a color value of the second pixel, a difference between a color value of the second pixel and a color value of the characterized pixel, and a difference between an intensity value of the second pixel and an intensity value of the characterized pixel.

21. The method of claim 18, wherein the object characteristics are taken from the group consisting of:

a number of concavities of the object, a size of the object, an elliptical fit of the object, and an average intensity of pixels in the object.