Learning pixel visual context from object characteristics to generate rich semantic images

• US 10,445,557 B2
• Filed: 08/03/2017
• Issued: 10/15/2019
• Est. Priority Date: 08/29/2014
• Status: Active Grant

• Current Assignee: Definiens AG (Astrazeneca PLC)
• Original Assignee: Definiens AG (Astrazeneca PLC)
• Inventors: Pauly, Olivier, Brieu, Nicolas, Schmidt, Guenter, Zimmermann, Johannes, Binnig, Gerd
• Primary Examiner(s): Bella, Matthew C
• Assistant Examiner(s): Rosario, Dennis
• Application Number:
  US15/668,695
• Publication Number:
  US 20170337415A1
• Time in Patent Office: 803 Days
• CPC Class Codes: G06F 18/24323 Tree-organised classifiers
• CPC Class Codes: G06T 7/0012 Biomedical image inspection
• CPC Class Codes: G06T 7/162 involving graph-based methods
• CPC Class Codes: G06V 10/25 Determination of region of interest [ROI] or a volume of interest [VOI]
• CPC Class Codes: G06V 10/454 Integrating the filters into a hierarchical structure, e.g. convolutional neural networks [CNN]
• CPC Class Codes: G06V 10/462 Salient features, e.g. scale invariant feature transforms [SIFT]
• CPC Class Codes: G06V 10/50 by performing operations within image blocks; by using histograms, e.g. histogram of oriented gradients [HoG]; by summing image-intensity values; Projection analysis
• CPC Class Codes: G06V 10/56 relating to colour
• CPC Class Codes: G06V 10/764 using classification, e.g. of video objects
• CPC Class Codes: G06V 10/772 Determining representative reference patterns, e.g. averaging or distorting patterns; Generating dictionaries
• CPC Class Codes: G06V 20/695 Preprocessing, e.g. image segmentation
• CPC Class Codes: G06V 20/698 Matching; Classification

Abstract

Both object-oriented analysis and the faster pixel-oriented analysis are used to recognize patterns in an image of stained tissue. Object-oriented image analysis is used to segment a small portion of the image into object classes. Then the object class to which each pixel in the remainder of the image most probably belongs is determined using decision trees with pixelwise descriptors. The pixels in the remaining image are assigned object classes without segmenting the remainder of the image into objects. After the small portion is segmented into object classes, characteristics of object classes are determined. The pixelwise descriptors describe which pixels are associated with particular object classes by matching the characteristics of object classes to the comparison between pixels at predetermined offsets. A pixel heat map is generated by giving each pixel the color assigned to the object class that the pixelwise descriptors indicate is most probably associated with that pixel.

Claims

1. A method comprising:

segmenting a first portion of an image into objects of a data network that belong to classes, wherein the segmenting the first portion of the image into objects involves generating the data network in which the objects are linked to selected pixels of the first portion of the image, wherein at least one of the objects is linked to a plurality of subobjects, and wherein each of the subobjects is linked to other subobjects or to pixels of the image;

determining class characteristics of each of the classes of objects;

assigning a color to each of the classes of objects;

generating pixelwise descriptors used to describe which pixels are associated with particular classes of objects;

generating a pixel heat map by applying the pixelwise descriptors to each pixel in a second portion of the image without segmenting the second portion of the image into objects of any data network, wherein each pixel of the second portion of the image has the color assigned to the class of objects that the pixelwise descriptors indicate is most probably associated with that pixel; and

displaying the pixel heat map on a graphical user interface.

2. The method of claim 1, wherein a first class characteristic of a first class of objects is an elliptical fit of the objects of the first class, and wherein the pixelwise descriptors indicate that a characterized pixel is most probably associated with the objects of the first class that exhibit a particular elliptical fit without ever segmenting the second portion of the image into objects whose elliptical fit can be determined.

3. The method of claim 2, wherein no pixels of the second portion of the image are linked to objects of any data network.

4. The method of claim 1, wherein the image depicts a tissue slice that has been immunohistochemically stained for the expression of a protein.

5. The method of claim 1, wherein one of the classes of objects represents nuclei that have been stained by hematoxylin and eosin (H&

E).

6. The method of claim 1, wherein the characteristics of a class of objects are taken from the group consisting of:

an average number of concavities of the objects of the class, an average size of the objects of the class, a variation in sizes of the objects of the class, an amount of intensity variation within the objects of the class, an elliptical fit of the objects of the class, and an average intensity of the objects of the class.

7. The method of claim 1, wherein the pixelwise descriptors for a characterized pixel are taken from the group consisting of:

a color value of a second pixel at a predetermined offset, a difference between a color value of the characterized pixel and of a second pixel at a predetermined offset, an average of the color values of pixels in a box of predetermined size at a predetermined offset from the characterized pixel, a standard deviation among the color values of pixels in a box of predetermined size at a predetermined offset from the characterized pixel, a sum of gradient magnitude of the color values of pixels in a box of predetermined size at a predetermined offset from the characterized pixel, and an orientation of a gradient edge at a predetermined offset.

8. The method of claim 1, wherein the pixelwise descriptors are used in a random forest decision tree to indicate the probability that a characterized pixel whose surrounding pixels are defined by the pixelwise descriptors belongs to a particular class of objects.

9. The method of claim 1, wherein each of the pixelwise descriptors indicates a probability that a characterized pixel belongs to a particular class of objects based on a characteristic of a second pixel at a predetermined offset from the characterized pixel.

10. The method of claim 1, wherein the color assigned to each class of objects is a shade of gray.

11. The method of claim 1, wherein one of the class characteristics of one of the classes of objects defines a characteristic shape of the objects of the class.

12. A method comprising:

segmenting a first portion of an image into network objects of a data network that belong to classes, wherein the image depicts a slice of tissue from a cancer patient that has been stained, wherein the segmenting involves generating the data network in which the network objects are linked to selected pixels of the image, wherein at least one of the network objects is linked to a plurality of network subobjects, and wherein each of the network subobjects is linked to other network subobjects or to pixels of the image;

determining class characteristics of each of the classes of network objects;

assigning a color to each of the classes of network objects;

generating pixelwise descriptors used to describe which pixels are associated with particular classes of network objects; and

generating a pixel heat map by applying the pixelwise descriptors to each pixel in a second portion of the image without segmenting the second portion of the image into network objects of any data network, wherein each pixel of the second portion of the image has the color assigned to the class of network objects that the pixelwise descriptors indicate is most probably associated with that pixel.

13. The method of claim 12, wherein the tissue has been immunohistochemically stained for the expression of a protein.

14. The method of claim 12, wherein the tissue has been stained by hematoxylin and eosin (H&

E).

15. The method of claim 12, wherein each of the pixelwise descriptors indicates a probability that a characterized pixel belongs to a particular class of network objects based on a characteristic of a second pixel at a predetermined offset from the characterized pixel.

16. A method comprising:

segmenting a first portion of an image into objects of a data network using object-oriented image analysis, wherein a second portion of the image is not segmented into objects of the data network, wherein the objects of the data network are linked to subobjects of the data network and to selected pixels of the first portion of the image;

defining classes of objects, wherein each of the objects belongs to one of the classes of objects;

determining class characteristics of each of the classes of objects, wherein one of the class characteristics of a first class of objects defines a characteristic shape of the objects of the first class;

assigning a color to each of the classes of objects;

generating pixelwise descriptors used to describe which pixels are associated with particular class characteristics of classes of objects;

performing pixel-oriented image analysis by applying the pixelwise descriptors individually to each pixel in the second portion of the image, wherein the pixelwise descriptors associate a first pixel in the second portion of the image with the characteristic shape of the objects of the first class;

generating a pixel heat map in which each pixel of the second portion of the image has the color assigned to the class of objects that the pixelwise descriptors indicate is most probably associated with that pixel; and

displaying the pixel heat map on a graphical user interface.

17. The method of claim 16, wherein the class characteristics of the first class of objects are taken from the group consisting of:

an elliptical fit of the objects of the class, an average number of concavities of the objects of the class, an average size of the objects of the class, and a variation in sizes of the objects of the class.

18. The method of claim 16, wherein the image depicts a tissue slice that has been immunohistochemically stained for the expression of a protein.

19. The method of claim 16, wherein one of the classes of objects represents nuclei that have been stained by hematoxylin and eosin (H&

E).

20. The method of claim 16, wherein each of the pixelwise descriptors indicates a probability that a characterized pixel belongs to a particular class of objects based on a characteristic of a second pixel at a predetermined offset from the characterized pixel.