Generating image-based diagnostic tests by optimizing image analysis and data mining of co-registered images

• US 9,159,129 B2
• Filed: 03/04/2014
• Issued: 10/13/2015
• Est. Priority Date: 07/12/2011
• Status: Active Grant

• Current Assignee: Definiens AG (Astrazeneca PLC)
• Original Assignee: Definiens AG (Astrazeneca PLC)
• Inventors: Schoenmeyer, Ralf, Binnig, Gerd, Schmidt, Guenter
• Primary Examiner(s): Abdi, Amara
• Application Number:
  US14/197,197
• Publication Number:
  US 20140185891A1
• Time in Patent Office: 588 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06T 11/206 Drawing of charts or graphs
• CPC Class Codes: G06T 2207/10061 from scanning electron microscope
• CPC Class Codes: G06T 2207/20016 Hierarchical, coarse-to-fine, multiscale or multiresolution image processing; Pyramid transform
• CPC Class Codes: G06T 2207/20021 Dividing image into blocks, subimages or windows
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 7/0012 Biomedical image inspection

Abstract

A method for generating an image-based test improves diagnostic accuracy by iteratively modifying rule sets governing image and data analysis of coregistered image tiles. Digital images of stained tissue slices are divided into tiles, and tiles from different images are coregistered. First image objects are linked to selected pixels of the tiles. First numerical data is generated by measuring the first objects. Each pixel of a heat map aggregates first numerical data from coregistered tiles. Second objects are linked to selected pixels of the heat map. Measuring the second objects generates second numerical data. The method improves how well second numerical data correlates with clinical data of the patient whose tissue is analyzed by modifying the rule sets used to generate the first and second objects and the first and second numerical data. The test is defined by those rule sets that produce the best correlation with the clinical data.

Claims

1. A method comprising:

(a) dividing each digital image of a plurality of slices of tissue of a patient into tiles, wherein the tiles obtained from digital images of different slices of tissue are coregistered with each other to form a stack of tiles;

(b) generating first objects that are linked to pixels of the tiles, wherein a first rule set defines which pixels are linked to each of the first objects;

(c) using a second rule set to generate first numerical data by measuring a first characteristic of the first objects located in each of the tiles;

(d) generating a heat map by aggregating the first numerical data associated with each stack of tiles, wherein each stack of tiles is used to generate a pixel of the heat map;

(e) generating second objects from the heat map, wherein a third rule set defines which pixels of the heat map are linked to each of the second objects;

(f) using a fourth rule set to generate second numerical data by measuring a second characteristic of the second objects detected in the heat map;

(g) determining how well the second numerical data correlates with clinical data for the patient;

(h) improving how well the second numerical data correlates with the clinical data for the patient by modifying the first rule set, the second rule set, the third rule set and the fourth rule set and then repeating (b) through (g); and

(i) defining an image-based diagnostic test based on the first rule set, the second rule set, the third rule set and the fourth rule set.

2. The method of claim 1, further comprising:

(j) displaying the heat map and the second objects on a graphical user interface.

3. The method of claim 1, wherein the first rule set, the second rule set, the third rule set and the fourth rule are modified in (h) in a manner that reduces computational resources required to repeat (b) through (g).

4. The method of claim 1, wherein the image-based diagnostic test predicts a probability that the patient will remain disease free for a predetermined period of time after a clinical action on the patient.

5. The method of claim 1, wherein one of the slices of tissue is stained with H&

E and another of the slices of tissue is immunohistochemically stained for the expression of a protein.

6. The method of claim 1, wherein the second numerical data includes a mean pixel intensity value of a group of second objects detected in the heat map.

7. The method of claim 1, wherein the second numerical data correlates well with the clinical data for the patient when a survival time of the patient predicted by the image-based diagnostic test equals an actual survival time of the patient.

8. The method of claim 1, wherein the image-based diagnostic test predicts a probability of recurrence of cancer.

9. The method of claim 1, wherein the first characteristic and other characteristics of the first objects are listed in an ontology, and wherein the second rule set is modified in (h) based on a topological distance in the ontology between the first characteristic and another characteristic measured using a modified second rule set.

10. A method comprising:

(a) dividing a first digital image and a second digital image into tiles, wherein the first digital image is of a first slice of tissue of a patient that is stained with a first biomarker, wherein the second digital image is of a second slice of tissue of the patient that is stained with a second biomarker, and wherein the tiles of the first digital image are coregistered with the tiles of the second digital image;

(b) generating first objects that are linked to pixels of the tiles, wherein a first rule set defines which pixels are linked to each of the first objects;

(c) generating first numerical data by measuring the first objects using a second rule set;

(d) generating a heat map by aggregating the first numerical data associated with coregistered tiles, wherein each pixel of the heat map is generated from coregistered tiles;

(e) generating second objects from the heat map, wherein a third rule set defines which pixels of the heat map are linked to each of the second objects;

(f) generating second numerical data by measuring the second objects using a fourth rule set;

(g) improving how well the second numerical data correlates with clinical data for the patient by modifying the fourth rule set and then repeating the generating of the second numerical data in (f); and

(h) defining an image-based diagnostic test based on the first rule set, the second rule set, the third rule set and the fourth rule set.

11. The method of claim 10, further comprising:

(i) displaying the heat map and the second objects on a graphical user interface.

12. The method of claim 10, wherein the first biomarker is H&

E, and the second biomarker is a protein specific antibody.

13. The method of claim 12, wherein the image-based diagnostic test predicts a probability that prostate cancer will recur in the patient, and wherein the protein specific antibody stains basal cells.

14. The method of claim 12, wherein the image-based diagnostic test predicts whether the patient belongs to a short-lived group of patients or to a long-lived group of patients, and wherein the protein specific antibody stains immune cells.

15. The method of claim 12, wherein the image-based diagnostic test predicts a response to an adjuvant cancer therapy, and wherein the protein specific antibody stains membranes of cancer cells.

16. The method of claim 10, wherein characteristics of the first objects are listed in an ontology, wherein the second rule set is modified in (g) using the ontology, wherein the fourth rule set is modified in (g) based on a genetic algorithm, and wherein those modified rule sets are selected to define the image-based diagnostic test that improve how well the second numerical data correlates with the clinical data.

17. The method of claim 10, wherein the second rule set is modified in (g) based on a topological distance between levels of the ontology.

18. A method comprising:

(a) dividing a first digital image and a second digital image into tiles, wherein the first digital image is of a first slice of tissue of a patient that is stained with a first biomarker, wherein the second digital image is of a second slice of tissue of the patient that is stained with a second biomarker, and wherein a first tile of the first digital image is coregistered with a second tile of the second digital image;

(b) generating a data network that includes first tile objects that are linked to selected pixels of the first tile and second tile objects that are linked to pixels of the second tile;

(c) generating a heat map, wherein each pixel of the heatmap has a color dependent on information obtained by analyzing both the first tile objects and the second tile objects;

(d) generating heat map objects that are linked to selected pixels of the heatmap;

(e) using a rule set to generate numerical data by measuring the heat map objects;

(f) improving how well the numerical data correlates with an actual clinical outcome of the patient by modifying the rule set and repeating (e); and

(g) defining an image-based test that generates a predicted clinical outcome using the rule set.

19. The method of claim 18, further comprising:

(h) displaying the heat map and the heat map objects on a graphical user interface.

20. The method of claim 18, wherein the color of each pixel of the heatmap is a shade of gray.

21. The method of claim 18, wherein the information obtained by analyzing both the first tile objects and the second tile objects indicates an average distance between clusters of mitotic objects.

22. The method of claim 18, wherein the information obtained by analyzing the second tile objects indicates a number of mitotic objects stained by the second biomarker.

23. The method of claim 18, wherein the numerical data correlates well with the actual clinical outcome of the patient when a survival time of the patient predicted by the image-based test equals an actual survival time of the patient.

24. The method of claim 18, wherein the rule set generates the numerical data by measuring a first characteristic of the heat map objects, wherein the first characteristic and a second characteristic of the heat map objects are listed in an ontology, and wherein the rule set is modified in (f) based on a topological distance in the ontology between the first characteristic and the second characteristic.

25. The method of claim 18, wherein the second characteristic is measured using the rule set as modified in (f).