Multi-modality contrast and brightfield context rendering for enhanced pathology determination and multi-analyte detection in tissue

• US 9,310,302 B2
• Filed: 10/07/2010
• Issued: 04/12/2016
• Est. Priority Date: 10/12/2009
• Status: Active Grant

• Current Assignee: Ventana Medical Systems Incorporated (Roche Holding AG)
• Original Assignee: Ventana Medical Systems Incorporated (Roche Holding AG)
• Inventors: Garsha, Karl, Pestano, Gary, Otter, Michael, Nagy, Alexandra Dea, Nagle, Ray B., Miller, Phillip, Froehlich, Jan, Day, William
• Primary Examiner(s): Koziol, Stephen R
• Assistant Examiner(s): Gillard, Delomia L
• Application Number:
  US13/499,959
• Publication Number:
  US 20120200694A1
• Time in Patent Office: 2,014 Days
• Field of Search: 382/128
• US Class Current: 1/1
• CPC Class Codes: G01N 21/6456 Spatial resolved fluorescence measurements; Imaging

Abstract

Multiple modality contrast can be used to produce images that can be combined and rendered to produce images similar to those produced with wavelength absorbing stains viewed under transmitted white light illumination. Images obtained with other complementary contrast modalities can be presented using engineered color schemes based on classical contrast methods used to reveal the same anatomical structures and histochemistry, thereby providing relevance to medical training and experience. Dark-field contrast images derived from refractive index and fluorescent DAPI counterstain images are combined to produce images similar to those obtained with conventional H&E staining for pathology interpretation. Such multi-modal image data can be streamed for live navigation of histological samples, and can be combined with molecular localizations of genetic DNA probes (FISH), sites of mRNA expression (mRNA-ISH), and immunohistochemical (IHC) probes localized on the same tissue sections, used to evaluate and map tissue sections prepared for imaging mass spectrometry.

Claims

1. An image generation method, comprising:

receiving a fluorescent image of a specimen, wherein the specimen is fluorescently stained and a first beam has been selected to produce fluorescence by the fluorescent stain so that the first image is a fluorescence image of the specimen;

receiving a refractive dark field image of the specimen, wherein a second stimulus beam has been applied to the specimen so that the second image is a refractive dark field image;

applying a color mapping to the refractive dark field image to produce a pseudo-color dark field image;

applying a color lookup table to the fluorescence image, and generating a converted fluorescent image wherein the color lookup table is associated with at least one absorptive stain;

combining the pseudo-color dark field image and the converted fluorescence image and generating a refractive dark field and fluorescence combined image; and

inverting the refractive dark field and fluorescence combined image to produce a brightfield rendered image.

2. The method of claim 1, further comprising recording the fluorescence image and the refractive dark field image as corresponding recorded images.

3. The method of claim 1, wherein the absorptive stain is an eosin stain.

4. The method of claim 1, wherein the fluorescence is based on DAPI fluorescence, and the color lookup table associated with the fluorescence image is based on a hematoxylin stain.

5. The method of claim 1, further comprising producing a pseudo-brightfield recorded image based on the refractive dark field and fluorescence combined recorded image.

6. The method of claim 5, further comprising applying color lookup tables to the refractive dark field image and the fluorescence image so as to produce an image having image contrast associated with hematoxylin and eosin staining.

7. The method of claim 1, further comprising generating a mass spectroscopic image of the specimen.

8. The imaging apparatus, comprising:

at least one image capture device that receives first and second images, wherein the first image is a refractive dark field image and the second image is a fluorescence image; and

an image processor coupled to the image capture device that applies a color lookup table to at least one of the first and second recorded images and generates a pseudo-colored rendering of at least one of the first and second images,

wherein the image processor combines the pseudo-colored rendering of at least one of the first and second images with the other of the at least one of the first and second images and generates a refractive dark field and fluorescence combined image, based on the pseudo-colored rendering of at least one of the first and second images,

wherein when image processer generates a pseudo-colored rendering of the first image, the image processor processes the first image based on a color lookup table associated with an eosin stain, and when the image processor generates a pseudo-colored rendering of the second image, the image processor processes the second image based on color lookup table associated with a hematoxylin stain, and

wherein the image processor inverts the refractive dark field and fluorescence combined image to produce a brightfield rendered image.

9. The imaging apparatus of claim 8, further comprising rendering the refractive dark field and fluorescence combined image as a pseudo-brightfield image.

10. The imaging apparatus of claim 8, wherein the image capture device is configured to receive the first image and the second image as side by side images.

11. The imaging apparatus of claim 8, wherein the image processor is configured to overlay the first and second images to produce the refractive dark field and fluorescence combined image.

12. The imaging apparatus of claim 8, further comprising a display configured to receive and display the refractive dark field and fluorescence combined image.

13. At least one non-transitory computer readable storage media comprising computer-executable instructions for:

receiving a first image and a second image associated with a common portion of a specimen section, wherein the first image is a refractive dark field image and the second image is a fluorescence image;

combining the refractive dark field image and the fluorescence image and generating a refractive dark field and fluorescence combined image, wherein before the refractive dark field image is combined with the fluorescence image, the refractive dark field image is processed based on a color lookup table associated with an eosin stain and the fluorescence image is processed based on color lookup table associated with a hematoxylin stain, and wherein the refractive dark field and fluorescence combined image is based on the processed first and second images;

inverting the refractive dark field and fluorescence combined image, and generating a pseudo-color brightfield hematoxylin and eosin image based on the processed first image and the processed second image.

14. The at least one computer readable storage media of claim 13, further comprising computer executable instructions for receiving a mass spectroscopic image of the common specimen section, and including the mass spectroscopic image in the refractive dark field and fluorescence combined image.

15. An image processor, comprising:

image inputs configured that receives a first image and a second image, wherein the first image is a refractive dark field image and the second image is a fluorescent image;

a color lookup table input that receives, a first color lookup table associated with an eosin stain and a second color lookup table associated with a hematoxylin stain;

an image combiner that processes the refractive dark field image based on the first color lookup table associated with an eosin stain and that processes the fluorescent image based on the second color lookup table associated with the eosin stain and produces a pseudo-color refractive dark field image and a pseudo-color fluorescent image, and combines the pseudo-color refractive dark field image with the pseudo-color fluorescent image and generates a combined refractive dark field and fluorescent image, and produces a brightfield image rendering based on the combined refractive dark field and fluorescent image after inverting the combined refractive dark field and fluorescent image.