Automated microscopic image acquisition, compositing, and display

• US 7,027,628 B1
• Filed: 11/13/2001
• Issued: 04/11/2006
• Est. Priority Date: 11/14/2000
• Status: Expired due to Term

• Current Assignee: The United States of America As Represented By The Secretary of Department of Health And Human Services, Center For Disease Control And Prevention, United States Department Of Health And Human Services
• Original Assignee: United States Department Of Health And Human Services
• Inventors: Lee, Tommy, Taylor, Roger, Gagnon, MariBeth, Kujawski, Edward, Lange, James V., Collins, Carlyn, Draut, Richard
• Primary Examiner(s): Choobin, Barry
• Application Number:
  US10/001,268
• Time in Patent Office: 1,610 Days
• Field of Search: 382/128, 382/134, 382/133, 382/166, 382/232, 382/260, 382/284, 382/294, 382/305, 382/293, 382/129, 382/233, 128/922, 703/11, 359/385, 356/417, 356/39, 377/10
• US Class Current: 382/128
• CPC Class Codes: G01N 15/1433 using image recognition
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10056 Microscopic image
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 7/33 using feature-based methods

Abstract

An automated microscope and computer system captures a set of images for a capture area in a plurality of focal planes. The images can then be integrated into composite images for browsing to simulate viewing an item, such as a biological sample, under a microscope. A corrective filter can be constructed from the images to avoid an effect called “tiling.” Before capture, variable focal plane error can be avoided by collecting z locations for a set of points in the capture area. During image browsing, entire composite images can be loaded into memory in compressed form. Compressed image portions can be pre-decompressed to avoid delay as a browsing user navigates throughout the composite images. Pre-decompression can be done by a thread separate from the thread performing navigation operations.

Claims

1. A computer-implemented method for constructing a composite image representing an item viewed under a microscope at a plurality of focal planes, the method comprising:

with a camera viewing the item under the microscope, capturing a plurality of images from a capture area for the plurality of focal planes;

for the plurality of focal planes, within a set of images in a same focal plane, identifying locations at which adjacent image in the same focal plane can be joined;

for the plurality of focal planes, joining the adjacent images in the same focal plane at the identified locations into a composite image for the focal plane;

storing the plurality of composite images for the plurality of focal planes in a format retrievable by image browsing software;

combining pixel characteristics for the plurality of images into a corrective filter; and

applying the corrective filter to the plurality of images.

2. A computer-implemented method for constructing a composite image representing an item viewed under a microscope at a plurality of focal planes, the method comprising;

with a camera viewing the item under the microscope, capturing a plurality of images from a capture area for the plurality of focal planes;

for the plurality of focal planes, within a set of images is a same focal plane, identifying locations at which adjacent images in the same focal plane can be joined;

for the plurality of focal planes, joining the adjacent images in the same focal plane at the identified locations into a composite image for the focal plane;

combing pixel characteristics for the plurality of images into a corrective filter; and

with the corrective filter, removing color shift from the plurality of images.

3. The method of claim 2 wherein the corrective filter is an image of a same size as the plurality of images.

4. The method of claim 3 wherein the corrective filter is a two-dimensional array of red, green, and blue correction values.

5. A computer-implemented method for constructing a composite image representing an item viewed under a microscope at a plurality of focal planes, the method comprising;

with a camera viewing the item under the microscope, capturing a plurality of images from a capture area for the plurality of focal planes;

for the plurality of focal planes, within a set of images in a same focal plane, identifying locations at which adjacent images in the same focal plane can be joined;

for the plurality of focal planes, joining the adjacent images in the same focal plane at the identified locations into a composite image for the focal plane;

collecting z location readings for a plurality of locations in the capture area indicating a uniform z location with respect to the item at each location; and

during capturing, automatically adjusting z location of the microscope according to the z location.

6. The method of claim 5 wherein automatically adjusting z location comprises:

finding at least three near readings to a current location of the microscope; and

interpolating a proper z location of the microscope by finding a point relating to the current location on a plane defined by the at least three near readings.

7. The method of claim 5 wherein

the z location readings are collected for a single focal plane;

and

during capturing, z location of the microscope is adjusted for a plurality of focal planes according to the z location readings.

8. A computer-implemented method for constructing a composite image representing an item viewed under a microscope at a plurality of focal planes, the method comprising:

with a camera viewing the item under the microscope, capturing a plurality of images from a capture area for the plurality of focal planes;

for the plurality of focal planes, within a set of images in a same focal plane, identifying locations at which adjacent images in the same focal plane can be joined;

for the plurality of focal planes, joining the adjacent images in the same focal plane at the identified locations into a composite image for the focal plane;

browsing the composite image representing an item viewed under a microscope, wherein the composite image comprises a set of composite image portions, the browsing comprising;

decompressing the composite image portions of the composite image for a browsing area currently being browsed at a computer;

displaying at least portions of the decompressed images; and

decompressing composite image portions of the composite image that are outside the browsing area currently being browsed and likely to be next displayed during navigation of the composite image.

9. The method of claim 8 further comprising:

loading into memory at least one complete composite image for a focal plane in compressed form.

10. The method of claim 8 further comprising:

loading into memory complete composite images for plural focal planes in compressed form.

11. The method of claim 8 wherein the composite image portions likely to be next displayed are composite image portions adjacent to and in a same focal plane as composite image portions currently being displayed.

12. The method of claim 8 wherein the composite image portions likely to be next displayed are composite image portions adjacent to composite image portions currently being displayed and in a different focal plane.

13. The method of claim 8 wherein

displaying is performed by a first thread;

and

decompressing images likely to be next displayed is performed by a second thread.

14. The method of claim 8 further comprising:

in preparation for displaying at least portions of the decompressed images, loading the decompressed images directly into video memory;

tracking which images are loaded into video memory;

tracking when images loaded into video memory were last used; and

keeping images in video memory until display of an image requires discarding an image, wherein the least recently used image is discarded.

15. A computer-readable medium comprising computer-executable instructions for performing a method for constructing a composite image representing an item viewed under a microscope at a plurality of focal planes, the method comprising:

with a camera viewing the item under the microscope, capturing a plurality of images from a capture area for the plurality of focal planes;

for the plurality of focal planes, within a set of images in a same focal plane, identifying locations at which adjacent images in the same focal plane can be joined;

for the plurality of focal planes, joining the adjacent images in the same focal plane, at the identified locations into a composite image for the focal plane;

browsing the composite image representing an item viewed under a microscope, wherein the composite image comprises a set of composite image portions, the browsing comprising;

decompressing the composite image portions of the composite image for a browsing area currently being browsed at a computer;

displaying at least portions of the decompressed images; and

decompressing composite image portions of the composite image that are outside the browsing area currently being browsed and likely to be next displayed during navigation of the composite image.

16. A system for constructing a composite image representing an item viewed under an automated microscope at a plurality of focal planes, the system comprising:

an automated microscope with a computer-controllable stage and focus controls;

a computer system interfaced to the automated microscope;

a camera positioned on the microscope and to send a photographic representation of a microscopic view of the item to the computer system for capture;

software running on the computer system and to navigate the microscope to a plurality of locations at a plurality of focal planes, and capture images via the camera;

software to integrate the images into composite images for a plurality of focal planes; and

a browsing system for displaying the composite images comprising a set of compressed images forming portions of the composite images, the browsing system comprising;

variables for training the current browsing area;

a composite image manager for providing access to the portions of the composite images;

a decompression cache manager for tracking pre-decompressed images;

a decompressed image store for storing decompressed versions of the portions of the composite images; and

a video memory cache manager for tracking which of the decompressed versions of the portions of the composite images are currently loaded in video memory.

17. The system of claim 16 wherein the composite image manager maintains a store of composite images for a plurality of focal planes.

18. The system of claim 16 wherein the composite image manager maintains in memory a store of at least one complete composite image in compressed form.

19. The system of claim 16 wherein the composite image manager maintains in memory a store of a plurality of complete composite images for a plurality of focal planes in compressed form.

20. The system of claim 16 wherein the decompression cache manager prioritizes decompression according to the following, ordered priority:

image portions for the curt browsing area;

image portions in a same focal plane and adjacent to the current browsing area; and

image portions in a different focal plane and adjacent to the current browsing area.

21. A system for constructing a composite image representing an item viewed under an automated microscope at a plurality of focal planes, the system comprising:

an automated microscope with a computer-controllable stage and focus controls;

a computer system interfaced to the automated microscope;

means positioned on the microscope and operable to send a photographic representation of a microscopic view of the item to the computer system for capture;

means for navigating the microscope to a plurality of locations at a plurality of focal planes, and capture images via the camera;

means for integrating the images into composite images for a plurality of focal planes;

a browsing system for displaying the composite images comprising a set of compressed images forming portions of the composite images, the browsing system comprising;

means for tracking the current browsing area;

means for providing access to the portions of the composite images;

means for tracking pre-decompressed images;

means for storing decompressed versions of the portions of the composite images; and

means for tracking which of the decompressed versions of the portions of the composite images are currently loaded in video memory.