Apparatus and method for digital microscopy imaging

• US 9,224,063 B2
• Filed: 08/02/2012
• Issued: 12/29/2015
• Est. Priority Date: 08/02/2011
• Status: Active Grant

• Current Assignee: ViewsIQ Inc.
• Original Assignee: ViewsIQ Inc.
• Inventors: Lo, Hok Man Herman, Lin, Li-Heng, Cecchetto, Benjamin Thomas, Afrasiabi, Mohammadhossein, Lallement, Sebastien
• Primary Examiner(s): Mehta, Bhavesh
• Assistant Examiner(s): Dunphy, David F
• Application Number:
  US14/236,580
• Publication Number:
  US 20140270537A1
• Time in Patent Office: 1,244 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: G06F 18/22 Matching criteria, e.g. proximity measures
• CPC Class Codes: G06T 11/60 Editing figures and text; Combining figures or text
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 3/4038 Image mosaicing, e.g. composing plane images from plane sub-images

Abstract

A microscopy imaging system and method for rendering a mosaic representation of an object from a series of image frames of the object are disclosed. A current image frame is processed to determine its relative location or position within the mosaic representation based on relative displacement of the current image from one or more keyframes. Once the current image frame'"'"'s position has been determined, it is rendered along with its neighboring keyframes to provide the mosaic representation of the object.

Claims

1. A method for rendering a mosaic representation of an object from a series of image frames of the object, the method comprising:

acquiring a current image frame of the object;

determining a current image frame tracking position of the current image frame based on a previous image frame position of a previous image frame and a relative displacement of the current image frame from the previous image frame;

identifying at least one neighboring keyframe within a search area defined relative to the current image frame tracking position, the at least one neighboring keyframe being one of a plurality of keyframes stored in a keyframe database, each keyframe having an associated keyframe position;

determining relative displacements of the current image frame from each of the at least one neighboring keyframes;

determining current image frame positions of the current image frame based on the keyframe positions of each of the at least one neighboring keyframes and the relative displacement of the current image frame from each of the at least one neighboring keyframes;

determining a current image frame position as an average of the current image frame positions;

storing the current image frame as a new keyframe in the keyframe database upon determining that each of the relative displacements of the current image frame from each of the at least one neighboring keyframes is greater than a threshold value;

storing the current image frame position as a new keyframe position associated with the new keyframe in the keyframe database;

rendering the current image frame at its current image frame position and the keyframes from the keyframe database on a display at their respective associated keyframe positions to obtain the mosaic representation of the object.

2. The method of claim 1, wherein storing the new keyframe includes:

storing the relative displacements of the current image frame from each of the at least one neighboring keyframes.

3. The method of claim 2, further comprising:

adjusting the keyframe positions of the plurality of keyframes to correct errors in the keyframe positions based on optimization of each keyframe position to satisfy one or more constraints on the relative displacements of each keyframe with its neighboring keyframes.

4. The method of claim 1, wherein upon failure to identify at least one neighboring keyframe within the search area or upon failure to determine the relative position of the current image frame to any neighboring keyframe, the method further comprising:

iteratively increasing the search area to identify at least one neighboring keyframe.

5. The method of claim 4, wherein upon failure to identify at least one neighboring keyframe after iteratively increasing the search area or upon failure to determine the relative position of the current image frame to any neighboring keyframe, the method further comprising:

assigning the current image frame position to be the current image frame tracking position or, upon failure to determine the current image frame tracking position, to be the previous image frame position.

6. The method of claim 1, further comprises identifying a set of features in each of the series of image frames by

setting a feature strength threshold;

obtaining a list of potential features, the list including at least one potential feature that exceeds the feature strength threshold;

identifying the set of features in each of the series of image frames by selecting from the list of potential features only features having the maximum feature strength within a pre-defined neighborhood region.

7. The method of claim 1, wherein the relative displacement of the current image frame from the previous image frame is determined by registering the current image frame and the previous image frame based on at least one feature in the current image frame and the previous image frame.

8. The method of claim 1, wherein the relative displacements of the current image frame from each of the at least one neighboring keyframes is determined by registering the current image frame and each of the at least one keyframe based on the at least one feature in the current image frame and each of the at least one neighboring keyframes.

9. The method of claim 8, wherein registering the current image frame and each of the at least one neighboring keyframes includes:

matching a set of features identified in the current image frame with a corresponding set of features in each of the at least one neighboring keyframes to obtain a set of matched feature pairs;

selecting acceptable matched feature pairs from the set of matched feature pairs; and

determining the relative displacement between the current image frame and each of the least one neighboring keyframes based on the average displacements of the acceptable matched feature pairs.

10. The method of claim 9, wherein each identified feature includes a coordinate, a descriptor row vector of N elements, N being an integer, and the matching of the set of features comprises:

comparing the descriptor row vector for each feature in the set of features identified in the current image frame with the descriptor row vector of each feature in the corresponding set of features in each of the at least one neighboring keyframes; and

identifying a best-match feature in the at least one neighboring keyframe for each feature in the current image frame to form a matched feature pair, the best-match feature having the least descriptor distance from the feature in the current image frame.

11. The method of claim 9, wherein each identified feature includes a coordinate, an orientation parameter, a scale parameter, and a descriptor row vector of N elements, N being an integer, and the matching of the set of features comprises:

determining a set of match candidates for each feature in the set of features identified in the current image frame from the corresponding set of features in each of the at least one neighboring keyframes, each match candidate having its orientation parameter and scale parameter similar to that of the corresponding feature in the current image frame; and

for each feature in the set of features identified in the current image frame, identifying a best-match feature in the at least one neighboring keyframe based on a match candidate from the set of match candidates having the least descriptor distance from the feature in the current image frame to form a matched feature pair.

12. The method of claim 9, further comprising:

computing a feature displacement for each matched feature pair based on their respective x-coordinates and y-coordinates;

accumulating computed feature displacements in a 2D histogram; and

selecting the matched feature pairs accumulated in a bin having the highest accumulation in the 2D histogram and its neighboring bins as acceptable matched feature pairs.

13. The method of claim 9, wherein selecting acceptable matched feature pairs from the set of matched feature pairs comprises:

determining an average feature displacement from the set of matched feature pairs;

determining residual values of each matched feature pair from the average displacement;

determining the average residual value for all matched feature pairs; and

selecting the matched feature pairs having a residual value less than a threshold residual value as acceptable matched feature pairs.

14. The method of claim 1, wherein rendering the keyframes further comprises:

generating a set of texture atlases, each texture atlas having an associated resolution and including at least one keyframe downsampled at the associated resolution;

selecting a display area and a display resolution for the rendering;

selecting and updating a texture atlas having a resolution closest to the display resolution from the set of texture atlases; and

rendering the keyframes from the texture atlas whose associated keyframe positions are within the display area.

15. The method of claim 14, wherein updating the texture atlas further comprises:

identifying a set of keyframes from the keyframe database having associated keyframe positions within the display area and not yet in the texture atlas;

downsampling the set of keyframes at the associated resolution of the texture atlas;

and inserting the downsampled set of keyframes into the texture atlas.

16. The method of claim 1, further comprising:

upon determining that at least one of the relative displacements of the current image frame from each of the at least one neighboring keyframes is less than the threshold value,

identifying at least one neighboring keyframe whose relative displacement is less than the threshold value;

determining a region of overlap between the current image frame and the at least one neighboring keyframe to obtain a region of interest in each of the current image frame and the at least one neighboring keyframe;

replacing the region of interest of the at least one neighboring keyframe with the region of interest of the current image frame when the region of interest of the current image frame is determined to be sharper than the region of interest of the one of the neighboring keyframes based on a image sharpness parameter.

17. The method of claim 16, wherein the image sharpness parameter is a gradient sum obtained from gradient images of the region of interest of the current image frame and the at least one neighboring keyframe, respectively, and the region of interest of the current image frame is determined to be sharper than the region of interest of the one of the neighboring keyframes when the gradient sum of the region of interest of the current image frame is greater than that of the one of the neighboring keyframes.

18. The method of claim 16, wherein replacing the region of interest of the at least one neighboring keyframe includes overwriting contents of the region of interest of the at least one neighboring keyframe with that contents of the region of interest of the current image frame.

19. The method of claim 16, further comprising blending contents at the boundaries of the region of interest of the current image frame with those of the at least one neighboring keyframe.

20. The method of claim 1, further comprising:

assigning each keyframe in the keyframe database to one of a plurality of cells based on its associated keyframe position; and

tagging each keyframe to map its associated keyframe position and its respective assigned cell.

21. The method of claim 20, wherein the plurality of cells are aligned in a grid and the cells are indexed using a 2-dimensional index.

22. The method of claim 21, wherein the search area comprises at least one or more indexed cells.

23. The method of claim 1, wherein the rendering of the mosaic representation of the object includes rendering mosaic representations obtained from different magnification objectives, and the method further comprises:

maintaining a separate keyframe database associated with each magnification objective;

associating the mosaic representations obtained from each magnification objective based on a relative displacement of respective objective optical centers; and

rendering a combined mosaic representation of the object based on the associated mosaic representations.

24. The method of claim 1, further comprising:

pre-processing the current image frame, the pre-processing including image calibration to correct illumination variance across a field of view, conversion of color scales to a grayscale, downsampling, feature detection, or gradient computation.

25. The method of claim 1, wherein rendering the keyframes comprises blending of overlapping keyframes.

26. The method of claim 25, wherein blending of overlapping keyframes includes

decomposing keyframes into a plurality of bands;

blending the plurality of bands; and

merging the blended plurality of bands to obtain the mosaic representation of the object.

27. The method of claim 26, wherein decomposing keyframes is based on spatial frequency.

28. The method of claim 1, further comprising

adjusting the gain of the current image frame to a substantially uniform level of brightness by comparing against the gains of the plurality of keyframes.

29. The method of claim 1, further comprising

detecting a occlusion in the current image frame based on an average image pixel intensity falling below a threshold intensity value;

and

detecting a change in an exposure time of subsequent image frames

to determine a change in objective lens used for acquiring image frames.

30. The method of claim 1, wherein a first image frame in the series of image frames is assigned as a reference keyframe having a reference keyframe position.

33. A computer program product comprising a non-transitory computer-readable memory storing computer executable instructions thereon that when executed by a computer perform the method of claim 1.

31. An imaging system comprising:

an image acquisition module to acquire a current image frame of an object;

a processor configured to;

determine a current image frame tracking position of the current image frame based on a previous image frame position of a previous image frame and a relative displacement of the current image frame from the previous image frame;

identify at least one neighboring keyframe within a search area defined relative to the current image frame tracking position, the at least one neighboring keyframe being one of a plurality of keyframes stored in a keyframe database, each keyframe having an associated keyframe position;

determine relative displacements of the current image frame from each of the at least one neighboring keyframes;

determine current image frame positions of the current image frame from the associated keyframe frame position of each of the at least one neighboring keyframes and the relative displacement of the current image frame from each of the at least one neighboring keyframes;

determine a current image frame position as an average of the current image frame positions;

store the current image frame as a new keyframe in the keyframe database upon determining that each of the relative displacements of the current image frame from each of the at least one neighboring keyframes is greater than a threshold value;

store the current image frame position as a new keyframe position associated with the new keyframe in the keyframe database; and

,

a renderer to render the current image frame at its current image frame position and the keyframes from the keyframe database on a display at their respective associated keyframe positions to display a mosaic representation of the object.

32. The imaging system of claim 31, further comprising:

a controller to receive input from an input device or the processor to control the image acquisition module or the renderer in response to the received inputs,

and the image acquisition module includes

a microscope having at least one objective; and

a video camera for acquiring a series of image frames of the object.

34. A method for generating keyframes from a series of image frames of an object for rendering a mosaic representation of the object, the method comprising:

determining a current image frame tracking position based on a displacement of a current image frame relative to known position of a previous image frame;

identifying at least one neighboring keyframe within a predetermined search area surrounding the current image frame tracking position;

determining a current position of the current image frame relative to each of the at least one neighboring keyframe based on known respective keyframes positions of each of the at least one neighboring keyframes and relative displacements of the current image frame from each of the at least one neighboring keyframes;

storing the current image frame as a new keyframe when the relative displacements of the current image frame from each of the at least one neighboring keyframes is greater than a threshold value.

35. A method of registering a first image and a second image comprising:

identifying a set of features in the first image and a corresponding set of features in the second image, each identified feature including a coordinate, an orientation parameter, a scale parameter, and a descriptor row vector of N elements, N being an integer;

determining a set of match candidates for each feature in the set of features of the first image frame from the corresponding set of features in the second image frame, each match candidate having its orientation parameter and scale parameter similar to that of the corresponding feature in the first image frame;

for each feature in the set of features of the first image, determining a best-match candidate from the set of match candidates, the best-match candidate having the least descriptor distance from the feature in the first image, to form a matched feature pair;

computing a feature displacement for each matched feature pair based on their respective x-coordinates and y-coordinates;

accumulating the computed feature displacements in a 2D histogram;

selecting the matched feature pairs accumulated in a bin having the highest accumulation in the 2D histogram and in its neighboring bins;

determining an average feature displacement from the selected matched feature pairs;

determining the residual values of each selected matched feature pair from the average feature displacement;

determining the average residual value from all selected matched feature pairs;

retaining matched feature pairs having a residual value less than a threshold factor of the average residual value as acceptable matched feature pairs; and

determining average displacement between the first image and the second image based on the relative displacement of the acceptable matched feature pairs for image registration.