Image guidance system for detecting and tracking an image pose

• US 9,943,374 B2
• Filed: 09/16/2014
• Issued: 04/17/2018
• Est. Priority Date: 09/16/2014
• Status: Active Grant

• Current Assignee: X-Nav Technologies, Inc., X-Nav Technologies, LLC
• Original Assignee: X-Nav Technologies, LLC
• Inventors: Merritt, Scott A., Emery, III, Robert W., Marandola, Edward J., Scharff, Christopher W.
• Primary Examiner(s): Park, Chan
• Assistant Examiner(s): Summers, Geoffrey E
• Application Number:
  US14/488,004
• Publication Number:
  US 20160074127A1
• Time in Patent Office: 1,309 Days
• Field of Search: 382103, 382154
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2057 Details of tracking cameras
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/373 using light, e.g. by using optical scanners
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 2090/3983 Reference marker arrangements for use with image guided surgery
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 90/39 Markers, e.g. radio-opaque or breast lesions markers
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/30036 Dental; Teeth
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/593 from stereo images
• CPC Class Codes: G06T 7/74 involving reference images or patches

Abstract

An image guidance system for tracking a surgical instrument during a surgical procedure. The image guidance system includes a plurality of cameras adapted to be located external to a surgical area for capturing images of optically visible patterns. A processing system receives and processes the images to recognize the patterns and triangulate the locations and orientations of each pattern relative to a camera. The processing system uses a reference dataset to recognize the patterns.

Claims

1. A system for tracking the pose of an object displaceable in a coordinate reference frame, the system comprising:

an optically-visible target pattern on an object being tracked, the pattern containing a plurality of 2D contrasting shapes located on a tile, the contrasting shapes being in an arrangement so as to uniquely differentiate the pattern from other patterns;

a storage media containing stored data for a plurality of distinct patterns of contrasting shapes including, for each distinct pattern, the arrangement of contrasting shapes and locations of defined points in the pattern;

an optical sensor assembly spaced apart from the object, the optical sensor assembly located and configured to capture images of at least a portion of the pattern, the sensor assembly including at least two cameras, each camera configured to capture a 2D image of the portion of the pattern, each 2D image being at a different viewing angle of the portion of the pattern than the other 2D image(s); and

a processor configured to (i) retrieve the stored data for the plurality of distinct patterns of contrasting shapes, (ii) receive the 2D images from the sensor assembly, (iii) locate a plurality of defined points in each 2D image and determine a pair of basis vectors between each defined point and two of its adjacent defined points, (iv) determine a rectifying affine transform applicable to contrasting shapes in the pattern in each 2D image and predict grid locations in each 2D image, (v) determine a set of descriptors for each 2D image, each descriptor being determined based on coloring or shading of the contrasting shapes adjacent to each predicted grid location, and (vi) compare the descriptors to a library of descriptors in the stored data to select a specific pattern in the stored data that is closest to the pattern in each 2D image, and the processor configured to (vii) match defined points in one of the 2D images with corresponding defined points in the other 2D image and perform stereo triangulation with a location of the cameras to determine an estimate of a 3D location of each matched defined point, and (viii) calculate a second transform to convert the matched defined points to a target coordinate system to determine the pose of the object;

wherein the processor is configured to use the rectifying affine transform to provide 2D translation, rotation, and scaling of the located defined points in the 2D images for correlating with 3D coordinates in the stored data for the identified distinct pattern.

2. The system of claim 1, wherein the target pattern includes a plurality of the tiles, and wherein the contrasting shapes in the pattern are contrast boxes, the contrast boxes on each tile including groups of boxes of a same color adjacent to one another.

3. The system of claim 1, wherein the second transform is a rigid-body transformation.

4. The system of claim 1, wherein the second transform is 3D affine transformation.

5. The system of claim 1, wherein the rectifying affine transform includes non-linear deformations to conform the contrasting shapes to a non-planar surface, and wherein the non-planar surface is selected from a group including a surface defining a portion of a spherical surface, a cylinder surface, and a conical surface.

6. A system according to claim 1, wherein the contrasting shapes include adjacent boxes of distinct color, each set of adjacent boxes being separated from one another by an edge, and wherein the processor locates the defined points at a center point of two or four intersecting edges.

7. A system according to claim 1, wherein the contrasting shapes are a mixed grid of black and white boxes, the number of boxes of each color being enough to produce approximately 50% intensity, and wherein the grid includes a plurality of black contrast boxes adjacent to one another and a plurality of white contrast boxes adjacent to one another.

8. The system of claim 1, wherein the tile comprises a grid of boxes on the surface of the object, each box having a color or shade that is selected from a group of at least two colors or shading that are optically distinguishable from one another, wherein a first plurality of the boxes each have a first color or shading of the at least two colors or shading and are adjacent to one another within the grid, and a second plurality of the boxes each have a second color or shading of the at least two colors or shading and are adjacent to one another within the grid thereby forming an arrangement of boxes in the grid in a manner to uniquely differentiate the target pattern on the tile from other target patterns on other tiles.

9. The system of claim 8, wherein the target pattern comprises a plurality of said grids of boxes positioned adjacent to one another.

10. The system of claim 1, further comprising a second optically-visible target pattern on a second object being tracked, the second pattern containing a plurality of 2D contrasting shapes located on a tile, the contrasting shapes being in an arrangement so as to uniquely differentiate the pattern from the first said pattern and other patterns;

the processor configured to determine the relative pose of the second object and the first said object.

11. The system of claim 10, wherein one of the first and second objects is a surgical implement and the other of the first and second objects is a fixture to be attached to a patient.

12. A system for tracking the pose of an object displaceable in a coordinate reference frame, the system comprising:

an optically-visible target pattern on an object being tracked, the pattern containing a plurality of 2D contrasting shapes located on a tile, the contrasting shapes being in an arrangement so as to uniquely differentiate the pattern from other patterns;

a storage media containing stored data for a plurality of distinct patterns of contrasting shapes including, for each distinct pattern, the arrangement of contrasting shapes and locations of defined points in the pattern;

an optical sensor assembly spaced apart from the object, the optical sensor assembly located and configured to capture images of at least a portion of the pattern, the sensor assembly including at least a first and second camera, each of the first and second cameras arranged to capture a 2D image of the portion of the pattern, each 2D image being at a different viewing angle of the portion of the pattern than the other 2D image; and

a processor configured to (i) retrieve the stored data for the plurality of distinct patterns of contrasting shapes, (ii) receive the 2D images from the cameras in the sensor assembly, (iii) locate a plurality of defined points in each 2D image and determine a pair of basis vectors between each defined point and two of its adjacent defined points, (iv) determine a rectifying affine transform applicable to contrasting shapes in the pattern in each 2D image and predict grid locations in each 2D image, (v) determine a set of descriptors for each 2D image, each descriptor being determined based on coloring or shading of the contrasting shapes adjacent to each predicted grid location, and (vi) compare the descriptors associated with each 2D image to a library of descriptors in the stored data to select a specific pattern in the stored data that is closest to the pattern in the 2D image, and the processor configured to (vii) match defined points in one of the 2D images with corresponding defined points in the other 2D image and perform stereo triangulation with a location of the cameras to determine an estimate of a 3D location of each matched defined point, and (viii) calculate a second transform to convert the matched defined points to a target coordinate system to determine the pose of the object;

wherein the processor uses the rectifying affine transform to provide 2D translation, rotation, and scaling of the located defined points in the 2D images for correlating with 3D coordinates in the stored data for the identified distinct pattern.

13. The system of claim 12 wherein the processor is configured to compute 3D coordinates of the located defined points in the patterns in each of the 2D images.

14. The system of claim 12, wherein the target pattern includes a plurality of tiles, and wherein the 2D contrasting shapes are contrast boxes, the contrast boxes on each tile including groups of boxes of the same color adjacent to one another.

15. The system of claim 12, wherein the second transform is a rigid-body transformation.

16. The system of claim 12, wherein the second transform is 3D affine transformation.

17. The system of claim 12, wherein the rectifying affine transform includes non-linear deformations to conform the contrasting shapes to a non-planar surface, and wherein the non-planar surface is selected from a group including a surface defining a portion of a spherical surface, a cylinder surface, and a conical surface.

18. A system according to claim 12, wherein the contrasting shapes include adjacent boxes of distinct color, each set of adjacent boxes being separated from one another by an edge, and wherein the processor locates the defined points at a center point of two or four intersecting edges.

19. A system according to claim 12, wherein the contrasting shapes are a mixed grid of black and white boxes, the number of boxes of each color being enough to produce approximately 50% intensity, and wherein the grid includes a plurality of black contrast boxes adjacent to one another and a plurality of white contrast boxes adjacent to one another.

20. The system of claim 12, wherein the tile comprises a grid of boxes on the surface of the object, each box having a color or shade that is selected from a group of at least two colors or shading that are optically distinguishable from one another, wherein a first plurality of the boxes each have a first color or shading of the at least two colors or shading and are adjacent to one another within the grid, and a second plurality of the boxes each have a second color or shading of the at least two colors or shading and are adjacent to one another within the grid thereby forming an arrangement of boxes in the grid in a manner to uniquely differentiate the pattern on the tile from other patterns on other tiles.

21. The system of claim 20, wherein the target pattern comprises a plurality of said grids of boxes positioned adjacent to one another.

22. The system of claim 20, wherein determining the pose of the object comprises identifying a first feature of the pattern, identifying adjacent features along box edges, identifying basis vectors between the first feature and the adjacent features, and determining a rectifying transform to transform a shape formed by the identified basis vectors to a corresponding shape of the actual pattern.

23. A system for tracking the pose of an object displaceable in a coordinate reference frame, the system comprising:

an optically-visible target pattern on an object being tracked, the pattern containing a plurality of 2D contrasting shapes, the contrasting shapes defining a grid having a plurality of boxes, each box having a color selected from at least two optically distinguishable colors, the boxes arranged based on their color so as to uniquely differentiate the pattern from other patterns;

a storage media containing stored data for a plurality of distinct patterns of contrasting shapes including, for each distinct pattern, the arrangement of contrasting shapes and locations of defined points in the pattern;

an optical sensor assembly spaced apart from the object, the optical sensor assembly located and configured to capture images of at least a portion of the pattern, the sensor assembly including at least a first and second camera, each of the first and second cameras arranged to capture a 2D image of the portion of the pattern, each 2D image being at a different viewing angle of the portion of the pattern than the other 2D image; and

a processing system including a camera processor associated with each camera and a main processor, the processing system configured to (i) receive the 2D images from the cameras, (ii) retrieve the stored data for the plurality of distinct patterns of contrasting shapes, (iii) for each 2D image, detect defined points between adjacent contrasting colored boxes and determine basis vectors between a defined point and two of its adjacent defined points, (iv) for each 2D image, determine a rectifying affine transform based on the basis vectors that transforms a rhombohedral image patch into a square image patch, three of whose corners are the defined point and its two adjacent defined points, wherein the affine transform is used to predict where each grid location should be in the image, (v) analyze each 2D image at each predicted grid location to calculate a descriptor which describes the grid location based on locations of adjacent distinct color boxes, and (vi) compare the descriptors to a library of descriptors in the stored data to select a specific pattern in the stored data that is closest to the pattern in the 2D image, and use the selected pattern in the stored data to determine the pose of the object in a camera-centric frame of reference of each camera, and configured to (vii) match defined points in one of the 2D images with corresponding defined points in the other 2D image and perform stereo triangulation with a location of the cameras to determine an estimate of a 3D location of each matched defined point, and (viii) calculate a second transform to convert the matched defined points to a target coordinate system to determine the pose of the object.

24. The system of claim 23, wherein the pattern in the stored data includes a minimum of thirteen defined points.

25. The system of claim 23, wherein the processing system is configured to calculate a score based on how closely the descriptor matches stored information on each pattern in the stored data.

26. The system of claim 23, wherein the processing system is configured to analyze a sub-region of the 2D image and determine if the image of the pattern is nonplanar based on variations between locations of the defined points relative to the calculated grid locations, and if it determines that the image of the title is nonplanar, determines corrections to the affine transform.

27. The system of claim 23, wherein the processing system is configured to compare the calculated descriptors to a library of descriptors for each stored pattern, and calculate a score to identify a model for one or more known stored patterns in a library of models, and wherein for each stored pattern in a model, the stored data includes 3D model locations for each point on the grid where defined points are located, and wherein the processing system is configured to determine a correspondence between at least four corners in the pattern in the image and four model locations representing corners in the stored pattern, and wherein the processing system estimates a rigid-body transform that defines a spatial relationship of the model in a camera-centric coordinate system from the correspondences between at least the four corners in the pattern in the image and the four model locations representing corners in the stored pattern.

28. The system of claim 27, wherein each stored pattern is unique and wherein the selection of the specific pattern in the stored data determines a selected model from the library of models, the selected model having a plurality of defined points.

29. The system of claim 27, wherein several models in the library of models share the same stored pattern but in different arrangements, and wherein the processing system is configured to apply the patterns in each of the several models onto each 2D image using the rigid-body transform, and determine the number of correspondences between defined points in each of the models and the detected defined points in each 2D image, and wherein a model is selected from the several models which has the highest number of correspondences.

30. The system of claim 29, wherein the processing system is configured to (i) perform stereo reconstruction by triangulating the detected defined points in each 2D image to determine each detected defined point'"'"'s 3D location in each of the first and second camera'"'"'s coordinate system, (ii) compare the detected defined point'"'"'s 3D locations and the 3D location for each defined points in the selected model to determine an association between a set of pixel locations in each 2D image in the first camera'"'"'s coordinate system and a set of pixel locations in each 2D image in the second camera'"'"'s coordinate system, and based on the association of pixel locations to determine an estimate of a 3D location of each pixel in a model-centric coordinate system.

31. The system of claim 30, wherein the processing system is configured to find at least three correspondences between 3D locations of detected defined points in each 2D image and 3D locations of defined points in the selected model, and determine a rigid-body transformation to correlate the first and second camera'"'"'s coordinate system to the model-centric coordinate system, thereby determining spatial location and orientation of the selected model in the first and second camera'"'"'s coordinate system and permitting the pose of the pattern to be tied to the selected model for depicting movement of the tile.