FIDUCIAL MARKER DESIGN AND DETECTION FOR LOCATING SURGICAL INSTRUMENT IN IMAGES

• US 20100168562A1
• Filed: 04/23/2009
• Published: 07/01/2010
• Est. Priority Date: 12/31/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Incorporated (Intuitive Surgical Holdings, LLC)
• Inventors: Halabe, Daniel J., Nowlin, William C, Hoffman, Brian D., Zhao, Tao, Zhao, Wenyi
• Granted Patent: US 9,526,587 B2
• Time in Patent Office: Days
• US Class Current: 600/426
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 2090/3983 Reference marker arrangements for use with image guided surgery
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/94 coded with symbols, e.g. text

Abstract

The present disclosure relates to systems, methods, and tools for tool tracking using image-derived data from one or more tool-located reference features. A method includes: capturing a first image of a tool that includes multiple features that define a first marker, where at least one of the features of the first marker includes an identification feature; determining a position for the first marker by processing the first image; determining an identification for the first marker by using the at least one identification feature by processing the first image; and determining a tool state for the tool by using the position and the identification of the first marker.

Claims

1. A robotic surgical method comprising:

capturing a first image of a surgical tool, the surgical tool including a plurality of features defining a first marker, at least one of the features of the first marker comprising an identification feature;

determining a position for the first marker by processing the captured first image;

determining an identification of the first marker by using the at least one identification feature by processing the first image; and

determining a tool state for the imaged tool by using said position for the first marker and said identification of the first marker.

2. A method as in claim 1, wherein the first marker comprises redundant features defining error-checking data, the method further comprising:

processing the first image to detect the redundant features and read the error-checking data; and

validating the identification of the first marker by verifying that the first marker identification is consistent with the error-checking data.

3. A method as in claim 2, wherein the redundant features define check sum data, the method further comprising:

processing the first image to read the check sum data; and

validating the identification of the first marker by verifying that the first marker identification is consistent with the check sum data.

4. A method as in claim 1, wherein the tool includes a plurality of markers, each marker having at least one identification feature associated with an identification of the marker, the identifications of the markers of the tool differing from each other, wherein the processing of the first image is performed bar a processor having data indicating, for each marker, an associated predetermined positional relationship between the marker and a joint of the surgical tool, and wherein determining the tool state comprises determining a position of the tool by using the position of the first marker, the identification of the first marker, and the associated positional relationship data for the first marker.

5. A method as in claim 4, wherein the first image includes a second marker of the tool, the method further comprising:

determining a position for the second marker by processing the first image;

determining the identification of the second marker by processing the first image; and

determining a tool state for the imaged tool by using said position for the second marker, said identification of the second marker, and said predetermined positional relationship data associated with the second marker.

6. A method as in claim 4, wherein the tool includes a second marker, the second marker obscured in the first image, the method further comprising:

moving the tool after determining the tool state by using the first marker;

capturing a second image of the moved tool, the second image encompassing the second marker, the first marker being obscured in the second image;

determining a position for the second marker by processing the second image;

determining the identification of the second marker by processing the second image; and

determining a moved tool state for the tool by using said position for the second marker, said identification of the second marker, and said predetermined positional relationship data associated with the second marker.

7. A method as in claim 1, further comprising:

capturing a second image of the surgical tool, wherein the first and second images are captured by using a stereoscopic endoscope; and

determining three-dimensional positional data for the first marker by processing the first and second images, wherein the tool state is determine in three dimensions or more.

8. A method as in claim 1, wherein the surgical tool includes a plurality of markers, each marker having at least one localizer feature and at least one identification feature with a known positional relationship relative to the at least one localizer feature, wherein the position of the first marker is determined by using said at least one localizer feature, and wherein the identification of the first marker is determined by identifying said at least one localizer feature of the first marker and by reading said at least one identification feature according to the known positional relationship between the localizer feature and said at least one identification feature.

9. A method as in claim 1, wherein the plurality of features includes at least one localizer feature, and wherein the determining an identification of the first marker step comprises:

processing the first image so as to identify the at least one localizer feature;

selecting a candidate identity for the first marker;

generating a candidate view of a marker having the candidate identity by using the identified at least one localizer feature; and

comparing the candidate view with the first image so as to verify that the selected candidate identity is the first marker identity.

10. A method as in claim 9, wherein the selecting a candidate identity for the first marker step comprises generating an estimated pose for the surgical tool by using at least one prior tool state from a prior image of the tool or joint data from a robotic actuation system effecting movement of the tool, wherein the selected candidate identity results in a candidate pose for the surgical tool that is within a predetermined deviation of the estimated pose for the surgical tool.

11. A method as in claim 10, wherein the first image encompasses another surgical tool, each tool having an identity, and further comprising associating an identity with the imaged tool having the first marker by verifying that the candidate identity for the first marker results in a candidate pose that is within a predetermined deviation of the estimated pose for the surgical tool having the first marker.

12. A method as in claim 1, wherein a Maximum Stable Extremal Region (MSER) or adaptive thresholding is used to determine positional data for the first marker.

13. A robotic surgery system, comprising:

a surgical tool having a plurality of features defining a first marker, at least one of the features of the first marker comprising an identification feature;

an imaging device for capturing a first image of said tool during use and output first image data in response thereto; and

a processor coupled with the imaging device, the processor adapted to process said captured first image data so as to;

determine positional data for the first marker;

determine an identification of the first marker by using the identification feature; and

determine tool state data for the imaged tool by using said first marker positional data and said identification of the first marker.

14. A system as in claim 13, further comprising a tangible medium comprising machine-readable instructions executable by said processor for processing said captured first image.

15. A system as in claim 13, further comprising an input for non-endoscopically derived tool state data, the non-endoscopically derived tool state data derived from robotic joints supporting the tool, and wherein the processor is further configured to process the non-endoscopically derived tool state information and the image-derived tool state information for tracking the state of the tool.

16. A system as in claim 13, wherein the imaging device is adapted to capture a second image at substantially the same time as the first image, and output second image data in response thereto, and wherein the processor is further configured so as to determine three-dimensional positional data for the first marker by processing the first and second image data.

17. A system as in claim 16, wherein the imaging device comprises a stereoscopic endoscope.

18. A system as in claim 13, wherein said first marker comprises redundant features defining error-checking data, and wherein the processor is further configured to process said first image data so as to:

detect the first marker redundant features;

read the error-checking data; and

validate the identification of the first marker by verifying that the first marker identification is consistent with the error-checking data.

19. A system as in claim 18, wherein the redundant features define check-sum data, and wherein the processor is further configured to process said first image data so as to read the check-sum data;

and validate the identification of the first marker by verifying that the first marker identification is consistent with the check-sum data.

20. A system as in claim 13, wherein a plurality of markers are distributed around the tool, wherein the processor comprises data for each marker indicating an associated marker identification and an associated positional relationship between the marker and a joint of the surgical tool, and wherein the markers have identification features that differ sufficiently for the processor to determine the identification of the markers encompassed within the first image.

21. A system as in claim 20, wherein at least one of the markers comprises at least one localizer feature that is shared with an adjacent marker.

22. A system as in claim 20, wherein the features of each of the markers are arranged in a two-dimensional pattern.

23. A system as in claim 22, wherein each of the markers uses a plurality of circles as localizer features.

24. A system as in claim 22, wherein each of the markers uses a plurality of corners as localizer features.

25. A system as in claim 24, wherein the corners comprise saddle points.

26. A system as in claim 22, wherein each of the markers comprises three localizer features.

27. A system as in claim 22, wherein each of the markers comprises four localizer features.

28. A system as in claim 22, wherein each of the markers comprises four circles and a bar as localizer features.

29. A system as in claim 13, wherein the first marker comprises text.

30. A system as in claim 29, wherein the first marker comprises text modified to increase positional data or discriminative features.

31. A system as in claim 13, wherein the processor is coupled with a display and the processor outputs modified first image data to the display that alters a portion of the first image corresponding to a reference feature.

32. A surgical tool for use with a robotic surgery system, the surgery system including an imaging device for capturing an image of the surgical tool during use and a processor coupled with the imaging device for processing the captured image so as to determine image-derived positional information for the surgical tool, the surgical tool comprising a plurality of markers, each marker having at least one identification feature, the identification features of each marker differing sufficiently for the surgery system to discriminate between the markers based on images encompassing the markers.

33. A robotic surgical method comprising:

capturing a first image of a surgical tool, the surgical tool including a plurality of features defining a plurality of markers, each marker having a predetermined positional relationship with the surgical tool, the first image including one of the markers;

determining a position for the imaged marker by processing the first image;

generating an estimated tool state for the tool by using at least one prior tool state from a prior image of the tool or joint data from a robotic actuation system effectuating movement of the tool; and

determining a tool state for the tool by using said position for the imaged marker, said predetermined positional relationship between the surgical tool and the imaged marker, and said estimated tool state for the tool.