Fiducial marker design and detection for locating surgical instrument in images

• US 9,526,587 B2
• Filed: 04/23/2009
• Issued: 12/27/2016
• Est. Priority Date: 12/31/2008
• Status: Active Grant

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Zhao, Tao, Zhao, Wenyi, Halabe, Daniel J., Hoffman, Brian D., Nowlin, William C.
• Primary Examiner(s): Akar, Serkan
• Application Number:
  US12/428,657
• Publication Number:
  US 20100168562A1
• Time in Patent Office: 2,805 Days
• Field of Search: 318/568.21, 378/205, 382/159, 600/117, 600/407, 600/424, 600/426, 600/429, 606/130
• US Class Current: 1/1
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 2090/3983 Reference marker arrangements for use with image guided surgery
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: A61B 34/37 Master-slave robots A61B34/35 takes precedence
• CPC Class Codes: A61B 90/94 coded with symbols, e.g. text

Abstract

The present disclosure relates to systems, methods, and tools for tool tracking using image-derived data from one or more tool-located reference features. A method includes: capturing a first image of a tool that includes multiple features that define a first marker, where at least one of the features of the first marker includes an identification feature; determining a position for the first marker by processing the first image; determining an identification for the first marker by using the at least one identification feature by processing the first image; and determining a tool state for the tool by using the position and the identification of the first marker.

Claims

1. A method implemented by a processor, the method comprising:

receiving, at the processor, information of an image of a surgical tool that was captured by an image capture device, wherein the surgical tool has a first marker that includes a first identification feature which includes a first plurality of dots, the first plurality of dots arranged in a first pattern of dots which indicates values of first data bits and a value or values of at least one first error checking bit, wherein a value of a corresponding one of the first data bits and the at least one first error checking bit is determined by a presence or absence of a dot in each location of a predefined pattern of possible dot locations, wherein the values of the first data bits indicate an identity of the first marker, and wherein the value or values of the at least one first error checking bit indicates one of an informational redundancy and a correctness of the first data bits;

determining, by using the processor, a position and orientation for the first marker by processing the received image;

determining, by using the processor, the identity of the first marker by comparing the first pattern of dots against a plurality of patterns of dots, wherein each of the plurality of patterns of dots identifies a unique marker by depicting a different combination of the presence and absence of dots in the predefined pattern of possible dot locations than all others of the plurality of patterns of dots;

validating, by using the processor, the determined identity of the first marker by using the value or values of the at least one first error checking bit;

determining, by using the processor, a tool state for the surgical tool by using the determined position and orientation for the first marker and the determined identity of the first marker after being validated; and

commanding, by using the processor, a robotic mechanism to move the surgical tool in response to input from a control device by using the determined tool state for the surgical tool.

2. The method as in claim 1,

wherein the first plurality of dots is arranged to indicate the at least one first error checking bit so as to indicate the informational redundancy of the first data bits by an inverse of the first pattern in locations corresponding to the at least one first error checking bit in the predefined pattern of possible dot locations.

3. The method as in claim 2, wherein the first plurality of dots is arranged to indicate the at least one first error checking bit so as to indicate the correctness of the first data bits by indicating the value for the at least one error checking bit that is equal to a check-sum of the values of the first data bits.

4. The method as in claim 1, wherein the surgical tool has a shaft and a plurality of markers arranged in a single ring extending around a circumference of the shaft so that at least one of the plurality of markers is viewable at any angle of rotation of the shaft about a longitudinal axis of the shaft, wherein the plurality of markers includes the first marker and a second marker, the second marker includes a second identification feature which includes a second plurality of dots, the second plurality of dots arranged in a second pattern of dots so as to indicate values of second data bits and a value or values of at least one second error bit, wherein a value of a corresponding one of the second data bits and the at least one second error checking bit is determined by a presence or absence of a dot in each location of the predefined pattern of possible dot locations, wherein the values of the second data bits indicate an identity of the second marker, wherein the value or values of the at least one second error checking bit indicates one of an informational redundancy and correctness of the second data bits, wherein the identity of the second marker is different than the identity of the first marker, and wherein information of a positional relationship between each of the first and second markers with respect to the surgical tool is provided to the processor.

5. The method as in claim 1, wherein the image is a stereo image captured by using a stereoscopic endoscope, the method further comprising:

determining, by using the processor, three-dimensional positional data for the first marker by processing the stereo image, wherein the tool state is determined in at least three dimensions.

6. The method as in claim 1, wherein the first marker includes at least one localizer feature with a known positional relationship relative to the first identification feature, wherein the position of the first marker is determined by using said at least one localizer feature.

7. The method as in claim 1, wherein the first marker is on a curved surface of the surgical tool, and wherein the first marker is fully visible in the image.

8. A medical system, comprising:

a surgical tool having a first marker that includes a first identification feature that includes a first plurality of dots, the first plurality of dots arranged in a first pattern of dots which indicates values of first data bits and a value or values of at least one first error checking bit, wherein a value of a corresponding one of the first data bits and the at least one first error checking bit is determined by a presence or absence of a dot in each location of a predefined pattern of possible dot locations, wherein the values of the first data bits indicate an identity of the first marker, and wherein the value or values of the at least one first error checking bit indicates one of an informational redundancy and a correctness of the first data bits;

a robotic mechanism coupled to the surgical tool;

a control device;

an imaging device for capturing an image of the first marker on said surgical tool; and

a processor programmed to process said captured image so as to;

determine positional and orientational data for the first marker, determine the identity of the first marker by comparing the first pattern of dots against a plurality of patterns of dots, wherein each of the plurality of patterns of dots identifies a unique marker by depicting a different combination of the presence and absence of dots in the predefined pattern of possible dot locations than all others of the plurality of patterns of dots, validate the determined identity of the first marker by using the value or values of the at least one first error checking bit, determine a tool state for the surgical tool by using the determined first marker positional and orientational data and the determined identity of the first marker after being validated, and command the robotic mechanism to move the surgical tool in response to input from the control device by using the determined tool state for the surgical tool.

9. The medical system as in claim 8, further comprising a tangible medium comprising non-transitory machine-readable instructions executable by said processor for processing said captured image.

10. The medical system as in claim 8, further comprising an input for non-endoscopically derived tool state data, the non-endoscopically derived tool state data derived from robotic joints supporting the surgical tool, and wherein the processor is programmed to process the non-endoscopically derived tool state information and the image-derived tool state information for tracking the state of the tool.

11. The medical system as in claim 8, wherein the imaging device is adapted to capture a stereo image, and wherein the processor is programmed to determine three-dimensional positional and orientational data for the first marker by processing the stereo image.

12. The medical system as in claim 11, wherein the imaging device comprises a stereoscopic endoscope.

13. The medical system as in claim 8,

wherein the first plurality of dots is arranged to indicate the at least one first error checking bit so as to indicate the informational redundancy of the first data bits by an inverse of the first pattern in locations corresponding to the at least one first error checking bit in the predefined pattern of possible dot locations.

14. The medical system as in claim 13, wherein the first plurality of dots is arranged to indicate the at least one first error checking bit so as to indicate the correctness of the first data bits by indicating the value for the at least one error checking bit that is equal to a check-sum of the values of the first data bits.

15. The medical system as in claim 8, wherein the surgical tool has a shaft and a plurality of markers arranged in a single ring extending around a circumference of the shaft so that at least one of the plurality of markers is viewable at any angle of rotation of the shaft about a longitudinal axis of the shaft, wherein the plurality of markers includes the first marker and a second marker, the second marker including a second identification feature which includes a second plurality of dots, the second plurality of dots arranged in a second pattern of dots so as to indicate values of second data bits and a value or values of at least one second error bit, wherein a value of a corresponding one of the second data bits and the at least one second error checking bit is determined by a presence or absence of a dot in each location of the predefined pattern of possible dot locations, wherein the values of the second data bits indicates an identity of the second marker, wherein the value or values of the at least one second error checking bit indicates one of an informational redundancy and correctness of the second data bits, wherein the identity of the second marker is different than the identity of the first marker, and wherein the processor is provided with information of a positional relationship between each of the first and second markers with respect to the surgical tool.

16. The medical system as in claim 15, wherein at least one of the plurality of markers includes at least one localizer feature that is shared with an adjacent one of the plurality of markers, wherein each of the at least one localizer feature is useful for localizing the marker which includes the localizer feature.

17. The medical system as in claim 16, wherein the identification and localizer features of the plurality of markers are arranged in a two-dimensional pattern.

18. The medical system as in claim 17, wherein the at least one localizer feature comprises a plurality of circles.

19. The medical system as in claim 17, wherein the at least one localizer feature comprises a plurality of corners.

20. The medical system as in claim 19, wherein the corners comprise saddle points.

21. The medical system as in claim 17, wherein individual of the plurality of markers includes three localizer features.

22. The medical system as in claim 17, wherein individual of the plurality of markers includes four localizer features.

23. The medical system as in claim 17, wherein individual of the plurality of markers includes four circles and a bar as localizer features.

24. The medical system as in claim 8, wherein the first marker is on a curved surface of the surgical tool, and wherein the first marker is fully visible in the image.