Methods and systems for tracking and guiding sensors and instruments

• US 9,561,019 B2
• Filed: 03/07/2013
• Issued: 02/07/2017
• Est. Priority Date: 03/07/2012
• Status: Active Grant

• Current Assignee: Ziteo, Inc.
• Original Assignee: Ziteo, Inc.
• Inventors: Mihailescu, Lucian, Negut, Victor Arie
• Primary Examiner(s): Leubecker, John P
• Assistant Examiner(s): TORRES DIAZ, ARNALDO
• Application Number:
  US13/789,143
• Publication Number:
  US 20130237811A1
• Time in Patent Office: 1,433 Days
• US Class Current: 1/1
• CPC Class Codes: A61B 2090/363 Use of fiducial points
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/064 using markers A61B5/062 takes precedence
• CPC Class Codes: A61B 5/065 Determining position of the probe employing exclusively positioning means located on or in the probe, e.g. using position sensors arranged on the probe
• CPC Class Codes: A61B 6/4258 for detecting non x-ray radiation, e.g. gamma radiation A61B6/037 takes precedence
• CPC Class Codes: A61B 6/4405 the apparatus being movable or portable, e.g. handheld or mounted on a trolley
• CPC Class Codes: A61B 6/4417 related to combined acquisition of different diagnostic modalities
• CPC Class Codes: A61B 8/13 Tomography A61B8/10, A61B8/12 take precedence; tomography for radiation diagnosis A61B6/02
• CPC Class Codes: A61B 8/4245 involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient
• CPC Class Codes: A61B 8/4254 using sensors mounted on the probe
• CPC Class Codes: A61B 8/4427 Device being portable or laptop-like
• CPC Class Codes: A61B 8/4438 Means for identifying the diagnostic device, e.g. barcodes
• CPC Class Codes: A61B 8/4444 related to the probe
• CPC Class Codes: A61B 8/462 characterised by constructional features of the display
• CPC Class Codes: A61B 8/463 characterised by displaying multiple images or images and diagnostic data on one display composite display in devices for short-range imaging with acoustic waves G01S7/52074
• CPC Class Codes: A61B 8/483 involving the acquisition of a 3D volume of data
• CPC Class Codes: A61B 8/485 involving measuring strain or elastic properties
• CPC Class Codes: A61B 8/5238 for combining image data of patient, e.g. merging several images from different acquisition modes into one image for image registration algorithms G06T7/30
• CPC Class Codes: A61B 8/5253 combining overlapping images, e.g. spatial compounding
• CPC Class Codes: A61B 8/5261 : combining images from different diagnostic modalities, e.g. ultrasound and X-ray
• CPC Class Codes: A61B 8/5269 : involving detection or reduction of artifacts for image enhancement algorithms G06T5/00
• CPC Class Codes: A61B 90/361 : Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61M 37/0069 : Devices for implanting pellets, e.g. markers or solid medicaments for introducing of radioactive sources for interstitial radiation therapy, i.e. brachytherapy A61N5/1027
• CPC Class Codes: G01S 15/8936 : using transducers mounted for mechanical movement in three dimensions
• CPC Class Codes: G01S 15/899 : Combination of imaging systems with ancillary equipment
• CPC Class Codes: G01S 15/8993 : Three dimensional imaging systems
• CPC Class Codes: G01S 17/66 : Tracking systems using electromagnetic waves other than radio waves
• CPC Class Codes: G01S 17/86 : Combinations of lidar systems with systems other than lidar, radar or sonar, e.g. with direction finders
• CPC Class Codes: G01S 17/894 : 3D imaging with simultaneous measurement of time-of-flight at a 2D array of receiver pixels, e.g. time-of-flight cameras or flash lidar
• CPC Class Codes: G01S 7/52065 : Compound scan display, e.g. panoramic imaging
• CPC Class Codes: G01S 7/52077 : with means for elimination of unwanted signals, e.g. noise or interference
• CPC Class Codes: G01S 7/52079 : Constructional features constructional features of transducers B06B; mounting transducers G10K11/00; constructional features of ultrasonic medical diagnostic devices A61B8/44

Abstract

A shared-housing ultrasound transducer and machine-vision camera system is disclosed for registering the transducer'"'"'s x, y, z position in space and pitch, yaw, and roll orientation with respect to an object, such as a patient'"'"'s body. The position and orientation are correlated with transducer scan data, and scans of the same region of the object are compared in order to reduce ultrasound artifacts and speckles. The system can be extended to interoperative gamma probes or other non-contact sensor probes and medical instruments. Methods are disclosed for computer or remote guiding of a sensor probe or instrument with respect to saved positions and orientations of the sensor probe.

Claims

1. A spatial registration apparatus, comprising:

a medical instrument or sensor probe;

a camera rigidly connected with the medical instrument or sensor probe;

a fiducial marker affixed to an object of interest, the object of interest being different from the medical instrument or sensor probe and different from the camera;

at least one first processor operatively coupled with a first memory, the first memory having first instructions for execution by the at least one first processor, wherein the first memory stores a relative location and orientation between the medical instrument or sensor probe and the rigidly connected camera, wherein the first instructions, when executed by the at least one first processor, cause the at least one first processor to determine a pose of the camera with respect to the fiducial marker using an image of the fiducial marker captured by the camera and then transform the pose, using the stored relative location and orientation between the medical instrument or sensor probe and the rigidly connected camera, to determine a first spatial position and orientation of the medical instrument or sensor probe with respect to the fiducial marker, the at least one first processor associating a first scan of the object of interest from the medical instrument or sensor probe with the first spatial position and orientation of the medical instrument or sensor probe to create a first spatially registered scan, the first scan being time synchronized with the first spatial position and orientation of the medical instrument or sensor probe;

a network; and

at least one second processor operatively coupled with a second memory, the second memory having second instructions for execution by the at least one second processor, the second instructions, when executed by the at least one second processor, cause the at least one second processor to derive visualization data from the first spatially registered scan from the medical instrument or sensor probe using the first spatial position and orientation of the medical instrument or sensor probe with respect to the fiducial marker, and display the visualization data, on a first display, to a first user.

2. The apparatus of claim 1, wherein the second instructions, when executed by the at least one second processor, cause the at least one second processor to produce a proposed spatial position and orientation of the medical instrument or sensor probe with respect to the fiducial marker.

3. The apparatus of claim 2, wherein the first instructions, when executed by the at least one first processor, cause the at least one first processor to display, on a second display, the visualization data and guidance elements indicating the proposed spatial position and orientation of the medical instrument or sensor probe with respect to the fiducial marker, to a second user.

4. The apparatus of claim 2, further comprising a mock-up instrument, a second camera rigidly connected with the mock-up instrument, and a second fiducial marker, wherein the second instructions, when executed by the at least one second processor, cause the at least one second processor to determine a second pose of the second camera with respect to the second fiducial marker using a second image of the second fiducial marker captured by the second camera and then transform the second pose, using a second relative location and orientation between the mock-up instrument and the second camera, the second relative location and orientation stored in the second memory, and wherein the production of the proposed spatial position and orientation by the at least one second processor is based on the second pose.

5. The apparatus of claim 2, further comprising an input device and a remote tracking system configured to track a position and orientation of the input device, wherein the production of the proposed spatial position and orientation by the at least one second processor is based on the tracked position and orientation of the input device, wherein the input device is one of a joystick, a mock-up instrument, and a computer mouse, and wherein the remote tracking system is one of a magnetic tracker, an electromagnetic tracker, an optical tracker, an inertial measurement unit, and a ranging camera.

6. The apparatus of claim 1, wherein the first instructions, when executed by the at least one first processor, cause the at least one first processor to render a three-dimensional model using the first spatially registered scan and a second spatially registered scan, the first spatially registered scan being from a first scanning plane of the medical instrument or sensor probe and the second spatially registered scan being from a second scanning plane of the medical instrument or sensor probe, the first scanning plane being different from the second scanning plane.

7. The apparatus of claim 6, wherein the first instructions, when executed by the at least one first processor, cause the at least one first processor to filter out a speckle or an artifact of the first spatially registered scan based on the second spatially registered scan.

8. The apparatus of claim 1, wherein a camera is configured to emit a signal and detect a reflection of the signal from the object of interest, and wherein the first instructions, when executed by the at least one first processor, cause the at least one first processor to determine a distance from the camera to at least three points on a surface of the object of interest based on the reflection of the signal.

9. The apparatus of claim 8, wherein the first instructions, when executed by the at least one first processor, cause the at least one first processor to determine a model of the surface of the object of interest based on the distance from the camera to the at least three points.

10. The apparatus of claim 8 wherein the first instructions, when executed by the at least one first processor, cause the at least one first processor to determine a second pose of the camera with respect to the object of interest based on the distance from the camera to the at least three points.

11. The apparatus of claim 1, wherein the second instructions, when executed by the at least one second processor, cause the at least one second processor to receive, via the network, the first spatially registered scan.

12. A spatial registration method, comprising:

storing, in a first memory, a relative location and orientation between a medical instrument or sensor probe and a camera rigidly connected with the medical instrument or sensor probe;

determining, by at least one first processor, a pose of the camera with respect to a fiducial marker using an image of the fiducial marker captured by the camera, the fiducial marker being affixed to an object of interest, the object of interest being different from the medical instrument or sensor probe and different from the camera;

transforming, by at least one first processor, the pose, using the stored relative location and orientation between the medical instrument or sensor probe and the rigidly connected camera, to determine a first spatial position and orientation of the medical instrument or sensor probe with respect to the fiducial marker;

associating, by at least one first processor, a first scan from the medical instrument or sensor probe with the first spatial position and orientation of the medical instrument or sensor probe to create a first spatially registered scan, the first scan being time synchronized with the first spatial position and orientation of the medical instrument or sensor probe;

deriving, by at least one second processor, visualization data from the first spatially registered scan from the medical instrument or sensor probe using the determined current spatial position and orientation of the medical instrument or sensor probe with respect to the fiducial marker; and

displaying the visualization data, on a first display, to a first user.

13. The method of claim 12, further comprising producing, by the at least one second processor, a proposed spatial position and orientation of the medical instrument or sensor probe with respect to the fiducial marker.

14. The method of claim 13, further comprising displaying, on a second display, the visualization data and guidance elements indicating the proposed spatial position and orientation of the medical instrument or sensor probe with respect to the fiducial marker, to a second user.

15. The method of claim 13, further comprising:

determining, by the at least one second processor, a second pose of a second camera with respect to a second fiducial marker using a second image of the second fiducial marker captured by the second camera, the second camera being rigidly connected with a mock-up instrument; and

transforming, by the at least one second processor, the second pose, using a second relative location and orientation between the mock-up instrument and the second camera, wherein the producing of the proposed spatial position and orientation by the at least one second processor is based on the second pose.

16. The method of claim 13, further comprising tracking, by a remote tracking system, a position and orientation of an input device, wherein the producing of the proposed spatial position and orientation by the at least one second processor is based on the tracked position and orientation of the input device, wherein the input device is one of a joystick, a mock-up instrument, and a computer mouse, and wherein the remote tracking system is one of a magnetic tracker, an electromagnetic tracker, an optical tracker, an inertial measurement unit, and a ranging camera.

17. The method of claim 12, further comprising rendering, by the at least one first processor, a three-dimensional model using the first spatially registered scan and a second spatially registered scan, the first spatially registered scan being from a first scanning plane of the medical instrument or sensor probe and the second spatially registered scan being from a second scanning plane of the medical instrument or sensor probe, the first scanning plane being different from the second scanning plane.

18. The method of claim 17, further comprising filtering, by the at least one first processor, out a speckle or an artifact of the first spatially registered scan based on the second spatially registered scan.

19. The method of claim 12, further comprising:

emitting, by the camera, a signal;

detecting, by the camera, a reflection of the signal from the object of interest; and

determining, by the at least one first processor, a distance from the camera to at least three points on a surface of the object of interest based on the reflection of the signal.

20. The method of claim 19, further comprising determining, by the at least one first processor, a model of the surface of the object of interest based on the distance from the camera to the at least three points.

21. The method of claim 19, further comprising determining, by the at least one first processor, a second pose of the camera with respect to the object of interest based on the distance from the camera to the at least three points.

22. The method of claim 12, further comprising receiving by the at least one second processor, via a network, the associated first spatially registered scan.

23. A hand-held spatial registration apparatus, comprising:

a fiducial marker affixed to an object of interest;

a medical instrument or sensor probe that is different from the object of interest;

a camera connected with the medical instrument or sensor probe and configured to capture images of the fiducial marker indicating distances between the camera and the fiducial marker;

a memory storing instructions and a relative location and orientation between the medical instrument or sensor probe and the rigidly connected camera; and

a processor operatively coupled with the memory and configured to execute the instructions to;

determine a first pose of the camera with respect to the fiducial marker based on a first image of the fiducial marker captured by the camera,

transform the first pose, using the stored relative location and orientation between the medical instrument or sensor probe and the rigidly connected camera, to determine a first spatial position and orientation of the medical instrument or sensor probe with respect to the fiducial marker,

generate a first spatially registered scan by associating a first scan from the medical instrument or sensor probe with the first pose,

determine a second pose of the camera with respect to the fiducial marker based on the second image of the fiducial marker captured by the camera,

transform the second pose, using the stored relative location and orientation between the medical instrument or sensor probe and the rigidly connected camera, to determine a second spatial position and orientation of the medical instrument or sensor probe with respect to the fiducial marker, the second spatial position and orientation being different from the first spatial position and orientation,

generate a second spatially registered scan by associating a second scan from the medical instrument or sensor probe with the second pose, and

render a three-dimensional model using the first spatially registered scan and the second spatially registered scan, the first spatially registered scan being from a first scanning plane of the medical instrument or sensor probe, the second spatially registered scan being from a second scanning plane of the medical instrument or sensor probe, the first scanning plane being different from the second scanning plane.

24. The apparatus of claim 23, wherein the processor is further configured to execute the instructions to display, on a display, the three-dimensional model and guidance elements indicating a third spatial position and orientation of the medical instrument or sensor probe with respect to the fiducial marker, the third spatial position and orientation being different from the first spatial position and orientation and the second spatial position and orientation.