System and methods for intraoperative guidance feedback

• US 9,119,670 B2
• Filed: 10/31/2012
• Issued: 09/01/2015
• Est. Priority Date: 04/28/2010
• Status: Active Grant

• Current Assignee: 7D Surgical ULC (Orthofix Medical Inc.)
• Original Assignee: Ryerson University
• Inventors: Yang, Victor Xiao Dong, Standish, Beau Anthony, Mariampillai, Adrian Linus Dinesh, Leung, Michael Ka Kit
• Primary Examiner(s): Jung, Unsu
• Assistant Examiner(s): Lamprecht, Joel
• Application Number:
  US13/664,613
• Publication Number:
  US 20130060146A1
• Time in Patent Office: 1,035 Days
• Field of Search: 600/407, 600/410, 600/473, 703/11, 703/7
• US Class Current: 1/1
• CPC Class Codes: A61B 2017/00203 with speech control or speech recognition
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2057 Details of tracking cameras
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/363 Use of fiducial points
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/366 using projection of images directly onto the body
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/373 using light, e.g. by using optical scanners
• CPC Class Codes: A61B 2090/3945 Active visible markers, e.g. light emitting diodes
• CPC Class Codes: A61B 2090/397 electromagnetic other than visible, e.g. microwave
• CPC Class Codes: A61B 2090/3979 infrared
• CPC Class Codes: A61B 2090/3983 Reference marker arrangements for use with image guided surgery
• CPC Class Codes: A61B 2560/0475 Special features of memory means, e.g. removable memory cards
• CPC Class Codes: A61B 2576/02 specially adapted for a particular organ or body part
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 6/032 : Transmission computed tomography [CT]
• CPC Class Codes: A61B 6/5229 : combining image data of a patient, e.g. combining a functional image with an anatomical image
• CPC Class Codes: A61B 6/5247 : combining images from an ionising-radiation diagnostic technique and a non-ionising radiation diagnostic technique, e.g. X-ray and ultrasound
• CPC Class Codes: A61B 8/5238 : for combining image data of patient, e.g. merging several images from different acquisition modes into one image for image registration algorithms G06T7/30
• CPC Class Codes: A61B 90/30 : Devices for illuminating a surgical field, the devices having an interrelation with other surgical devices or with a surgical procedure
• CPC Class Codes: A61B 90/361 : Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/39 : Markers, e.g. radio-opaque or breast lesions markers
• CPC Class Codes: G01B 11/24 : for measuring contours or curvatures
• CPC Class Codes: G01B 11/245 : using a plurality of fixed, simultaneously operating transducers G01B11/2408 - G01B11/2425, G01B11/255 take precedence
• CPC Class Codes: G01B 11/25 : by projecting a pattern, e.g. one or more lines, moiré fringes on the object G01B11/255 takes precedence ; image analysis for depth or shape recovery G06T7/50

Abstract

Systems and methods for surgical guidance and image registration are provided, in which three-dimensional image data associated with an object or patient is registered to topological image data obtained using a surface topology imaging device. The surface topology imaging device may be rigidly attached to an optical position measurement system that also tracks fiducial markers on a movable instrument. The instrument may be registered to the topological image data, such that the topological image data and the movable instrument are registered to the three-dimensional image data. The three-dimensional image data may be CT or MRI data associated with a patient. The system may also co-register images pertaining to a surgical plan with the three-dimensional image data. In another aspect, the surface topology imaging device may be configured to directly track fiducial markers on a movable instrument. The fiducial markers may be tracked according to surface texture.

Claims

1. A surgical guidance system comprising:

a storage medium for storing pre-operative image data associated with a patient;

an integrated surface topology imaging and optical position measurement device comprising;

an optical projection device configured to project optical radiation onto an exposed surface of the patient, such that backscattered optical radiation is suitable for optical surface topology detection;

an optical source having a wavelength selected to illuminate a set of fiducial markers provided on a movable instrument;

two or more cameras, wherein at least one of said two or more cameras is configured for imaging the backscattered optical radiation, and wherein at least two of said two or more cameras are configured for imaging the set of fiducial markers when illuminated; and

a surgical guidance controller operatively connected to said integrated surface topology imaging and optical position measurement device and said storage medium, wherein said surgical guidance controller includes a processor configured to;

control said optical projection device to illuminate the exposed surface and obtain, based on one or more images from said at least one camera, topological image data associated with the exposed surface; and

control said optical source to illuminate the set of fiducial markers and obtain, from said two or more cameras, positional image data associated with said set of fiducial markers;

determine a position and an orientation of said movable instrument relative to said exposed surface; and

register said topological image data, and said position and orientation of said movable instrument to said pre-operative image data;

wherein said optical projection device, said optical source, and said two or more cameras are rigidly mounted on a frame, such that a fixed spatial relationship exists therebetween, thereby maintaining a fixed calibration of said system without requiring inter-operative recalibration.

2. The surgical guidance system according to claim 1 wherein said surgical guidance controller is further configured to register surgical plan data to said pre-operative image data and said topological image data, wherein said surgical plan data is stored on said storage medium.

3. The surgical guidance system according to claim 1 wherein said optical projection device is a structured light projection device.

4. The surgical guidance system according to claim 1 wherein said optical projection device and said at least one camera is configured for obtaining said topological image data corresponding to two or more spectral bands.

5. The surgical guidance system according to claim 1 wherein said surgical guidance controller includes a processor configured to:

process said topological image data to identify clutter according to a local texture of said topological image data; and

remove said clutter from said topological image data.

6. The surgical guidance system according to claim 5 wherein said processor is configured to process said topological image data by:

obtaining, based on one or more images from said at least one camera, topological image data associated with said exposed surface at two or more spectral bands; and

processing said topological image data to identify clutter according to a local texture of said topological image data by identifying image data according to spectral criteria.

7. The surgical guidance system according to claim 6 wherein said spectral criteria comprises an acceptable range of a ratio obtained by dividing a first signal from a first spectral band by a second signal obtained from a second spectral band.

8. The surgical guidance system according to claim 7 wherein said spectral bands comprise two or more of red, blue and green spectral bands.

9. The surgical guidance system according to claim 6 wherein said topological image data comprises a mesh or points representing a portion of said exposed surface, and wherein said processor is configured to identify image data according to spectral criteria by:

determining, at each point in said mesh, a ratio obtained by dividing a first signal from a first spectral band by a second signal obtained from a second spectral band;

generating a set of seed points on said mesh;

calculating an averaged value of said ratio within a region surrounding each seed point; and

accepting or rejecting said seed points according to a criteria based on a value of said averaged value of said ratio.

10. The surgical guidance system according to claim 5 wherein said processor is configured to process said topological image data by identifying image data according to criteria based on surface roughness.

11. The surgical guidance system according to claim 10 wherein said topological image data comprises a mesh or points representing a portion of said exposed surface, and wherein said processor is configured to identify image data according to criteria based on surface roughness by:

calculating a curvature at each point in said mesh;

generating a set of seed points on said mesh;

calculating an averaged curvature value within a region surrounding each seed point; and

accepting or rejecting said seed points according to a criteria based on a value of said averaged curvature value.

12. The surgical guidance system according to claim 1 wherein said processor is further configured to alternate between:

controlling said optical projection device to illuminate the exposed surface and obtain, based on one or more images from said at least one camera, the topological image data associated with the exposed surface; and

between

controlling said optical source to illuminate the set of fiducial markers and obtain, from said two or more cameras, the positional image data associated with said set of fiducial marker.

13. The surgical guidance system according to claim 1 wherein an optical filter is associated with said at least one camera for transmitting a wavelength or wavelength band associated with said optical projection device.

14. The surgical guidance system according to claim 13 wherein said optical filter is further configured for transmitting a wavelength or wavelength band associated with said optical source.

15. The surgical guidance system according to claim 13 wherein said optical filter is a first optical filter, said system further comprising a second optical filter associated with said at least one camera, and such that said first optical filter and said second optical filter are removably positionable in front of said at least one camera.

16. The surgical guidance system according to claim 1 wherein said two or more cameras comprise at least one camera configured for imaging the backscattered optical radiation, and at least two additional cameras configured for imaging the set of fiducial markers when illuminated.

17. The surgical guidance system according to claim 1 wherein said integrated surface topology imaging and optical position measurement device are configured such that a first focal region associated with a depth of field of said optical projection device overlaps with a second focal region associated with a depth of field of said two or more cameras.

18. A method of registering surface topological image data to preoperative image data using an integrated system comprising a surface topology imaging device and an optical position measurement device;

the surface topology imaging device comprising;

an optical projection device configured to project optical radiation onto an exposed surface of a patient, such that backscattered optical radiation is suitable for optical surface topology detection; and

one or more first cameras configured for imaging the backscattered optical radiation wherein the optical projection device;

the optical position measurement device comprising;

an optical source having a wavelength selected to illuminate a set of fiducial markers provided on a movable instrument; and

two or more second cameras for imaging the set of fiducial markers when illuminated;

wherein the surface topology imaging device and the optical position measurement device are rigidly mounted on a frame such that a fixed spatial relationship exists therebetween;

the method comprising;

obtaining pre-operative image data associated with a patient;

obtaining pre-determined calibration data for relating the coordinate system of the optical position measurement device and the coordinate system of the surface topology imaging device;

optically scanning the exposed surface of the patient with the optical projection device and obtaining, based on one or more images from the one or more first cameras, topological image data associated with the exposed surface;

illuminating the set of fiducial markers by powering the optical source and obtaining, from the second cameras, optical images of the set of fiducial markers;

processing the optical images to determine a position and orientation of the movable instrument relative to the exposed surface, based on the pre-determined calibration data; and

registering the topological image data, and the position and orientation of the movable instrument, to the pre-operative image data.

19. The method according to claim 18 wherein the pre-determined calibration data between the surface topology imaging device and the optical position measurement device is obtained using a method comprising the steps of:

acquiring the center positions of at least three fiducial markers using the optical position measurement device;

acquiring the center positions of the same fiducials markers using the surface topology imaging device; and

calculating the transformation between these two point sets.

20. The method according to claim 19 further comprising the steps of:

obtaining a three-dimensional surface model of the fiducial markers;

determining center positions of the fiducial markers;

wherein the step of processing the topological image data to determine the center positions of fiducial markers comprises the steps of;

identifying, in the topological image data, surfaces corresponding to the fiducial markers; and

determining center locations of the fiducial markers in the topological image data.

21. The method according to claim 20 wherein the fiducial markers are spherical in shape, and wherein the center location of a given fiducial marker is determined by:

identifying a plurality of surface normals associated with a surface of said given fiducial marker;

backprojecting the surface normals beneath the surface of the given fiducial marker; and

processing the backprojected surface normals to identify the center location.

22. The method according to claim 21 wherein processing the backprojected surface normals comprises identifying the center location based on average point of closest approach between the surface normals.

23. The method according to claim 22 wherein the step of identifying, in the topological image data, surfaces corresponding to the fiducial markers comprises spectrally filtering the topological image data to identify the surfaces corresponding to the fiducial markers.

24. The method according to claim 18 wherein further comprising registering surgical plan data to the pre-operative image data and the topological image data.

25. The method system according to claim 18 wherein the optical projection device is a structured light projection device.

26. The method according to claim 18 wherein the optical projection device and the one or more first cameras are configured for obtaining the topological image data corresponding to two or more spectral bands.

27. The method according to claim 18 further comprising:

processing the topological image data to identify clutter according to a local texture of the topological image data; and

removing the clutter from the topological image data.

28. The method according to claim 27 wherein the topological image data is processed to identify clutter by:

obtaining, based on one or more images from the one or more first cameras, topological image data associated with the exposed surface at two or more spectral bands; and

processing the topological image data to identify clutter according to a local texture of the topological image data by identifying image data according to spectral criteria.

29. The method according to claim 28 wherein the spectral criteria comprises an acceptable range of a ratio obtained by dividing a first signal from a first spectral band by a second signal obtained from a second spectral band.

30. The method according to claim 29 wherein the spectral bands comprise two or more of red, blue and green spectral bands.

31. The method according to claim 28 wherein the topological image data comprises a mesh or points representing a portion of the exposed surface, and wherein the step of identifying image data according to spectral criteria is performed by:

determining, at each point in the mesh, a ratio obtained by dividing a first signal from a first spectral band by a second signal obtained from a second spectral band;

generating a set of seed points on the mesh;

calculating an averaged value of the ratio within a region surrounding each seed point; and

accepting or rejecting the seed points according to a criteria based on a value of the averaged value of the ratio.

32. The method according to claim 27 wherein further comprising processing the topological image data by identifying image data according to criteria based on surface roughness.

33. The method according to claim 32 wherein the topological image data comprises a mesh or points representing a portion of the exposed surface, and wherein the step of identifying image data according to criteria based on surface roughness is performed by:

calculating a curvature at each point in the mesh;

generating a set of seed points on the mesh;

calculating an averaged curvature value within a region surrounding each seed point; and

accepting or rejecting the seed points according to a criteria based on a value of the averaged curvature value.

34. A surgical guidance system comprising:

a storage medium for storing pre-operative image data associated with a patient;

a surface topology imaging device comprising;

an optical projection device configured to project optical radiation onto an exposed surface of the patient and onto a set of fiducial markers provided on a movable instrument, such that backscattered optical radiation is suitable for optical surface topology detection; and

one or more cameras configured for imaging the backscattered optical radiation; and

a surgical guidance controller operatively connected to said surface topology imaging device and said storage medium, wherein said surgical guidance controller includes a processor configured to;

control said optical projection device to illuminate the exposed surface and the set of fiducial markers and to obtain, based on one or more images from said one or more cameras, topological image data associated with the exposed surface and the set of fiducial markers;

determine a position and an orientation of said movable instrument relative to said exposed surface; and

register said topological image data, and said position and orientation of said movable instrument to said pre-operative image data;

wherein said optical projection device and said one or more cameras are rigidly mounted on a frame, such that a fixed spatial relationship exists therebetween, thereby maintaining a fixed calibration of said system without requiring inter-operative recalibration.

35. A method of registering a topological image data to pre-operative image data for surgical guidance, wherein the topological image data is obtained by a surface topology imaging device, the method comprising the steps of:

storing the pre-operative image data associated with a patient;

controlling the surface topology imaging device to;

project optical radiation onto an exposed surface of the patient and a set of fiducial markers provided on a movable instrument;

obtain, based on one or more images from one or more cameras, topological image data associated with the exposed surface and the fiducial markers;

processing the topological image data to;

identify, in the topological image data, surfaces corresponding to the fiducial markers; and

determine, based on the identified surfaces, a position and orientation of the movable instrument relative to the exposed surface; and

registering the topological image data, and the position and orientation of the movable instrument, to the pre-operative image data.

36. The method according to claim 35 further comprising the steps of:

obtaining a three-dimensional surface model of the movable instrument and the fiducial markers;

determining an orientation axis of the movable instrument and a position corresponding to a tip of the movable instrument;

determining predetermined center positions of the fiducial markers relative to the orientation axis and the tip; and

wherein the step of processing the topological image data to determine the position and orientation of the movable instrument relative to the exposed surface comprises the steps of;

determining center locations of the fiducial markers in the topological image data; and

calculating a transformation mapping the center locations to the predetermined center positions; and

applying the transformation to register the three-dimensional surface model to the topological image data.

37. The method according to claim 36 wherein the fiducial markers are spherical in shape, and wherein the center location of a given fiducial marker is determined by:

identifying a plurality of surface normals associated with a surface of said given fiducial marker;

backprojecting the surface normals beneath the surface of the given fiducial marker; and

processing the backprojected surface normals to identify the center location.

38. The method according to claim 37 wherein processing the backprojected surface normals comprises identifying the center location based on average point of closest approach between the surface normals.

39. The method according to claim 36 wherein the step of identifying, in the topological image data, surfaces corresponding to the fiducial markers comprises spectrally filtering the topological image data to identify the surfaces corresponding to the fiducial markers.

40. The method according to claim 36 wherein the surface topology backscattered radiation image acquisition system comprises a structured light projection device.