System and method for intra-operative, image-based, interactive verification of a pre-operative surgical plan

• US 6,301,495 B1
• Filed: 04/27/1999
• Issued: 10/09/2001
• Est. Priority Date: 04/27/1999
• Status: Active Grant

• Current Assignee: International Business Machines Corporation
• Original Assignee: International Business Machines Corporation
• Inventors: Gueziec, Andre, Kalvin, Alan David
• Primary Examiner(s): MANUEL, GEORGE C
• Application Number:
  US09/299,643
• Time in Patent Office: 896 Days
• Field of Search: 600/407, 600/410, 600/411, 600/414, 600/416, 600/417, 600/425, 600/426, 600/427, 382/128, 382/132
• US Class Current: 600/407
• CPC Class Codes: A61B 17/175 for preparing the femur for hip prosthesis insertion
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 2090/3937 Visible markers
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 6/00 Apparatus or devices for radiation diagnosis; Apparatus or devices for radiation diagnosis combined with radiation therapy equipment instruments measuring radiation intensity for application in the field of nuclear medicine, e.g. in vivo counting, G01T1/161; apparatus for taking X-ray photographs G03B42/02
• CPC Class Codes: A61B 6/5235 combining images from the same or different ionising radiation imaging techniques, e.g. PET and CT
• CPC Class Codes: A61B 90/36 Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/39 Markers, e.g. radio-opaque or breast lesions markers
• CPC Class Codes: G06T 3/00 Geometric image transformations in the plane of the image

Abstract

A system and method for intra-operatively providing a surgeon with visual evaluations of possible surgical outcomes ahead of time, and generating simulated data, includes a medical imaging camera, a registration device for registering data to a physical space, and to the medical imaging camera, and a fusion mechanism for fusing the data and the images to generate simulated data. The simulated data (e.g., such as augmented X-ray images) is natural and easy for a surgeon to interpret. In an exemplary implementation, the system preferably includes a data processor which receives a three-dimensional surgical plan or three-dimensional plan of therapy delivery, one or a plurality of two-dimensional intra-operative images, a three-dimensional model of pre-operative data, registration data, and image calibration data. The data processor produces one or a plurality of simulated post-operative images, by integrating a projection of a three-dimensional model of pre-operative data onto one or a plurality of two-dimensional intra-operative images.

Claims

1. A system for generating simulated data, comprising:

a medical imaging camera for generating images;

a registration device for registering data to a physical space, and to said medical imaging camera; and

a fusion mechanism for fusing said data and said images to generate simulated data,

wherein said simulated data comprises multiple alternative post-operative conditions.

2. The system according to claim 1, wherein said data comprises pre-operative images and said images comprise intra-operative images.

3. The system as in claim 2, wherein the pre-operative data comprises data of a surgical plan including a position of a component, and a three-dimensional shape of said component.

4. The system as in claim 3, wherein said component comprises an implant for a patient.

5. The system according to claim 1, wherein said data comprises a two-dimensional image.

6. The system according to claim 1, wherein said simulated data comprises simulated post-operative images.

7. The system according to claim 1, wherein said fusion mechanism generates said simulated data while a surgery is being performed.

8. The system as in claim 1, wherein said medical imaging camera comprises a two-dimensional (2D) X-ray camera.

9. The system as in claim 1, wherein said fusion mechanism comprises a data processor.

10. The system according to claim 1, wherein said fusion mechanism fuses said data with said images by integrating a two-dimensional projection of a silhouette of a three-dimensional implant model in an X-ray image.

11. The system according to claim 10, wherein said fusion mechanism uses a calibration of the X-ray image, to determine a center of perspective whose location represents an estimate of a location of an X-ray source,

said center of perspective being used to compute silhouette curves on the three-dimensional implant model.

12. The system according to claim 11, wherein said silhouette curves are such that rays emanating from the center of perspective and tangent to the three-dimensional model meet the three-dimensional implant model on a silhouette curve.

13. The system according to claim 1, wherein said simulated data are formed by fusing intra-operative images with registered pre-operative data.

14. A system for generating simulated data, comprising:

a medical imaging camera for generating images;

a registration device for registering data to a physical space, and to said medical imaging camera;

a fusion mechanism for fusing said data and said images to generate simulated data; and

another medical imaging camera for collecting said data,

wherein said medical imaging camera collects said images.

15. The system as in claim 14, wherein said another imaging camera comprises an X-ray, computed tomography (CT) scanner.

16. A system for generating simulated post-operative data, comprising:

an imaging camera;

a registration device for registering data to a physical space, and to the imaging camera; and

a fusion mechanism for fusing said data and intra-operative images to generate simulated data,

wherein said simulated data comprises multiple alternative post-operative conditions.

17. The system according to claim 16, wherein data comprises pre-operative images and other pre-operative data, and wherein said simulated data comprises simulated post-operative images.

18. The system as in claim 17, wherein the pre-operative data comprises data of a surgical plan including a position of a component, and a three-dimensional shape of said component.

19. The system as in claim 18, wherein said component comprises an implant for a patient.

20. The system according to claim 16, wherein said fusion mechanism generates said simulated data while a surgery is being performed.

21. The system as in claim 16, wherein said imaging camera comprises a two-dimensional (2D) X-ray camera.

22. The system as in claim 16, wherein said fusion mechanism comprises a data processor.

23. A system for generating simulated post-operative data, comprising:

an imaging camera;

a registration device for registering data to a physical space, and to the imaging camera;

a fusion mechanism for fusing said data and intra-operative images to generate simulated data;

a calibration mechanism for calibrating said imaging camera; and

another imaging camera for collecting said pre-operative images, wherein said imaging camera collects said intra-operative images,

wherein data comprises pre-operative images and other pre-operative data, and wherein said simulated data comprises simulated post-operative images.

24. The system as in claim 23, wherein said another imaging camera comprises an X-ray, computed tomography (CT) scanner.

25. A system for providing intra-operative visual evaluations of potential surgical outcomes, using medical images, comprising:

a first medical imaging camera for collecting pre-operative images;

a second medical imaging camera for collecting intra-operative images, while a surgery is being performed;

a registration mechanism for registering said pre-operative images and other pre-operative data to a physical space, and to said second medical imaging camera; and

a fusion mechanism for fusing said pre-operative data and said intra-operative images to generate simulated post-operative images.

26. A system for providing advice to a surgeon without requiring physical manipulation of objects inside and in a vicinity of a patient, said system comprising:

a manipulation device for manipulating one or a plurality of virtual objects;

a registration mechanism for registering the one or plurality of virtual objects to a physical space, and to an imaging camera; and

a fusion mechanism for fusing the one or plurality of virtual objects with an inter-operative image,

wherein said fusion mechanism generates images representing multiple simulated alternative post-operative conditions.

27. A robotically-assisted surgical system, comprising:

a surgical robot;

a first medical imaging camera for collecting pre-operative data;

a second medical imaging camera for collecting intra-operative images;

a registration mechanism for registering said pre-operative data to the surgical robot, and to the second medical imaging camera; and

a fusion mechanism for fusing said pre-operative data and said intra-operative images to generate simulated data.

28. A system for comparing a plurality of methods for registering pre-operative data with a physical space, comprising:

a first medical imaging device for collecting pre-operative data;

a second medical imaging device for collecting intra-operative images;

an image registration mechanism for registering the second medical imaging device to a physical space;

a plurality of pre-operative-data-to-physical-space registration mechanisms for registering said pre-operative data to the physical space; and

a fusion mechanism for fusing said pre-operative data, that are registered using the plurality pre-operative-data-to-physical-space registration mechanisms, and said intra-operative images, to generate new images.

29. A system for assisting a surgeon in a registration of pre-operative data with a physical space, comprising:

means for collecting pre-operative data;

means for collecting intra-operative images;

an image registration mechanism for registering the intra-operative image collecting means to a physical space;

a plurality of pre-operative-data-to-physical-space registration mechanisms for registering said pre-operative data to the physical space;

a fusion mechanism for fusing said pre-operative data, that are registered using the plurality of pre-operative-data-to-physical-space registration mechanisms, and said intra-operative images, to generate new images; and

a constraint mechanism for specifying positional constraints on a three-dimensional position of a physical object that is moved during surgery,

wherein said new images comprise images representing multiple alternative post-operative conditions.

30. A system for assisting a surgeon in a registration of pre-operative data with a physical space, comprising:

means for collecting pre-operative data;

means for collecting intra-operative images;

an image registration mechanism for registering the intra-operative image collecting means to a physical space;

a plurality of pre-operative-data-to-physical-space registration mechanisms for registering said pre-operative data to the physical space;

a fusion mechanism for fusing said pre-operative data, that are registered using the plurality of pre-operative-data-to-physical-space registration mechanisms, and said intra-operative images, to generate new images;

a constraint mechanism for specifying positional constraints on a three-dimensional position of a physical object that is moved during surgery; and

a refinement mechanism for refining a surgical plan by refining the constraints on the three-dimensional position of a physical object that is moved during surgery.

31. A robotically-assisted surgical system, comprising:

a surgical robot;

an image registration mechanism for registering a medical imaging camera to a physical space;

a plurality of pre-operative-data-to-physical-space registration mechanisms for registering pre-operative data to the physical space;

a fusion mechanism for fusing said pre-operative data, that are registered using the plurality of pre-operative-data-to-physical-space registration mechanisms, and intra-operative images produced by said imaging camera, to generate new images; and

a refinement mechanism for refining a surgical plan by refining constraints on a three-dimensional position of a physical object that is moved during surgery,

wherein said new images comprise images representing multiple alternative post-operative conditions.

32. The system according to claim 31, further comprising:

a constraint mechanism for specifying positional constraints on the three-dimensional position of the physical object that is moved during surgery.

33. The system according to claim 31, wherein said surgical robot includes an internal calibration system, said system further comprising:

a graphical warning mechanism for providing a visual representation of an outcome of a surgical plan, and potential failure of the surgical plan resulting from the internal failure of the calibration system of said robot.

34. A system for assessing accuracy of a proposed execution of a pre-operative surgical plan, comprising:

an image device for obtaining intra-operative images of an anatomical feature of interest; and

a data processor for computing a projection of a pre-operative three-dimensional model of an implant or planned trajectory onto the intra-operative image,

wherein said intra-operative image comprises an image representing multiple alternative post-operative conditions.

35. The system according to claim 34, further comprising a display system for displaying an output of said data processor while a surgery is on-going.

36. A method of generating simulated data, comprising:

registering data to a physical space, and to a medical imaging camera; and

fusing said data and said images to generate simulated data,

wherein said simulated data comprises multiple alternative post-operative conditions.

37. A method of generating simulated post-operative data, comprising:

collecting pre-operative data and collecting intra-operative images;

registering said pre-operative data to a physical space, and to a medical imaging camera; and

fusing said pre-operative data and said intra-operative images to generate simulated post-operative data,

wherein said simulated post-operative data comprises multiple alternative post-operative conditions.

38. A method of generating simulated post-operative data, comprising:

collecting pre-operative data and collecting intra-operative images;

registering said pre-operative data to a physical space, and to a medical imaging camera;

fusing said pre-operative data and said intra-operative images to generate simulated post-operative data; and

calibrating the medical imaging camera,

wherein the pre-operative data comprises data of a surgical plan including a position of a component, and a three-dimensional shape of said component.

39. The method of claim 38, wherein said component of said surgical plan includes an implant position.

40. A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform a method for computer-implemented generating of simulated data, comprising:

registering data to a physical space, and to a medical imaging camera; and

fusing said data and said images to generate simulated data,

wherein said simulated data comprises multiple alternative post-operative conditions.

41. A signal-bearing medium tangibly embodying a program of machine-readable instructions executable by a digital processing apparatus to perform a method for computer-implemented generating simulated post-operative data, comprising:

collecting pre-operative data and collecting intra-operative images;

registering said pre-operative data to a physical space, and to a medical imaging camera; and

fusing said pre-operative data and said intra-operative images to generate simulated post-operative data,

wherein said simulated post-operative data comprises multiple alternative post-operative conditions.

42. A system for generating simulated data, comprising:

a medical imaging camera for generating images;

a registration device for registering data to a physical space, and to said medical imaging camera; and

a fusion mechanism for fusing said data and said images to generate simulated data,

wherein said fusion mechanism fuses by projecting a silhouette curve of said data by considering in turn each new pixel, determining a line in three-dimensions corresponding to that pixel by image calibration, computing a distance from a line to the simhouette curve, and assigning a pixel gray-scale value depending on the distance.

43. The system according to claim 42, wherein said fusion mechanism assigns pixel gray-scale values corresponding to a distance,

wherein if the distance is less than a first predetermined value, then the gray-scale value is set to a first predetermined number.

44. The system according to claim 43, wherein if the distance is less than a second predetermined value, then the gray-scale value is set to a second predetermined number larger than said first predetermined number.

45. The system according to claim 44, wherein if the distance is less than a third predetermined value greater than said first and second predetermined values, then the gray-scale value is set to a third predetermined number larger than said first and second predetermined numbers.

46. The system according to claim 45, wherein if the distance is greater than or equal to said third predetermined value, then the gray-scale value is not modified for projecting the silhouette curves.