IMAGE-GUIDED SURGERY WITH SURFACE RECONSTRUCTION AND AUGMENTED REALITY VISUALIZATION

• US 20180262743A1
• Filed: 05/07/2018
• Published: 09/13/2018
• Est. Priority Date: 12/30/2014
• Status: Active Grant

• Current Assignee: Onpoint Medical, Inc.
• Original Assignee: Onpoint Medical, Inc.
• Inventors: Casas, Carlos Quiles
• Granted Patent: US 10,194,131 B2
• Time in Patent Office: Days
• CPC Class Codes: A61B 1/00 Instruments for performing medical examinations of the interior of cavities or tubes of the body by visual or photographical inspection, e.g. endoscopes examination of body cavities or body tracts using ultrasonic, sonic or infrasonic waves A61B8/12; endoscopic instruments for taking cell samples or for biopsy A61B10/04; for surgical purposes A61B17/00; endoscopic surgical cutting instruments A61B17/32; surgical instruments using a laser beam being directed along or through a flexible conduit A61B18/22; Illuminating arrangements therefor for the eyes A61B3/00
• CPC Class Codes: A61B 2034/2057 Details of tracking cameras
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: G02B 2027/0134 of stereoscopic type
• CPC Class Codes: G02B 2027/0138 comprising image capture systems, e.g. camera
• CPC Class Codes: G02B 2027/014 comprising information/image processing systems
• CPC Class Codes: G02B 27/0172 characterised by optical features
• CPC Class Codes: G06F 2203/04804 Transparency, e.g. transparent or translucent windows
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0304 Detection arrangements using opto-electronic means constructional details of pointing devices not related to the detection arrangement using opto-electronic means G06F3/033; optical digitisers G06F3/042
• CPC Class Codes: G06F 3/04845 for image manipulation, e.g. dragging, rotation, expansion or change of colour
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G16Z 99/00 Subject matter not provided for in other main groups of this subclass
• CPC Class Codes: H04N 13/111 Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
• CPC Class Codes: H04N 13/156 Mixing image signals
• CPC Class Codes: H04N 13/239 : using two 2D image sensors having a relative position equal to or related to the interocular distance H04N13/243 takes precedence
• CPC Class Codes: H04N 13/279 : the virtual viewpoint locations being selected by the viewers or determined by tracking
• CPC Class Codes: H04N 13/296 : Synchronisation thereof; Control thereof
• CPC Class Codes: H04N 13/366 : using viewer tracking

Abstract

Embodiments disclose a real-time surgery method and apparatus for displaying a stereoscopic augmented view of a patient from a static or dynamic viewpoint of the surgeon, which employs real-time three-dimensional surface reconstruction for preoperative and intraoperative image registration. Stereoscopic cameras provide real-time images of the scene including the patient. A stereoscopic video display is used by the surgeon, who sees a graphical representation of the preoperative or intraoperative images blended with the video images in a stereoscopic manner through a see through display.

Claims

1. A system comprising:

an optical head mounted display;

a computer means configured to receive preoperative, intraoperative or preoperative and intraoperative image information from a patient and to generate a 3D image of the spine of the patient; and

one or more markers configured to be attached to the patient'"'"'s spine,

wherein the computer means is configured to register the image information of the patient'"'"'s spine in a common coordinate system,

wherein the computer means is configured to register the one or more markers in the common coordinate system,

wherein the computer means is configured to track movement from a first position to a second position of the patient'"'"'s spine during surgery,

wherein the computer means is configured to track the location and orientation of the optical head mounted display in the common coordinate system, and

wherein the computer means is configured to adjust the display of the 3D image by the optical head mounted display, based on the movement from the first position to the second position of the patient'"'"'s spine and based on the location and orientation of the optical head mounted display.

2. The system of claim 1, wherein the computer means is configured to adjust the display of the 3D image based on the movement from the first position to the second position of the patient'"'"'s spine and based on the location and orientation of the optical head mounted display automatically in real time or fixed intervals.

3. The system of claim 1, wherein the computer means is configured to reconstruct one or more preoperative, intraoperative or preoperative and intraoperative two-dimensional images into three-dimensional images using 2D to 3D reconstruction.

4. The system of claim 1, wherein the computer means is configured to track movement from the first position to the second position of the patient'"'"'s spine by tracking the position of the one or more markers.

5. The system of claim 1, wherein the computer means is configured to compare tracking data of instruments and/or devices to the tracked position of the patient'"'"'s spine during surgery.

6. The system of claim 1, wherein the computer means is configured to track the location and/or orientation of one or more of the instruments, devices, the patient'"'"'s spine, the surgeon, or the optical head mounted display.

7. The system of claim 1, wherein the computer means comprises one or more of a mainframe computer, a server computer, a desktop computer, a laptop computer, a tablet computer, a home entertainment computer, a network computing device, a mobile computing device, a mobile communication device, and a gaming device.

8. The system of claim 1, wherein the computer means configured to receive preoperative, intraoperative or preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient is the same or different than one or more of the computer means configured to register the image information of the patient'"'"'s spine in the common coordinate system, the computer means configured to register the one or more markers in the common coordinate system, the computer means configured to track movement from the first position to the second position of the patient'"'"'s spine during surgery, the computer means configured to track the location and orientation of the optical head mounted display in the common coordinate system, and the computer means configured to adjust the display of the 3D image by the optical head mounted display.

9. The system of claim 1, wherein the computer means configured to register the image information of the patient'"'"'s spine in the common coordinate system is the same or different than one or more of the computer means configured to receive the preoperative, the intraoperative or the preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient, the computer means configured to register the one or more markers in the common coordinate system, the computer means configured to track movement from the first position to the second position of the patient'"'"'s spine during surgery,

the computer means configured to track the location and orientation of the optical head mounted display in the common coordinate system, and the computer means configured to adjust the display of the 3D image by the optical head mounted display.

10. The system of claim 1, wherein the computer means configured to register the one or more markers in the common coordinate system is the same or different than one or more of the computer means configured to receive the preoperative, the intraoperative or the preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient, the computer means configured to register the image information of the patient'"'"'s spine in the common coordinate system,

the computer means configured to track movement from the first position to the second position of the patient'"'"'s spine during surgery, the computer means configured to track the location and orientation of the optical head mounted display in the common coordinate system, and the computer means configured to adjust the display of the 3D image by the optical head mounted display.

11. The system of claim 1, wherein the computer means configured to track movement from the first position to the second position of the patient'"'"'s spine during surgery is the same or different than one or more of the computer means configured to receive the preoperative, the intraoperative or the preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient, the computer means configured to register the image information of the patient'"'"'s spine in the common coordinate system,

the computer means configured to register the one or more markers in the common coordinate system, the computer means configured to track the location and orientation of the optical head mounted display in the common coordinate system, and the computer means configured to adjust the display of the 3D image by the optical head mounted display.

12. The system of claim 1, wherein the computer means configured to track the location and orientation of the optical head mounted display in the common coordinate system is the same or different than one or more of the computer means configured to receive the preoperative, the intraoperative or the preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient, the computer means configured to register the image information of the patient'"'"'s spine in the common coordinate system, the computer means configured to register the one or more markers in the common coordinate system, the computer means configured to track movement from the first position to the second position of the patient'"'"'s spine during surgery, and the computer means configured to adjust the display of the 3D image by the optical head mounted display.

13. The system of claim 1, wherein the computer means configured to adjust the display of the 3D image by the optical head mounted display is the same or different than one or more of the computer means configured to receive the preoperative, the intraoperative or the preoperative and intraoperative image information from the patient and to generate the 3D image of the spine of the patient, the computer means configured to register the image information of the patient'"'"'s spine in the common coordinate system, the computer means configured to register the one or more markers in the common coordinate system, the computer means configured to track movement from the first position to the second position of the patient'"'"'s spine during surgery, and the computer means configured to track the location and orientation of the optical head mounted display in the common coordinate system.

14. The system of claim 1, wherein the image information of the patient'"'"'s spine is registered with one or more structures of the patient'"'"'s spine using a navigation system.

15. The system of claim 1, further comprising a 3D scanner, wherein the 3D scanner is configured to obtain a 3D surface of the patient'"'"'s spine and to determine the pose of the patient'"'"'s spine.

16. The system of claim 1, wherein the optical head mounted display is configured to provide for adjustment of transparency, color and contrast of the 3D image of the patient'"'"'s spine.

17. The system of claim 1, wherein the optical head mounted display is configured to display one or more virtual graphical representations of tracked instruments or devices.

18. The system of claim 1, wherein the 3D image of the patient'"'"'s spine includes 3D images of individually rendered vertebrae registered in the common coordinate system.

19. The system of claim 18, wherein the computer means is configured to track pose changes of the individual vertebrae in the common coordinate system and wherein the pose changes are translated into display of a different second 3D image at a location and orientation that corresponds to the location and orientation of the patient'"'"'s spine after the pose change.

20. The system of claim 6, wherein the tracking of the one or more of the instruments, devices, the patient'"'"'s spine, the surgeon, or the optical head mounted display is performed using one or more of attached passive optical markers, active optical markers, magnetic tracking, electromagnetic tracking, ultrasonic tracking, mechanical tracking and tracking using inertial measurement units.

21. The system of claim 1, wherein the optical head mounted display is configured to display one or more of graphical representations of instruments, surgical guides or techniques, or anatomical models.

22. The system of claim 6, wherein the instruments include one or more of a pin, screw, nail and plate.

23. The system of claim 1, wherein the optical head mounted display is a see through optical head mounted display.

24. A system comprising:

an optical head mounted display;

a computer means configured to receive preoperative, intraoperative, or preoperative and intraoperative image information from a patient and to generate one or more 3D images of individualized vertebrae; and

one or more markers configured to be attached to the patient'"'"'s spine,

wherein the computer means is configured to register the image information of the individualized vertebrae in a common coordinate system,

wherein the computer means is configured to track movement from a first position to a second position of the patient'"'"'s spine during surgery by tracking the position of the one or more markers,

wherein the computer means is configured to track the location and orientation of the optical head mounted display in the common coordinate system, and

wherein the computer means is configured to adjust the display of the 3D images of the one or more individualized vertebrae based on the movement of the patient'"'"'s spine and based on the location and orientation of the see through optical head mounted display.

25. The system of claim 24, wherein the computer means is configured to detect changes in the sagittal position of the patient'"'"'s spine and to adjust the display of the one or more 3D images of the individualized vertebrae based on the detected changes in sagittal position.

26. The system of claim 24, wherein the computer means is configured to detect changes in lateralization or rotation of the patient'"'"'s spine and to adjust the display of the one or more 3D images of the individualized vertebrae based on the detected changes in lateralization or rotation of the patient'"'"'s spine.

27. The system of claim 24, wherein the optical head mounted display is a see through optical head mounted display.

28. The system of claim 24, wherein the location and orientation of the one or more 3D images of the individualized vertebrae displayed by the optical head mounted display matches the location and orientation of the individual vertebrae of the patient'"'"'s spine.

29. The system of claim 24, wherein the computer means is configured to track position changes of the individual vertebrae in the common coordinate system and wherein the position changes are translated into display of a different, second set of 3D images of the individualized vertebrae displayed at a location and orientation that matches the location and orientation of the individual vertebrae of the patient'"'"'s spine after the position change.

30. A system comprising:

an optical head mounted display;

a computer means configured to receive preoperative, intraoperative, or preoperative and intraoperative image information from a patient and to generate one or more 3D images of individualized vertebrae; and

one or more markers configured to be attached to the patient'"'"'s spine; and

wherein the computer means is configured to register the image information of the individualized vertebrae in a common coordinate system;

wherein the computer means is configured to track movement from a first position to a second position of the individual vertebrae during surgery by tracking the position of the one or more markers,

wherein the computer means is configured to track the location and orientation of the optical head mounted display in the common coordinate system,

wherein the computer means is configured to adjust the display of the 3D images of the one or more individualized vertebrae based on the movement of the patient'"'"'s spine and based on the location and orientation of the see through optical head mounted display.