Surgeon head-mounted display apparatuses

• US 10,650,594 B2
• Filed: 09/25/2017
• Issued: 05/12/2020
• Est. Priority Date: 02/03/2015
• Status: Active Grant

• Current Assignee: Globus Medical, Inc.
• Original Assignee: Globus Medical, Inc.
• Inventors: Jones, Kenneth Milton, Popoolapade, John, Calloway, Thomas, Lemoine, Thierry, Jutteau, Christian, Bruzy, Christophe, James, Yannick, Laguarda, Joachim, Xing, Dong-Mei Pei, Gorges, Sebastien, Yarin, Paul Michael
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US15/713,839
• Publication Number:
  US 20180012416A1
• Time in Patent Office: 960 Days
• Field of Search: 345633
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/368 changing the image on a display according to the operator's position
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

Claims

1. An augmented reality surgical system comprising:

a display comprising a see-through display screen that display images while allowing transmission of ambient light therethrough;

a motion sensor connected to the display and configured to output a motion signal, wherein the motion sensor includes at least one inertial sensor configured to output the motion signal indicating a measurement of movement or static orientation of a user'"'"'s head while wearing the display; and

at least one camera configured to observe a first set of reference markers connected to a patient, and a second set of reference markers connected to a surgical tool located within a surgical room;

a computer configured to;

compute the relative location and orientation of the display and the first and second set of reference markers;

generate a three dimensional anatomical image using patient data created by medical imaging equipment that has imaged a portion of the patient;

generate a video signal based on at least a portion of the three dimensional anatomical image and the location and orientation of the first and second set of reference markers coupled to the patient and the surgical instrument,

output the video signal to the display screen on the display,

the computer including a gesture interpretation module configured to sense gestures of the user, wherein the computer changes the displayed video signal on the display based on the sensed gestures of the user.

2. The augmented reality surgical system of claim 1, wherein the user sees a graphical representation on the display screen of the at least a portion of a three dimensional anatomical model oriented and scaled to provide a displayed overlay on the portion of the patient that was imaged by the medical imaging equipment.

3. The augmented reality surgical system of claim 1, wherein the user sees a graphical representation of a medical procedure on the display overlaid on the portion of the patient that was imaged by the medical imaging equipment.

4. The augmented reality surgical system of claim 1, wherein the gesture interpretation module is configured to sense movement of at least one finger of the user.

5. The augmented reality surgical system of claim 4, wherein the computer adjusts the position of the displayed video signal on the display based on the sensed finger movement of the user.

6. The augmented reality surgical system of claim 4, wherein the gesture interpretation module is configured to sense movement of a hand of the user through at least one camera and the computer adjusts the position of the displayed video signal on the display based on the sensed hand movement of the user.

7. The augmented reality surgical system of claim 1, wherein the computer is configured to position the displayed video signal to appear to hover in space above and adjacent to the location of the operation site.

8. An augmented reality surgical system comprising:

a positioning tracking system for tracking a location of a surgical tool, a head mounted display, and a surgical site;

a computer system using patient data from an imaging device to generate an image model of a targeted site of the patient;

wherein the image model includes reference markers that assist in correlating between virtual locations and physical locations of the patient,

wherein the computer system processes the present locations of the head mounted display, the surgical site and the surgical tool obtained by the positioning tracking system and uses the reference markers contained in the image model to generate a graphical representation of the patient to the head mounted display for a user to view,

wherein the computer system provides a graphical representation of a virtual trajectory of a surgical tool to the patient'"'"'s surgical site,

wherein a motion sensor includes at least one inertial sensor configured to output a motion signal indicating a measurement of movement or static orientation of a user'"'"'s head while wearing the head mounted display,

the computer system including a gesture interpretation module configured to sense gestures of the user, wherein the computer changes the displayed graphical representation of the patient on the display based on the sensed gestures of the user.

9. The system of claim 8, wherein the computer system may display graphical representations of the surgical tool or surgical site on the display screen of the head mounted display.

10. The system of claim 8, wherein the computer system may animate movement of the displayed graphical representation of the patient, surgical tool or surgical site to illustrate a planned procedure relative to a defined location of the surgical site or other targeted anatomical site.

11. The system of claim 8, wherein imaging device may include one of an x-ray device, endoscope cameras, magnetic resonance imaging equipment, computed tomography scanning equipment, three-dimensional ultrasound equipment, and endoscopic equipment.

12. The system of claim 8, wherein the computer may receive non-image patient data and display the non-image data into a graphical representation to be displayed on the head mounted display.

13. The system of claim 8, wherein the positioning tracking system includes a plurality of cameras spaced apart at defined locations within an operating room and each having a field of view that can observe objects to be tracked.

14. The system of claim 8, wherein the positioning tracking system may use electromagnetic ranging and trackers, ultrasound emitters/sensors for ranging and trackers.

15. An augmented reality surgical system for displaying multiple video streams to a user, the augmented reality surgical system comprising:

a display comprising a see-through display screen that display images while allowing transmission of ambient light therethrough;

a motion sensor connected to the display and configured to output a motion signal, wherein the motion sensor includes at least one inertial sensor configured to output the motion signal indicating a measurement of movement or static orientation of a user'"'"'s head while wearing the display; and

a computer configured to receive a plurality of patient data including a three dimensional anatomical image of the patient and relative location and orientation of the patient and to control which or of the plurality of patient data to output as a video signal to the display screen,

wherein the computer provides a graphical representation of a virtual trajectory of a surgical tool to the patient'"'"'s surgical site,

the computer including a gesture interpretation module configured to sense gestures of the user, wherein the computer changes the displayed video signal on the display screen based on the sensed gestures of the user.

16. The augmented reality surgical system of claim 15, wherein:

the computer is communicatively connected to a data server to receive the patient data, one of the which is a data generated by medical imaging equipment, another one of the patient data is information from a patient database defining a patient'"'"'s medical history, and another one of the patient data comprises data generated by medical equipment based on real-time monitoring of the patient'"'"'s vitals.

17. The augmented reality surgical system of claim 15, wherein:

the computer is configured to convert the patient data into a graphical display and to output the selected one or all of the plurality of patient data into video streams as a video signal to the display screen.

18. The augmented reality surgical system of claim 15, wherein the computer communicates video from the video camera through a network to another display worn by another person.

19. The augmented reality surgical system of claim 15, wherein the surgical tool is a drill bit.