Surgeon head-mounted display apparatuses

• US 10,580,217 B2
• Filed: 06/05/2018
• Issued: 03/03/2020
• Est. Priority Date: 02/03/2015
• Status: Active Grant

• Current Assignee: Globus Medical, Inc.
• Original Assignee: Globus Medical, Inc.
• Inventors: Jones, Kenneth Milton, Popoolapade, John, Calloway, Thomas, Lemoine, Thierry, Jutteau, Christian, Bruzy, Christophe, James, Yannick, Laguarda, Joachim, Xing, Dong-Mei Pei, Gorges, Sebastien, Yarin, Paul Michael
• Primary Examiner(s): Amini, Javid A
• Application Number:
  US16/000,393
• Publication Number:
  US 20180286136A1
• Time in Patent Office: 637 Days
• Field of Search: 345633
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/368 changing the image on a display according to the operator's position
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/012 Head tracking input arrangements
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/0346 with detection of the device orientation or free movement in a 3D space, e.g. 3D mice, 6-DOF [six degrees of freedom] pointers using gyroscopes, accelerometers or tilt-sensors
• CPC Class Codes: G06T 19/006 Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 2219/2016 Rotation, translation, scaling

Abstract

An augmented reality surgical system includes a head mounted display (HMD) with a see-through display screen, a motion sensor, a camera, and computer equipment. The motion sensor outputs a head motion signal indicating measured movement of the HMD. The computer equipment computes the relative location and orientation of reference markers connected to the HMD and to the patient based on processing a video signal from the camera. The computer equipment generates a three dimensional anatomical model using patient data created by medical imaging equipment, and rotates and scales at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the HMD. The rotated and scaled three dimensional anatomical model is displayed on the display screen.

Claims

1. An augmented reality surgical system comprising:

a head mounted display comprising a see-through display screen that display images while allowing transmission of ambient light therethrough;

a motion sensor connected to the head mounted display and configured to output a head motion signal indicating measured movement of the head mounted display; and

at least one camera configured to observe reference markers connected to the head mounted display, reference markers connected to a patient, and reference markers connected to a surgical tool located within a surgical room;

computer equipment configured to;

compute the relative location and orientation of the reference markers connected to the head mounted display and the reference markers connected to the patient based on processing a video signal from the at least one camera;

generate a three dimensional anatomical model using patient data created by medical imaging equipment that has imaged a portion of the patient;

rotate and scale at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers connected to the head mounted display and the reference markers connected to the patient, and further rotate at least a portion of the three dimensional anatomical model based on the head motion signal to track measured movement of the head mounted display; and

generate a video signal based on the rotated and scaled three dimensional anatomical model;

output the video signal to the display screen of the head mounted display; and

output a second video signal that is a graphical representation of a virtual trajectory of a surgical instrument that intersects a target location on a patient, the target location corresponds to a point where the surgical instrument impact the three dimensional anatomical image which is oriented and scaled to match the patient'"'"'s anatomy, and wherein the virtual trajectory is continuously updated and dynamically displayed to a surgeon who is repositioning and reorienting the surgical instrument.

2. The augmented reality surgical system of claim 1, wherein:

the computer equipment is configured to rotate and scale the at least a portion of the three dimensional anatomical model based on the relative location and orientation of the reference markers connected to the head mounted display and the reference markers connected to the patient to transform the at least a portion of the three dimensional anatomical model to a present perspective view of a user wearing the head mounted display so the user sees a graphical representation on the display screen of the at least a portion of the three dimensional anatomical model oriented and scaled to provide a displayed overlay on the portion of the patient that was imaged by the medical imaging equipment.

3. The augmented reality surgical system of claim 1, further comprising at least one video camera connected to the head mounted display to be forward-facing away from the user, wherein the computer equipment compares patterns of anatomical objects in a video stream from the at least one video camera to patterns of anatomical objects in the patient data created by the medical imaging equipment, and controls generation of the three dimensional anatomical model from the patient data responsive to identifying a threshold level of correspondence between the compared patterns of anatomical objects.

4. The augmented reality surgical system of claim 1, further comprising at least one video camera connected to the head mounted display to be forward-facing away from the user, wherein the computer equipment compares patterns of anatomical objects in a video stream from the at least one video camera to patterns of anatomical objects in the patient data created by the medical imaging equipment, and displays a graphical indicia on the display screen aligned with one of the anatomical objects displayed on the display screen from the rotated and scaled three dimensional anatomical model responsive to identifying a threshold level of correspondence between a pattern of the one of the anatomical objects and a pattern of one of the anatomical objects in the video stream from the at least one video camera.

5. The augmented reality surgical system of claim 1, wherein:

the at least one camera is further configured to observe reference markers connected to a surgical tool located within a surgical room;

the computer equipment is further configured to;

compute the relative location and orientation of the reference markers connected to the head mounted display, the reference markers connected to the patient, and the reference markers connected to the surgical tool based on processing a video signal from the at least one camera; and

generate the video signal to include a graphical representation of the surgical tool illustrated at a position relative to the three dimensional anatomical model that is determined based on the relative location and orientation of the reference markers connected to the head mounted display, the reference markers connected to the patient, and the reference markers connected to the surgical tool.

6. The augmented reality surgical system of claim 5, wherein the computer equipment is further configured to:

generate a graphical representation of a virtual trajectory extending from the surgical tool into the three dimensional anatomical model based on the relative location and orientation of the reference markers connected to the head mounted display, the reference markers connected to the patient, and the reference markers connected to the surgical tool; and

generate the video signal to include the graphical representation of the virtual trajectory.

7. The augmented reality surgical system of claim 6, wherein:

the computer equipment is further configured to select an image slice from among a plurality of image slices contained in the three dimensional anatomical model, the image slice being selected based on the computer equipment determining that the image slice is traversed by the virtual trajectory extending from the surgical tool, and to generate the video signal to include the image slice.

8. The augmented reality surgical system of claim 7, wherein:

the computer equipment is further configured to rotate a graphical representation of the image slice and the graphical representation of the virtual trajectory that are displayed on the display screen responsive to the head motion signal from the motion sensor and/or responsive to a command received from a user.

9. The augmented reality surgical system of claim 7, wherein:

the computer equipment is further configured to rotate a graphical representation of the image slice and the graphical representation of the virtual trajectory that are displayed on the display screen to provide a view of the image slice that is perpendicular to a direction of the virtual trajectory.

10. The augmented reality surgical system of claim 9, wherein:

the computer equipment is further configured to identify a target location within the image slice, and to display the target location relative to the graphical representation of the virtual trajectory.

11. An augmented reality surgical system for displaying multiple video streams to a user, the augmented reality surgical system comprising:

a head mounted display comprising a see-through display screen that display images while allowing transmission of ambient light therethrough;

a motion sensor connected to the head mounted display and configured to output a head motion signal indicating measured movement of the head mounted display; and

a computer equipment configured to receive a plurality of video streams from one or more source devices and to control which of the video streams are output as a video signal to the display screen based on the head motion signal

wherein the video signal is a graphical representation of a virtual trajectory of a surgical instrument that intersects a target location on a patient the target location corresponds to a point where the surgical instrument impact the three dimensional anatomical image which is oriented and scaled to match the patient'"'"'s anatomy, and wherein the virtual trajectory is continuously updated and dynamically displayed to a surgeon who is repositioning and reorienting the surgical instrument.

12. The augmented reality surgical system of claim 11, wherein:

the computer equipment is communicatively connected to a surgical video server to receive the video streams, one of the video streams comprises data generated by medical imaging equipment, another one of the video streams comprises information from a patient database defining a patient'"'"'s medical history, and another one of the video streams comprises data generated by medical equipment based on real-time monitoring of the patient'"'"'s vitals.

13. The augmented reality surgical system of claim 11, further comprising:

a headband configured to grasp the user'"'"'s head;

an arm apparatus connected between the headband and the head mounted display, the arm apparatus configured to allow rotation of the head mounted display relative to the headband by the user to position the display screen relative to the user'"'"'s line-of-sight.

14. The augmented reality surgical system of claim 11, wherein:

the arm apparatus comprises a pivot joint configured to allow the display screen to be flipped down to a deployed position within the user'"'"'s line-of sight and flipped-up to a stored position outside the user'"'"'s line-of sight.

15. The augmented reality surgical system of claim 11, wherein:

the computer equipment is configured to select one of the video streams from among the video streams based on the head motion signal, and to output the selected one of the video streams as a video signal to the display screen.

16. The augmented reality surgical system of claim 11, wherein the motion sensor is configured to output the head motion signal containing a pitch component that provides an indication of pitch angle of the head mounted display.

17. The augmented reality surgical system of claim 16, wherein the computer equipment is configured to set an origin reference point for measurement of a yaw component of the head motion signal based on processing the yaw component of the head motion signal during times when the pitch component of the head motion signal is below a defined threshold value.

18. The augmented reality surgical system of claim 17, wherein the motion sensor is a sourceless orientation sensor, and the computer equipment is configured to correct yaw drift of the sourceless orientation sensor towards zero responsive to determining that the pitch component of the head motion signal is below the defined threshold value.

19. The augmented reality surgical system of claim 18, wherein the sourceless orientation sensor comprises a gyroscope, and the computer equipment is configured to estimate biases of the gyroscope giving rise to the yaw drift, using yaw drift correction pseudo-measurements determined while the pitch component of the head motion signal is below the defined threshold value.

20. The augmented reality surgical system of claim 11, wherein the computer equipment recognizes voice commands and/or gesture commands from the user, and defines and/or adjusts the yaw and pitch coordinates of the virtual floating panels displayed through the display screen responsive to the recognized voice commands and/or gesture commands.