Augmented reality navigation systems for use with robotic surgical systems and methods of their use

• US 10,646,283 B2
• Filed: 02/22/2018
• Issued: 05/12/2020
• Est. Priority Date: 02/19/2018
• Status: Active Grant

• Current Assignee: Globus Medical, Inc.
• Original Assignee: Globus Medical, Inc.
• Inventors: Johnson, Norbert, Forsyth, Jeffrey, Crawford, Neil, Joshi, Sanjay, Al Jewad, Bessam, Healy, Weston, Russ, Christine, Jones, Ken
• Primary Examiner(s): He, Yingchun
• Application Number:
  US15/902,053
• Publication Number:
  US 20190254754A1
• Time in Patent Office: 810 Days
• Field of Search: None
• CPC Class Codes: A61B 2017/00203 with speech control or speech recognition
• CPC Class Codes: A61B 2017/00207 with hand gesture control or hand gesture recognition
• CPC Class Codes: A61B 2017/00216 with eye tracking or head position tracking control
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/2057 Details of tracking cameras
• CPC Class Codes: A61B 2034/2068 using pointers, e.g. pointers having reference marks for determining coordinates of body points
• CPC Class Codes: A61B 2034/2074 Interface software
• CPC Class Codes: A61B 2034/254 being adapted depending on the stage of the surgical procedure
• CPC Class Codes: A61B 2034/256 having a database of accessory information, e.g. including context sensitive help or scientific articles
• CPC Class Codes: A61B 2034/258 providing specific settings for specific users
• CPC Class Codes: A61B 2090/3616 Magnifying glass
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/371 with simultaneous use of two cameras
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 2090/374 NMR or MRI
• CPC Class Codes: A61B 2090/376 using X-rays, e.g. fluoroscopy
• CPC Class Codes: A61B 2090/378 using ultrasound
• CPC Class Codes: A61B 2090/502 Headgear, e.g. helmet, spectacles
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/30 : Surgical robots
• CPC Class Codes: A61B 34/76 : Manipulators having means for providing feel, e.g. force or tactile feedback
• CPC Class Codes: A61B 90/03 : Automatic limiting or abutting means, e.g. for safety
• CPC Class Codes: G02B 27/017 : Head mounted
• CPC Class Codes: G06F 3/011 : Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/63 : for local operation
• CPC Class Codes: G16H 50/50 : for simulation or modelling of medical disorders

Abstract

The present disclosure is directed to augmented reality navigation systems and methods of their use that, inter alia, address the need for systems and methods of robotic surgical system navigation with reduced distraction to surgeons. Augmented reality navigation systems disclosed herein enable a surgeon to maintain focus on a surgical site and/or surgical tool being used in a surgical procedure while obtaining a wide range of navigational information relevant to the procedure. Navigational information can appear in the augmented reality navigation system as being presented on virtual displays that sit in a natural field of view of a surgeon during a procedure. Navigational information can also appear to be overlaid over a patient'"'"'s anatomy. Augmented reality navigation systems comprise a head mounted display comprising an at least partially transparent display screen, at least one detector connected to the head mounted display for identifying real-world features, and a computer subsystem.

Claims

1. An augmented reality navigation system comprising:

a head mounted display comprising an at least partially transparent display screen configured to display augmentation graphics, wherein the display screen is configured to allow the user to see a natural field of view of the user while seeing the augmentation graphics overlaid on the natural field of view;

at least one detector for identifying real-world features, the at least one detector connected to the head mounted display;

a surgical tool having markers and configured to be detected by at the at least one detector, wherein a representation of at least a portion of the surgical tool and/or a trajectory of the surgical tool is presented in the head mounted display,

wherein a detector input signal from the at least one detector corresponds to a field of view of the at least one detector and the field of view comprises at least a portion of anatomy of a patient during a surgical procedure,

wherein the detector input signal includes a relative location and/or orientation for each of one or more of the real-world features

wherein the display screen displays at least three operational views simultaneously, and wherein the at least three operational views may be separately activated by changing a pitch angle of the head mounted display.

2. The augmented reality navigation system of claim 1, wherein a camera system for detecting real-world features is electrically coupled to the head mounted display.

3. The augmented reality navigation system of claim 1, wherein the head mounted display provides a representation of the surgical tool and a trajectory of the surgical tool overlaid on the anatomy of the patient.

4. The augmented reality navigation system of claim 1, further includes a motion sensor connected to the head mounted display for outputting a motion signal based on measured motion of the head mounted display.

5. The augmented reality navigation system of claim 1, wherein the at least one detector comprises a detector with at least a minimum field of view of 40 degrees.

6. The augmented reality navigation system of claim 1, wherein the display screen has a resolution of at least 1280×

720 pixels.

7. The augmented reality navigation system of claim 1, comprising a pointer tool for making surgical planning selections, wherein the pointer tool is configured to be detected by the at least one detector.

8. The augmented reality navigation system of claim 1, wherein the at least one detector comprises a video camera and transmits a video signal to the head mounted display to display augmentation graphics which appear to the user to be superimposed on at least a portion of a natural field of view of the user.

9. The augmented reality navigation system of claim 1, wherein the navigation system is used to perform a surgical procedure.

10. An augmented reality navigation system for use with a robotic surgical system, the system comprising:

a head mounted display comprising an at least partially transparent display screen configured to display augmentation graphics, wherein the display screen is configured to allow the user to see a natural field of view of the user while seeing the augmentation graphics overlaid on the natural field of view;

at least one detector for identifying real-world features, the at least one detector connected to the head mounted display; and

a computer subsystem configured to generate and/or access a representation of at least a portion of a surgical tool and/or a trajectory of the surgical tool during a surgical procedure, modify at least a portion of the representation based on a relative position and/or orientation of one or more real-world features in a detector input signal received from the at least one detector, and display, on the display screen, surgical tool augmentation graphics based on the modified representation

wherein the display screen displays at least three operational views simultaneously, and wherein the at least three operational views may be separately activated by changing a pitch angle of the head mounted display.

11. The augmented reality navigation system of claim 10, wherein the computer subsystem is configured to render a surgical tool augmentation graphic for each of a plurality of surgical tool trajectories, and display, on the display screen, the plurality of surgical tool augmentation graphics such that the surgical tool augmentation graphics appear overlaid on an anatomy of the patient and each of the trajectory augmentation graphics indicate a physical trajectory that could be followed during the surgical procedure.

12. The augmented reality navigation system of claim 10, wherein

the computer subsystem is configured to modify an anatomical model of a patient based on one or more relative location(s) and/or orientation(s) determined from the detected input signal, thereby forming an updated anatomical model, and

the computer subsystem is configured to display, on the display screen, anatomical model augmentation graphics corresponding to the updated anatomical model such that the updated anatomical model appears overlaid on an anatomy of the patient.

13. The augmented reality navigation system of claim 10, comprising:

a motion sensor connected to the head mounted display for outputting a motion signal based on measured motion of the head mounted display.

14. An augmented reality navigation system comprising:

a head mounted display comprising an at least partially transparent display screen configured to display augmentation graphics, wherein the display screen is configured to allow the user to see a natural field of view of the user while seeing the augmentation graphics overlaid on the natural field of view; and

at least one detector for identifying real-world features, the at least one detector connected to the head mounted display;

wherein a detector input signal from the at least one detector corresponds to a field of view of the at least one detector and the field of view comprises at least a portion of anatomy of a patient during a surgical procedure,

wherein the detector input signal includes a relative location and/or orientation for each of one or more of the real-world features,

wherein the display screen displays at least three operational views simultaneously, and wherein the at least three operational views may be separately activated by changing a pitch angle of the head mounted display.

15. The augmented reality navigation system of claim 14, wherein a camera system for detecting real-world features is electrically coupled to the head mounted display.

16. The augmented reality navigation system of claim 14, wherein the head mounted display provides a representation of a surgical tool and a trajectory of the surgical tool overlaid on an anatomy of the patient.

17. The augmented reality navigation system of claim 14, further includes a motion sensor connected to the head mounted display for outputting a motion signal based on measured motion of the head mounted display.

18. The augmented reality navigation system of claim 14, comprising a pointer tool for making surgical planning selections, wherein the pointer tool is configured to be detected by the at least one detector.

19. The augmented reality navigation system of claim 14, wherein the at least one detector comprises a video camera and transmits a video signal to the head mounted display to display augmentation graphics which appear to the user to be superimposed on at least a portion of a natural field of view of the user.

20. The augmented reality navigation system of claim 14, wherein the navigation system is used to perform a surgical procedure.