Augmented reality visualization for guiding bone cuts including robotics

• US 10,278,777 B1
• Filed: 12/12/2018
• Issued: 05/07/2019
• Est. Priority Date: 03/12/2016
• Status: Active Grant

• Current Assignee: Philipp K. Lang
• Original Assignee: Philipp K. Lang
• Inventors: Lang, Philipp K.
• Primary Examiner(s): Beccia, Christopher J
• Application Number:
  US16/217,555
• Publication Number:
  US 20190110842A1
• Time in Patent Office: 146 Days
• CPC Class Codes: A61B 17/15 Guides therefor arrangements for guiding straight saw blades in general B23D51/025
• CPC Class Codes: A61B 17/155 Cutting femur
• CPC Class Codes: A61B 17/157 Cutting tibia
• CPC Class Codes: A61B 17/1666 for the acetabulum
• CPC Class Codes: A61B 17/17 Guides or aligning means for drills , mills, pins or wires
• CPC Class Codes: A61B 17/1703 using imaging means, e.g. by X-rays
• CPC Class Codes: A61B 17/1742 for the hip
• CPC Class Codes: A61B 17/1764 for the knee
• CPC Class Codes: A61B 17/1775 for the foot or ankle
• CPC Class Codes: A61B 17/1778 for the shoulder
• CPC Class Codes: A61B 2017/00216 with eye tracking or head position tracking control
• CPC Class Codes: A61B 2017/568 produced with shape and dimensions specific for an individual patient
• CPC Class Codes: A61B 2034/104 Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2059 Mechanical position encoders
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/368 changing the image on a display according to the operator's position
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 2090/373 : using light, e.g. by using optical scanners
• CPC Class Codes: A61B 2090/374 : NMR or MRI
• CPC Class Codes: A61B 2090/376 : using X-rays, e.g. fluoroscopy
• CPC Class Codes: A61B 2090/3762 : using computed tomography systems [CT]
• CPC Class Codes: A61B 2090/502 : Headgear, e.g. helmet, spectacles
• CPC Class Codes: A61B 34/10 : Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/74 : Manipulators with manual electric input means
• CPC Class Codes: A61B 90/36 : Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/37 : Surgical systems with images on a monitor during operation
• CPC Class Codes: A61F 2/32 : for the hip
• CPC Class Codes: A61F 2/3859 : Femoral components
• CPC Class Codes: A61F 2/389 : Tibial components A61F2/3868 takes precedence
• CPC Class Codes: A61F 2002/4018 : Heads or epiphyseal parts of humerus
• CPC Class Codes: A61F 2002/4205 : Tibial components
• CPC Class Codes: A61F 2002/4207 : Talar components
• CPC Class Codes: H05K 999/99 : dummy group
• CPC Class Codes: Y02A 90/10 : Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Abstract

Devices and methods for performing a surgical step or surgical procedure with visual guidance using an optical head mounted display are disclosed.

Claims

1. A method of preparing a physical joint in a patient comprising:

generating, by at least one computer, a virtual surgical guide, wherein the virtual surgical guide is a virtual plane, wherein the virtual plane is a three-dimensional digital representation indicating at least one of a predetermined resection level, a predetermined resection angle, a predetermined varus or valgus angle, a predetermined flexion, or a predetermined slope, for superimposing one or more physical surgical tools or physical surgical instruments; and

displaying, at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion, or predetermined slope, using a see through optical head mounted display, the virtual plane onto a surface of the physical joint visible directly through the see through optical head mounted display, wherein the virtual plane is configured to allow superimposition of the one or more physical surgical tools or physical surgical instruments with the virtual plane,

wherein the display of the virtual plane at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion, or predetermined slope is configured to be maintained by the at least one computer in relationship to one or more anatomic structures, wherein the one or more anatomic structures are registered in a coordinate system,

wherein the physical joint of the patient, and the one or more physical surgical tools or physical surgical instruments are visible directly through the see through optical head mounted display,

wherein the one or more physical surgical tools or physical surgical instruments are configured to perform one or more bone cuts or a bone removal of the joint,

wherein the virtual plane is registered in the coordinate system, and

wherein the see through optical head mounted display is registered in the coordinate system.

2. The method of claim 1, wherein the physical surgical instrument is attached to a robot.

3. The method of claim 2, wherein the robot compromises a robotic arm.

4. The method of claim 2, wherein the robot comprises an attached tissue cutter, wherein the tissue cutter is a saw, a burr, a blade, a drill, a pin, a scalpel, a reamer, a broach, or a mill.

5. The method of claim 1, wherein the virtual plane intersects with the surface of the joint and wherein the virtual plane is a virtual predetermined path for a saw blade.

6. The method of claim 1, comprising displaying using the see through optical head mounted display a virtual saw blade intersecting at least a portion of the joint.

7. The method of claim 6, wherein the virtual saw blade is a three-dimensional digital representation of a physical saw blade.

8. The method of claim 1, wherein the virtual plane is configured to allow at least partial superimposition of the one or more physical surgical tools or physical surgical instruments with the virtual plane.

9. The method of claim 1, wherein the display of the virtual plane at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion, or predetermined slope is configured to be maintained by the at least one computer in relationship to the one or more anatomic structures when the one or more physical surgical tool or physical surgical instrument moves in the field of view of a user or when the physical joint of the patient moves.

10. The method of claim 1, wherein the display of the virtual plane at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion, or predetermined slope is configured to be maintained by the at least one computer in relationship to the one or more anatomic structures when the one or more physical surgical tool or physical surgical instrument is superimposed with the virtual plane.

11. The method of claim 1, wherein the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion, or predetermined slope comprises at least one of a predetermined start position, a predetermined start point, a predetermined intermediate position, a predetermined intermediate point, a predetermined end position, a predetermined end point, or combinations thereof, or wherein the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion, or predetermined slope comprises at least one of a predetermined angle, a predetermined start orientation, a predetermined start alignment, a predetermined intermediate orientation, a predetermined intermediate alignment, a predetermined end orientation, a predetermined end alignment, or combinations thereof.

12. The method of claim 1, wherein the display of the virtual plane at the at least one of the predetermined resection level, or predetermined resection angle determines a position, orientation, or combination thereof of a femoral component in a hip joint replacement or a leg length in a hip joint replacement.

13. The method of claim 1, wherein the display of the virtual plane at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion, or predetermined slope determines a position, orientation, varus or valgus angle, flexion, rotation, or combination thereof of a femoral component or a position, orientation, varus or valgus angle, slope, rotation, or combination thereof of a tibial component in a knee joint replacement.

14. The method of claim 1, wherein the display of the virtual plane at the at least one of the predetermined resection level, or predetermined resection angle determines a position, orientation, or combination thereof a humeral component in a shoulder joint replacement.

15. The method of claim 1, wherein the display of the virtual plane at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, or predetermined flexion determines a position, orientation, rotation, or combination thereof a tibial or talar component in an ankle joint replacement.

16. The method of claim 1, wherein the one or more physical surgical instruments or physical surgical tools are one or more of a bone saw, burr, blade, drill, pin, scalpel, reamer, broach or mill.

17. The method of claim 1, comprising obtaining by the at least one computer x, y, and z coordinates of the one or more anatomic structures using at least one of an infrared marker, an RF marker, an active marker, a passive marker, an IMU, an LED, an optical marker, a geometric pattern, a surgical navigation system, a camera, a video system, an image capture system, a depth sensor, a laser scanner, a 3D scanner, a patient specific marker, a patient specific template, an x-ray, an imaging system, or combinations thereof, and wherein the x, y, and z coordinates are registered in the coordinate system.

18. The method of claim 1, wherein the one or more anatomic structures comprise an anatomic landmark, an anatomic plane, an articular surface, a cartilage surface, a subchondral bone surface, a cortical bone surface, a cut bone surface, a reamed bone surface, a milled bone surface, an impacted bone surface, a tissue resection, a surface, one or more surface points, an anatomic axis, a biomechanical axis, a mechanical axis or combinations thereof.

19. A method of preparing a physical joint in a patient comprising:

generating, by at least one computer, a virtual surgical guide, wherein the virtual surgical guide is a virtual plane, wherein the virtual plane is a three-dimensional digital representation indicating at least one of a predetermined resection level, a predetermined resection angle, a predetermined varus or valgus angle, a predetermined flexion, or a predetermined slope for superimposing a physical tissue cutter, wherein the physical tissue cutter is attached to a robot; and

displaying, at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion or predetermined slope, using a see through optical head mounted display, the virtual plane onto a surface of the physical joint visible directly through the see through optical head mounted display,

wherein the virtual plane is configured to allow superimposition of the tissue cutter with the virtual plane,

wherein the display of the virtual plane at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion or predetermined slope is configured to be maintained by the at least one computer in relationship to one or more anatomic structures, wherein the one or more anatomic structures are registered in a coordinate system,

wherein the physical joint of the patient, and the physical tissue cutter are visible directly through the see through optical head mounted display,

wherein the physical tissue cutter is configured to perform one or more bone cuts or a bone removal of the joint,

wherein the virtual plane is registered in the coordinate system, and

wherein the see through optical head mounted display is registered in the coordinate system.

20. The method of claim 19, wherein the tissue cutter is a saw, a burr, a blade, a drill, a pin, a scalpel, a reamer, a broach, or a mill.

21. The method of claim 19, wherein the display of the virtual plane at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion or predetermined slope, is configured to be maintained by the at least one computer in relationship to the one or more anatomic structures when the physical tissue cutter moves in the field of view of a user or when the physical joint of the patient moves.

22. The method of claim 19, wherein the display of the virtual plane at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion, predetermined slope is configured to be maintained by the at least one computer in relationship to the one or more anatomic structures when the physical tissue cutter is at least partially superimposed with the virtual plane.

23. The method of claim 19, wherein the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion or predetermined slope comprises at least one of a predetermined start position, a predetermined start point, a predetermined intermediate position, a predetermined intermediate point, a predetermined end position, a predetermined end point, or combinations thereof.

24. The method of claim 19, wherein the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion or predetermined slope comprises at least one of a predetermined angle, a predetermined start orientation, a predetermined start alignment, a predetermined intermediate position, a predetermined intermediate point, a predetermined end orientation, a predetermined end alignment, or combinations thereof.

25. The method of claim 19, wherein the at least one computer is configured to display a safe zone by the optical head mounted display.

26. The method of claim 25, wherein the at least one computer is configured to display a visual alert by the optical head mounted display or to trigger an acoustic alert when the tissue cutter is within the safe zone, is close to the safe zone, or violates the safe zone.

27. The method of claim 19, wherein the virtual plane intersects with the surface of the joint and wherein the virtual plane is a virtual predetermined path for a saw blade.

28. The method of claim 19, wherein the at least one computer is configured to display, by the see through optical head mounted display, a predetermined removal of tissue or a predetermined tissue resection from the physical joint of the patient.

29. The method of claim 19, comprising displaying using the see through optical head mounted display a virtual saw blade intersecting at least a portion of the joint.

30. A system for preparing a physical joint in a patient comprising:

at least one computer configured to generate a virtual surgical guide, wherein the virtual surgical guide is a virtual plane, wherein the virtual plane is a three-dimensional digital representation indicating at least one of a predetermined resection level, a predetermined resection angle, a predetermined varus or valgus angle, a predetermined flexion, or a predetermined slope, for superimposing one or more physical surgical tools or physical surgical instruments; and

a see through optical head mounted display configured to display, at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion, or predetermined slope, the virtual plane onto a surface of the physical joint visible directly through the see through optical head mounted display,

wherein the virtual plane is configured to allow superimposition of the one or more physical surgical tools or physical surgical instruments with the virtual plane,

wherein the at least one computer is configured to maintain the display of the virtual plane at the at least one of the predetermined resection level, predetermined resection angle, predetermined varus or valgus angle, predetermined flexion, or predetermined slope in relationship to one or more anatomic structures, wherein the one or more anatomic structures are registered in a coordinate system,

wherein the physical joint of the patient, and the one or more physical surgical tools or physical surgical instruments are visible directly through the see through optical head mounted display,

wherein the one or more physical surgical tools or physical surgical instruments are configured to perform one or more bone cuts or a bone removal of the joint,

wherein the virtual surgical guide is registered in the coordinate system, and

wherein the see through optical head mounted display is registered in the coordinate system.