Augmented reality visualization for guiding physical surgical tools and instruments including robotics

• US 10,405,927 B1
• Filed: 05/08/2019
• Issued: 09/10/2019
• Est. Priority Date: 03/12/2016
• Status: Active Grant

• Current Assignee: Philipp K. Lang
• Original Assignee: Philipp K. Lang
• Inventors: Lang, Philipp K.
• Primary Examiner(s): Chang, Olivia C
• Application Number:
  US16/406,392
• Publication Number:
  US 20190262078A1
• Time in Patent Office: 125 Days
• CPC Class Codes: A61B 17/15 Guides therefor arrangements for guiding straight saw blades in general B23D51/025
• CPC Class Codes: A61B 17/155 Cutting femur
• CPC Class Codes: A61B 17/157 Cutting tibia
• CPC Class Codes: A61B 17/1666 for the acetabulum
• CPC Class Codes: A61B 17/17 Guides or aligning means for drills , mills, pins or wires
• CPC Class Codes: A61B 17/1703 using imaging means, e.g. by X-rays
• CPC Class Codes: A61B 17/1742 for the hip
• CPC Class Codes: A61B 17/1764 for the knee
• CPC Class Codes: A61B 17/1775 for the foot or ankle
• CPC Class Codes: A61B 17/1778 for the shoulder
• CPC Class Codes: A61B 2017/00216 with eye tracking or head position tracking control
• CPC Class Codes: A61B 2017/568 produced with shape and dimensions specific for an individual patient
• CPC Class Codes: A61B 2034/104 Modelling the effect of the tool, e.g. the effect of an implanted prosthesis or for predicting the effect of ablation or burring
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2059 Mechanical position encoders
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/368 changing the image on a display according to the operator's position
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 2090/373 : using light, e.g. by using optical scanners
• CPC Class Codes: A61B 2090/374 : NMR or MRI
• CPC Class Codes: A61B 2090/376 : using X-rays, e.g. fluoroscopy
• CPC Class Codes: A61B 2090/3762 : using computed tomography systems [CT]
• CPC Class Codes: A61B 2090/502 : Headgear, e.g. helmet, spectacles
• CPC Class Codes: A61B 34/10 : Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/74 : Manipulators with manual electric input means
• CPC Class Codes: A61B 90/36 : Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/37 : Surgical systems with images on a monitor during operation
• CPC Class Codes: A61F 2/32 : for the hip
• CPC Class Codes: A61F 2/3859 : Femoral components
• CPC Class Codes: A61F 2/389 : Tibial components A61F2/3868 takes precedence
• CPC Class Codes: A61F 2002/4018 : Heads or epiphyseal parts of humerus
• CPC Class Codes: A61F 2002/4205 : Tibial components
• CPC Class Codes: A61F 2002/4207 : Talar components
• CPC Class Codes: Y02A 90/10 : Information and communication technologies [ICT] supporting adaptation to climate change, e.g. for weather forecasting or climate simulation

Abstract

Devices and methods for performing a surgical step or surgical procedure with visual guidance using an optical head mounted display are disclosed.

Claims

1. A system for preparing a physical joint in a patient, the system comprising:

at least one processor,

at least one see through optical head mounted display, and

at least one user interface,

wherein the system is configured to generate a virtual surgical guide, wherein the virtual surgical guide is a virtual axis, wherein the virtual axis is a three-dimensional digital representation indicating a predetermined position, predetermined orientation, predetermined version, or combination thereof in relationship to at least one anatomic structure of the physical joint for at least one physical surgical tool, physical surgical instrument, or combination thereof,

wherein the system is configured to display, by the at least one see through optical head mounted display, at the predetermined position, predetermined orientation, predetermined version, or combination thereof, the virtual surgical guide onto a surface of the physical joint,

wherein the system is configured to adjust the display of the virtual surgical guide responsive to movement of the physical joint to maintain the display of the virtual surgical guide at the predetermined position, predetermined orientation, predetermined version, or combination thereof in relationship to at least one anatomic structure of the physical joint, wherein the system is configured to allow alignment between the at least one physical surgical tool, physical surgical instrument, or combination thereof, and at least a portion of the virtual surgical guide displayed by the at least one see through optical head mounted display,

wherein the at least one physical surgical tool, physical surgical instrument, or combination thereof, is configured to perform a reaming, a drilling, a pinning, a burring, a milling, a broaching or an impacting of the joint,

wherein the at least one see through optical head mounted display is registered in a coordinate system, wherein the at least one anatomic structure is registered in the coordinate system, and wherein the virtual surgical guide is registered in the coordinate system.

2. A system for preparing a physical joint in a patient, the system comprising:

at least one processor,

at least one see through optical head mounted display, and

at least one user interface,

wherein the system is configured to generate a virtual surgical guide, wherein the virtual surgical guide is a virtual axis, wherein the virtual axis is a three-dimensional digital representation indicating a predetermined position, predetermined orientation, predetermined version, or combination thereof in relationship to at least one anatomic structure of the physical joint for at least one physical surgical tool, physical surgical instrument, or combination thereof,

wherein the system is configured to display, by the at least one see through optical head mounted display, at the predetermined position, predetermined orientation, predetermined version, or combination thereof the virtual surgical guide onto a surface of the physical joint,

wherein the system is configured to adjust the display of the virtual surgical guide responsive to movement of the physical joint to maintain the display of the virtual surgical guide at the predetermined position, predetermined orientation, predetermined version, or combination thereof in relationship to at least one anatomic structure of the physical joint,

wherein the system is configured to allow superimposition and alignment between the at least one physical surgical tool, physical surgical instrument, or combination thereof and at least a portion of the virtual surgical guide displayed by the at least one see through optical head mounted display,

wherein the at least one physical surgical tool, physical surgical instrument, or combination thereof is configured to perform a reaming, a drilling, a pinning, a burring, a milling, a broaching or an impacting of the joint,

wherein the at least one see through optical head mounted display is registered in a coordinate system, wherein the at least one anatomic structure is registered in the coordinate system, and wherein the virtual surgical guide is registered in the coordinate system.

3. The system of claim 2 wherein the system is configured to maintain the display at the predetermined position, predetermined orientation, predetermined version, or combination thereof, of the virtual surgical guide on the surface of the physical joint when the at least one physical surgical tool, physical surgical instrument, or combination thereof moves in the field of view of a user.

4. The system of claim 2, wherein the at least one user interface comprises a graphical user interface, a voice recognition, a gesture recognition, a virtual interface displayed by the at least one see through optical head mounted display, a virtual keyboard displayed by the at least one see through optical head mounted display, a physical keyboard, a physical computer mouse, a physical track pad, or combinations thereof.

5. The system of claim 2, wherein the system is configured to adjust the display of the virtual surgical guide responsive to the movement of a first and a second articular side to maintain the display of the virtual surgical guide at the predetermined position, predetermined orientation, predetermined version, or combination thereof in relationship to at least one anatomic structure of the physical joint.

6. The system of claim 2, wherein the virtual axis is a placement indicator of the at least one physical surgical tool, physical surgical instrument, or combination thereof wherein the placement indicator is a three-dimensional digital representation corresponding to at least one portion or an axis of the at least one physical surgical tool, physical surgical instrument, or combination thereof.

7. The system of claim 2, wherein the system is for preparing the physical joint for an implant component, and wherein the implant component is for a knee joint replacement, hip joint replacement, shoulder joint replacement or ankle joint replacement.

8. The system of claim 2, wherein the system is for preparing the physical joint for a reconstruction of an anterior cruciate ligament, and wherein the virtual axis is aligned with at least one of a predetermined femoral tunnel, a predetermined tibial tunnel, a predetermined position of an anterior cruciate ligament graft, or a predetermined orientation of an anterior cruciate ligament graft.

9. The system of claim 2, wherein the predetermined position comprises a predetermined start position, a predetermined start point, a predetermined end position, a predetermined end point, or combinations thereof, or wherein the predetermined orientation comprises a predetermined angle, a predetermined start orientation, a predetermined start alignment, a predetermined end orientation, a predetermined start point and end point, a predetermined end alignment, or combinations thereof.

10. The system of claim 2, wherein the predetermined position, predetermined orientation, predetermined version or combination thereof, of the virtual axis is configured to determine a resection level, a flexion angle, a varus or valgus alignment, or a rotation of a femoral component or a resection level, a slope, a varus or valgus alignment, or a rotation of a tibial component in a knee joint replacement, or a position, orientation, inclination, or anteversion of an acetabular component or a position, orientation, offset, version or leg length of a femoral component in a hip joint replacement, or a position, orientation or version of a glenoid or a humeral component in a shoulder joint replacement, or a position, orientation, or rotation of a tibial or a talar component in an ankle joint replacement.

11. The system of claim 2, wherein the at least one physical surgical tool, physical surgical instrument, or combination thereof comprises a drill, a pin, a burr, a mill, an awl, a reamer, an impactor, a broach, a saw, a saw blade, a cutting tool, a cut block, a pin guide, a tibial template, a tibial drill tower, a tibial keel punch or combinations thereof.

12. The system of claim 2, wherein the system is configured to obtain coordinates of the at least one anatomic structure using an infrared marker, a radiofrequency marker, an IMU, an LED, an optical marker, a geometric pattern, a patient specific marker, a patient specific template, a surgical navigation system, a camera, a video system, an image capture system, a depth sensor, a laser scanner, a 3D scanner, an x-ray, an imaging system or combinations thereof, and wherein the coordinates are registered in the coordinate system.

13. The system of claim 2, wherein the at least one anatomic structure comprises an anatomic landmark, anatomic plane, articular surface, cartilage surface, subchondral bone surface, cortical bone surface, cut bone surface, reamed bone surface, milled bone surface, impacted bone surface, tissue resection, tissue surface, or combinations thereof.

14. The system of claim 2, wherein the at least one anatomic structure comprises information about an anatomic axis, a biomechanical axis, a mechanical axis, a distance between two or more anatomic landmarks, a dimension between two or more anatomic landmarks, a dimension between an anterior and a posterior anatomic structure, a dimension between a medial and a lateral anatomic structure, a dimension between a superior and an inferior anatomic structure, a curvature, a surface, an edge, a shape, a length, a width, a depth, or combinations thereof.

15. The system of claim 13, wherein the at least one anatomic landmark comprises one or more of a portion of or an entire acetabulum, a portion of or an entire edge of an acetabulum, multiple portions of an edge of an acetabulum, a portion of an iliac wall, a portion of a pubic bone, a portion of an ischial bone, an anterior superior iliac spine, an anterior inferior iliac spine, a symphysis pubis, a portion of or an entire greater trochanter, a portion of or an entire lesser trochanter, a center of the femoral head, a portion of or an entire femoral shaft, a portion of or an entire femoral neck, a portion of or an entire femoral head, a fovea capitis, a transverse acetabular ligament, a pulvinar, a ligamentum teres, a labrum, one or more femoral osteophytes, one or more acetabular osteophytes, a portion of or an entire medial femoral condyle, a portion of or an entire lateral femoral condyle, a portion of or an entire femoral notch, an origin of an anterior cruciate ligament, an insertion of an anterior cruciate ligament, a portion of or an entire trochlea, a portion of an anterior cortex of the femur, a portion of an anterior cortex of the femur with adjacent portions of the trochlea, a portion of an anterior cortex of the femur with adjacent portions of the trochlea and osteophytes, one or more femoral osteophytes, one or more tibial osteophytes, an epicondylar eminence, a medial epicondylar eminence, a lateral epicondylar eminence, a portion of or an entire medial tibial plateau, a portion of or an entire lateral tibial plateau, a portion of or an entire medial tibial spine, a portion of or an entire lateral tibial spine, a portion of an anterior cortex of the tibia, a portion of an anterior cortex of the tibia and a portion of a tibial plateau, a portion of or an entire patella, a medial edge of a patella, a lateral edge of a patella, a superior pole of a patella, an inferior pole of a patella, a patellar osteophyte, a medial wall of the femoral notch, a lateral wall of the femoral notch, a roof of the femoral notch, a medial wall of the medial condyle, a lateral wall of the lateral condyle, a medial femoral condyle shape, a medial femoral condyle radius, a medial femoral condyle convexity, a medial femoral condyle concavity, a lateral femoral condyle shape, a lateral femoral condyle radius, a lateral femoral condyle convexity, a lateral femoral condyle concavity, an intercondylar notch shape, an intercondylar notch surface feature, a medial tibial spine, a lateral tibial spine, an anteromedial tibial rim, an anterolateral tibial rim, a medial tibial rim, a lateral tibial rim, a lowest point of the medial plateau, a lowest point of the lateral plateau, a highest point of the medial plateau, a highest point of the lateral plateau, a medial tibial plateau shape, a lateral tibial plateau shape, a medial tibial plateau surface, a medial tibial plateau surface feature, a medial tibial plateau radius, a medial tibial plateau convexity, a medial tibial plateau concavity, a lateral tibial plateau shape, a lateral tibial plateau surface, a lateral tibial plateau surface feature, a lateral tibial plateau radius, a lateral tibial plateau convexity, a lateral tibial plateau concavity, a portion of or an entire glenoid, a portion of or an entire coracoid process, a portion of or an entire acromion, a portion of a clavicle, a portion of or an entire humeral head, a portion of or an entire humeral neck, a portion of a humeral shaft, one or more humeral osteophytes, one or more glenoid osteophytes, a portion of or an entire glenoid labrum, a portion of or an entire shoulder ligament, a portion of or an entire coracoacromial ligament, or a portion of or an entire glenohumeral ligament.

16. The system of claim 2, wherein the surface of the physical joint comprises one or more of a bone, a cartilage, an opposing joint surface, a skin, a subcutaneous tissue, a ligament, a ligament remnant, a meniscus, a labrum, or an intra-articular structure.

17. The system of claim 2, wherein the movement of the physical joint of the patient includes movement of one or more of a cartilage surface, a cortical bone surface, a subchondral bone surface, an articular surface, at least a portion of one or more of an osteophyte, a cut bone surface, a reamed bone surface, a milled bone surface, an impacted bone surface, a burred articular surface, a surgically altered surface, a skin, a subcutaneous tissue, an acetabulum, an edge of an acetabulum, an iliac wall, a pubic bone, an ischial bone, an anterior superior iliac spine, an anterior inferior iliac spine, a symphysis pubis, a greater trochanter, a lesser trochanter, a femoral shaft, a femoral neck, a femoral head, a fovea capitis, a transverse acetabular ligament, a pulvinar, a ligamentum teres, a labrum, a femoral condyle, a femoral notch, a trochlea, an anterior cortex of the femur, an epicondylar eminence, a medial tibial plateau, a lateral tibial plateau, a tibial spine, a cortex of the tibia, a patella, an anterior cruciate ligament, a posterior cruciate ligament, a medial collateral ligament, a lateral collateral ligament, a meniscus, a glenoid, a coracoid process, an acromion, a clavicle, a humeral head, a humeral neck, a humeral shaft, a glenoid labrum, a shoulder ligament, a distal tibia, or a talus.

18. The system of claim 2, wherein one or more markers are attached to a first articular side and a second articular side, wherein the position and/or orientation of the one or more markers on the first articular side and the second articular side is tracked during movement of the joint, and wherein the one or more markers comprise one or more of an IMU, an optical marker, a geometric pattern, a bar code, a QR code, a radiofrequency marker, an infrared marker, a retroreflective marker, an active marker, or a passive marker.

19. The system of claim 2, wherein the system is configured to track the at least one physical surgical tool, physical surgical instrument, or combination thereof in the coordinate system.

20. The system of claim 19, wherein the system is configured to display, by the at least one see through optical head mounted display, the virtual surgical guide with one or more colors when the at least one physical surgical tool, physical surgical instrument, or combination thereof, deviates from the predetermined position, predetermined orientation, predetermined version, or combination thereof.

21. The system of claim 2, wherein the at least one physical surgical tool, physical surgical instrument, or combination thereof, is handheld.

22. The system of claim 21, wherein the at least one physical surgical tool, physical surgical instrument, or combination thereof is attached to a robot.

23. The system of claim 22, wherein the system is configured to display, by the least one see through optical head mounted display, a safe zone onto the surface of the physical joint, wherein the safe zone is indicative of a predetermined operating range for the at least one physical surgical tool, physical surgical instrument, or combination thereof.

24. The system of claim 23, wherein the system is configured to display, by the at least one see through optical head mounted display, a visual alert, and/or wherein the system is configured to trigger an acoustic alert when the at least one physical surgical tool, physical surgical instrument, or combination thereof, is within the safe zone or outside the safe zone.

25. The system of claim 22, wherein the robot has a robotic arm.

26. The system of claim 22, wherein the robot has an attached physical drill, burr, mill, saw, cut block, pin guide, or reamer.

27. The system of claim 2, wherein the system is configured to display the virtual surgical guide continuously.

28. The system of claim 2, wherein the system is configured to display the virtual surgical guide intermittently.

29. The system of claim 2, wherein the at least one see through optical head mounted display comprises the at least one processor.

30. The system of claim 2, wherein the at least one processor is separate from the at least one see through optical head mounted display.