System and method for three-dimensional augmented reality guidance for use of medical equipment

• US 10,636,323 B2
• Filed: 01/23/2018
• Issued: 04/28/2020
• Est. Priority Date: 01/24/2017
• Status: Active Grant

• Current Assignee: Tienovix, LLC
• Original Assignee: Tienovix, LLC
• Inventors: Buras, William R., Russell, Craig S., Nguyen, Kyle Q.
• Primary Examiner(s): Fernstrom, Kurt
• Application Number:
  US15/878,314
• Publication Number:
  US 20180225993A1
• Time in Patent Office: 826 Days
• Field of Search: 434262, 434267, 434268, 434272, 600424
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/378 using ultrasound
• CPC Class Codes: A61B 2090/3945 Active visible markers, e.g. light emitting diodes
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 8/06 Measuring blood flow measuring volume flow in general G01F, e.g. G01F1/66, G01F1/72; measuring speed of fluids in general G01P5/00
• CPC Class Codes: A61B 8/085 for locating body or organic structures, e.g. tumours, calculi, blood vessels, nodules
• CPC Class Codes: A61B 8/4245 involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient
• CPC Class Codes: A61B 8/4254 using sensors mounted on the probe
• CPC Class Codes: A61B 8/4263 using sensors not mounted on the probe, e.g. mounted on an external reference frame
• CPC Class Codes: A61B 8/466 adapted to display 3D data 3D imaging and stereoscopic displays in devices for short-range imaging with acoustic waves G01S15/8993 and G01S7/52068, respectively; 3D image rendering G06T15/00; manipulating 3D models for computer graphics G06T19/00
• CPC Class Codes: A61B 8/467 characterised by special input means
• CPC Class Codes: A61B 8/52 Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves image processing per se G06T
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G09B 23/286 : for scanning or photography techniques, e.g. X-rays, ultrasonics
• CPC Class Codes: G09B 5/065 : Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 40/63 : for local operation

Abstract

A medical guidance system providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a novice user of medical equipment having limited medical training, to achieve improved diagnostic or treatment outcomes.

Claims

1. A medical guidance system for providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a user of a medical equipment system, the medical guidance system comprising:

a medical equipment interface to a medical equipment system, wherein said medical equipment interface is capable of receiving real-time data from the medical equipment system during a medical procedure performed by a user, wherein the real-time data is data relating to an outcome of the medical procedure;

an augmented reality user interface (ARUI) for presenting data pertaining to both real and virtual objects to the user during at least a portion of the performance of the medical procedure;

a three-dimensional guidance system (3DGS) that is capable of sensing real-time user positioning data relating to one or more of the movement, position, and orientation of at least a portion of the medical equipment system during said medical procedure performed by the user;

a library containing

     1) stored reference positioning data relating to one or more of the three-dimensional movement, position, and orientation of at least a portion of the medical equipment system during a reference medical procedure and

     2) stored reference outcome data relating to an outcome of said reference medical procedure; and

a machine learning module (MLM) for providing

     1) real-time position-based 3D AR feedback to the user based on the sensed user positioning data and the reference positioning data, and

     2) real-time outcome-based 3D AR feedback to the user based on comparing the real-time data relating to an outcome of the medical procedure received from the medical equipment system during the medical procedure performed by the user to reference outcome data.

2. The medical guidance system of claim 1, further comprising a computer system containing said medical equipment interface, wherein the medical equipment system comprises an ultrasound system coupled to the computer system via the medical equipment interface.

3. The medical guidance system of claim 2, wherein the 3DGS senses the user'"'"'s real-time movement of an ultrasound probe relative to a body of a patient, and the stored reference outcome data comprises ultrasound images.

4. The medical guidance system of claim 1 further comprising a computer system containing said medical equipment interface to the medical equipment system, wherein said computer system is coupled to one or more of the ARUI, the 3DGS, the library, and the MLM.

5. The medical guidance system of claim 1, wherein the 3DGS comprises one or more of a magnetic GPS system, a digital camera tracking system, an infrared camera system, an accelerometer, and a gyroscope.

6. The medical guidance system of claim 1, wherein the 3DGS senses real-time user positioning data by sensing at least one of:

a magnetic field generated by said at least a portion of the medical equipment system;

the movement of one or more passive visual markers coupled to one or more of a patient, a hand of the user, or a portion of the medical equipment system; and

the movement of one or more active visual markers coupled to one or more of a patient, a hand of the user, or a portion of the medical equipment system.

7. The medical guidance system of claim 1, wherein the library further includes one or more of:

instructions for performing one or more medical procedures using the medical equipment system;

generic data associated with the use of the medical equipment system on a number of different patients; and

patient-specific data relating to the use of the medical equipment system on one or more specific patients.

8. The medical guidance system of claim 1, wherein at least one of said real-time position-based 3D AR feedback and said real-time outcome-based 3D AR feedback is selected from a virtual still image, a virtual video image, sounds, or tactile information.

9. The medical guidance system of claim 1, wherein at least one of said real-time position-based 3D AR feedback and said real-time outcome-based 3D AR feedback is selected from:

a virtual prompt indicating a movement correction to be performed by a user;

a virtual image or video instructing the user to change the orientation of the at least a portion of the medical equipment system to match a desired orientation;

a virtual image or video of a correct motion path to be taken by the user in performing a medical procedure;

a color-coded image or video indicating correct and incorrect portions of the user'"'"'s motion in performing a medical procedure;

an instruction to a user to press an ultrasound probe deeper or shallower into tissue to focus the ultrasound image on a desired target structure of the patient'"'"'s body; and

an auditory instruction, virtual image, or virtual video indicating a direction for the user to move the at least a portion of the medical equipment system.

10. The medical guidance system of claim 1 further comprising:

a computer system containing said interface to the medical equipment system, wherein said ARUI comprises a software interface located in said computer; and

a head mounted display (HMD) coupled to said ARUI.

11. The medical guidance system of claim 1, wherein the three-dimensional guidance system (3DGS) is capable of sensing real-time user positioning data relating to at least a portion of the medical equipment system within a volume of the user'"'"'s environment during said medical procedure performed by the user.

12. The medical guidance system of claim 1, wherein the real-time outcome-based feedback comprises a feedback selected from

feedback that is different from the position-based feedback, and

feedback that is in addition to the position-based feedback.

13. The medical guidance system of claim 1, wherein

the medical equipment system comprises an ultrasound system;

the medical procedure comprises an ultrasound procedure performed by the user;

the reference medical procedure comprises an ultrasound procedure performed by one of an expert user or a credentialed user;

the reference outcome data comprises reference ultrasound images captured during one or more reference medical procedures; and

the real-time outcome-based feedback comprises feedback based on comparing one or more reference ultrasound images to images generated in real-time by the ultrasound system during the ultrasound procedure performed by the user.

14. The medical guidance system of claim 13, wherein the real-time outcome-based feedback is generated by a neural network.

15. The medical guidance system of claim 1, wherein the MLM provides real-time outcome-based feedback comprising an indication of whether an acceptable outcome to the medical procedure has been achieved.

16. A medical guidance system for providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance in the use of a medical equipment system, the medical guidance system comprising:

a computer comprising

a medical equipment interface to a medical equipment system, wherein said medical equipment interface receives real-time data from the medical equipment system during a medical procedure performed by a user to achieve a medical procedure outcome, wherein the real-time data is data relating to the medical procedure outcome;

an AR interface to an AR head mounted display (HMD) for presenting information pertaining to both real and virtual objects to the user during the performance of the medical procedure;

a guidance system interface (GSI) to a three-dimensional guidance system (3DGS) that senses real-time user positioning data relating to one or more of the three-dimensional movement, position, and orientation of at least a portion of the medical equipment system within a volume of a user'"'"'s environment during a medical procedure performed by the user;

a library containing

     1) stored reference positioning data relating to one or more of the three-dimensional movement, position, and orientation of at least a portion of the medical equipment system during a reference medical procedure and

     2) stored reference outcome data relating to an outcome of a reference performance of the reference medical procedure; and

a machine learning module (MLM) for providing

     1) real-time position-based 3D AR feedback to the user based on the sensed user positioning data and

     2) real-time outcome-based 3D AR feedback to the user based on the real-time data relating to the medical procedure outcome, the MLM comprising

a position-based feedback module comprising

a first module for receiving and analyzing real-time user positioning data;

a second module for comparing the user positioning data to the stored reference positioning data, and

a third module for generating real-time position-based 3D AR feedback based on the output of the second module, and providing said real-time position-based 3D AR feedback to the user via the ARUI; and

an outcome-based feedback module comprising

a fourth module for receiving the real-time data relating to an outcome of the medical procedure from the medical equipment system via said medical equipment interface as the user performs the medical procedure;

a fifth module for comparing the real-time data received from the medical equipment system as the user performs the medical procedure to the stored reference outcome data, and

a sixth module for generating real-time outcome-based 3D AR feedback based on the output of the fifth module, and providing said real-time outcome-based 3D AR feedback to the user via the ARUI.

17. The medical guidance system of claim 16, further comprising the AR HMD.

18. The medical guidance system of claim 16, wherein at least one of the fifth module and the sixth module comprises a neural network.

19. The medical guidance system of claim 18, wherein the neural network is a convolutional neural network.

20. The medical guidance system of claim 16, wherein said third module provides said real-time position-based 3D AR feedback to the user visually in an AR HMD relative to at least a portion of the patient'"'"'s body, and said sixth module provides said real-time outcome-based 3D AR feedback to the user visually in the AR HMD relative to at least a portion of the patient'"'"'s body.