System and method for three-dimensional augmented reality guidance for use of equipment

• US 10,796,605 B2
• Filed: 12/26/2019
• Issued: 10/06/2020
• Est. Priority Date: 01/24/2017
• Status: Active Grant

• Current Assignee: Tienovix, LLC
• Original Assignee: Tienovix, LLC
• Inventors: Buras, William R., Russell, Craig S., Nguyen, Kyle Q.
• Primary Examiner(s): Fernstrom, Kurt
• Application Number:
  US16/727,840
• Publication Number:
  US 20200135055A1
• Time in Patent Office: 285 Days
• Field of Search: 434262, 434267, 434268, 434272, 600424
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/378 using ultrasound
• CPC Class Codes: A61B 2090/3945 Active visible markers, e.g. light emitting diodes
• CPC Class Codes: A61B 2562/0219 Inertial sensors, e.g. accelerometers, gyroscopes, tilt switches
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 8/06 Measuring blood flow measuring volume flow in general G01F, e.g. G01F1/66, G01F1/72; measuring speed of fluids in general G01P5/00
• CPC Class Codes: A61B 8/085 for locating body or organic structures, e.g. tumours, calculi, blood vessels, nodules
• CPC Class Codes: A61B 8/4245 involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient
• CPC Class Codes: A61B 8/4254 using sensors mounted on the probe
• CPC Class Codes: A61B 8/4263 using sensors not mounted on the probe, e.g. mounted on an external reference frame
• CPC Class Codes: A61B 8/466 adapted to display 3D data 3D imaging and stereoscopic displays in devices for short-range imaging with acoustic waves G01S15/8993 and G01S7/52068, respectively; 3D image rendering G06T15/00; manipulating 3D models for computer graphics G06T19/00
• CPC Class Codes: A61B 8/467 characterised by special input means
• CPC Class Codes: A61B 8/52 Devices using data or image processing specially adapted for diagnosis using ultrasonic, sonic or infrasonic waves image processing per se G06T
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G02B 27/017 Head mounted
• CPC Class Codes: G06F 3/011 Arrangements for interaction with the human body, e.g. for user immersion in virtual reality blind teaching G09B21/00
• CPC Class Codes: G06F 3/016 Input arrangements with force or tactile feedback as computer generated output to the user
• CPC Class Codes: G06N 20/00 Machine learning
• CPC Class Codes: G09B 23/286 : for scanning or photography techniques, e.g. X-rays, ultrasonics
• CPC Class Codes: G09B 5/065 : Combinations of audio and video presentations, e.g. videotapes, videodiscs, television systems
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 40/63 : for local operation

Abstract

Methods for providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a user of an equipment system to achieve improved outcomes in the use of the equipment. The methods involve providing real-time real-time position-based 3D AR feedback and the real-time outcome-based 3D AR feedback to the user via an augmented reality user interface (ARUI). The feedback may be provided to the user via a head mounted display (HMD).

Claims

1. A method for providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a user of an equipment system, the method comprising:

receiving data from an equipment system during an equipment procedure performed by a user of the equipment system to achieve an equipment procedure outcome;

sensing real-time user positioning data relating to one or more of the three-dimensional movement, position, and orientation of at least a portion of the equipment system within a volume of the user'"'"'s environment during the equipment procedure performed by the user, wherein the sensing comprises sensing at least one of

     1) a magnetic field generated by said at least a portion of the equipment system;

     2) the movement of one or more passive visual markers coupled to one or more of the user and a portion of the equipment system; and

     3) the movement of one or more active visual markers coupled to one or more of the user and a portion of the equipment system;

retrieving from a library

     1) stored reference positioning data relating to one or more of the three-dimensional movement, position, and orientation of at least a portion of the equipment system during a reference equipment procedure, and

     2) stored reference outcome data relating to an outcome of said reference equipment procedure;

comparing

     1) the sensed real-time user positioning data to the retrieved reference positioning data, and

     2) the data received from the equipment system during the equipment procedure performed by the user to the retrieved reference outcome data;

generating real-time position-based 3D AR feedback based on the comparison of the sensed real-time user positioning data to the retrieved reference positioning data;

generating real-time outcome-based 3D AR feedback based on the comparison of the data received from the equipment system during the equipment procedure performed by the user to the retrieved reference outcome data; and

providing each of the real-time position-based 3D AR feedback and the real-time outcome-based 3D AR feedback to the user via an augmented reality user interface (ARUI), wherein providing at least one of the real-time position-based 3D AR feedback and the real-time outcome-based 3D AR feedback to the user comprises providing a feedback selected from

     1) a virtual prompt indicating a movement correction to be performed by the user;

     2) a virtual image or video instructing the user to change the orientation of the at least a portion of the equipment system to match a desired orientation;

     3) a virtual image or video of a correct motion path to be taken by the user in performing the equipment procedure;

     4) a color-coded image or video indicating one or more correct portions of the user'"'"'s motion in performing the equipment procedure;

     5) a color-coded image or video indicating one or more incorrect portions of the user'"'"'s motion in performing the equipment procedure;

     6) an auditory instruction, virtual image, or virtual video indicating a direction for the user to move at least a portion of the equipment system; and

     7) tactile information.

2. The method of claim 1, wherein providing each of the real-time position-based 3D AR feedback and the real-time outcome-based 3D AR feedback to the user comprises providing said feedback to a head mounted display (HMD) worn by the user.

3. The method of claim 2, wherein providing each of the real-time position-based 3D AR feedback and the real-time outcome-based 3D AR feedback to the user comprises providing at least one of 1) real-time position-based 3D AR feedback to the user visually in a HMD relative to at least a portion of the equipment system, and 2) real-time outcome-based 3D AR feedback visually in a HMD relative to at least a portion of the equipment system.

4. The method of claim 1, wherein receiving data from an equipment system during an equipment procedure performed by a user comprises receiving equipment procedure image data during a first equipment procedure performed by a user, and wherein retrieving stored reference outcome data comprises retrieving stored equipment procedure images obtained during a reference performance of the first equipment procedure by at least one of an expert user and a credentialed user of the equipment prior to the user'"'"'s performance of the first equipment procedure.

5. The method of claim 4, wherein generating real-time outcome-based 3D AR feedback is based on a comparison, using a neural network, of images generated by the user during an equipment procedure to retrieved images generated during a reference performance of the same equipment procedure prior to the user'"'"'s performance of the equipment procedure.

6. The method of claim 5, wherein the comparison is performed by a convolutional neural network.

7. A method for providing real-time, three-dimensional (3D) augmented reality (AR) feedback guidance to a user of an equipment system, the method comprising:

receiving data from an equipment system during an equipment procedure performed by a user of the equipment system to achieve an equipment system outcome;

sensing real-time user positioning data relating to one or more of the three-dimensional movement, position, and orientation of at least a portion of the equipment system within a volume of the user'"'"'s environment during the equipment procedure performed by the user;

retrieving from a library at least one of

     1) stored reference positioning data relating to one or more of the three-dimensional movement, position, and orientation of at least a portion of the equipment system during a reference equipment procedure, and

     2) stored reference outcome data relating to an outcome of said reference equipment procedure;

comparing at least one of

     1) the sensed real-time user positioning data to the retrieved reference positioning data, and

     2) the data received from the equipment system during the equipment procedure performed by the user to the retrieved reference outcome data;

generating real-time position-based 3D AR feedback based on the comparison of the sensed real-time user positioning data to the retrieved reference positioning data;

generating real-time outcome-based 3D AR feedback based on the comparison of the data received from the equipment system during the equipment procedure performed by the user to the retrieved reference outcome data; and

providing each of the real-time position-based 3D AR feedback and the real-time outcome-based 3D AR feedback to a head mounted display (HMD) worn by the user via an augmented reality user interface (ARUI).

8. The method of claim 7, wherein providing each of the real-time position-based 3D AR feedback and the real-time outcome-based 3D AR feedback to the user comprises providing at least one feedback selected from 1) a virtual prompt indicating a movement correction to be performed by the user;

2) a virtual image or video instructing the user to change the orientation of the at least a portion of the equipment system to match a desired orientation;

     3) a virtual image or video of a correct motion path to be taken by the user in performing the equipment procedure;

     4) a color-coded image or video indicating one or more correct portions of the user'"'"'s motion in performing the equipment procedure;

     5) a color-coded image or video indicating one or more incorrect portions of the user'"'"'s motion in performing the equipment procedure;

     6) an auditory instruction, virtual image, or virtual video indicating a direction for the user to move at least a portion of the equipment system; and

     7) tactile information.