System and method for virtual reality data integration and visualization for 3D imaging and instrument position data

• US 10,258,426 B2
• Filed: 03/12/2018
• Issued: 04/16/2019
• Est. Priority Date: 03/21/2016
• Status: Active Grant

• Current Assignee: Washington University In St Louis
• Original Assignee: Washington University In St Louis
• Inventors: Silva, Jonathan, Silva, Jennifer
• Primary Examiner(s): Roy, Baisakhi
• Application Number:
  US15/918,418
• Publication Number:
  US 20180200018A1
• Time in Patent Office: 400 Days
• CPC Class Codes: A61B 18/02 by cooling, e.g. cryogenic techniques devices for cooling specific reflex points of the body within cell-life limits A61H39/06
• CPC Class Codes: A61B 18/1492 having a flexible, catheter-like structure, e.g. for heart ablation A61B18/1477 takes precedence
• CPC Class Codes: A61B 2017/00207 with hand gesture control or hand gesture recognition
• CPC Class Codes: A61B 2018/00351 Heart
• CPC Class Codes: A61B 2018/00577 Ablation
• CPC Class Codes: A61B 2018/00642 with feedback, i.e. closed loop control
• CPC Class Codes: A61B 2018/00839 Bioelectrical parameters, e.g. ECG, EEG
• CPC Class Codes: A61B 2018/0212 using an instrument inserted into a body lumen, e.g. catheter
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/107 Visualisation of planned trajectories or target regions
• CPC Class Codes: A61B 2034/2051 Electromagnetic tracking systems
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 2034/2063 Acoustic tracking systems, e.g. using ultrasound
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/252 indicating steps of a surgical procedure
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/367 creating a 3D dataset from 2D images using position information
• CPC Class Codes: A61B 2090/368 changing the image on a display according to the operator's position
• CPC Class Codes: A61B 2090/376 using X-rays, e.g. fluoroscopy
• CPC Class Codes: A61B 2090/3782 transmitter or receiver in catheter or minimal invasive instrument
• CPC Class Codes: A61B 2090/502 : Headgear, e.g. helmet, spectacles
• CPC Class Codes: A61B 34/20 : Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 5/066 : Superposing sensor position on an image of the patient, e.g. obtained by ultrasound or x-ray imaging
• CPC Class Codes: A61B 5/283 : Invasive
• CPC Class Codes: A61B 5/743 : Displaying an image simultaneously with additional graphical information, e.g. symbols, charts, function plots
• CPC Class Codes: A61B 5/745 : using a holographic display
• CPC Class Codes: A61B 90/361 : Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 : Surgical systems with images on a monitor during operation
• CPC Class Codes: G06F 3/017 : Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06T 15/205 : Image-based rendering
• CPC Class Codes: G06T 17/20 : Finite element generation, e.g. wire-frame surface description, tesselation
• CPC Class Codes: G06T 19/00 : Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/006 : Mixed reality object pose determination, tracking or camera calibration for mixed reality G06T7/00
• CPC Class Codes: G06T 2210/41 : Medical
• CPC Class Codes: G06T 2219/004 : Annotating, labelling
• CPC Class Codes: G06T 2219/024 : Multi-user, collaborative environment

Abstract

Systems and methods for virtual reality or augmented reality (VR/AR) visualization of 3D medical images using a VR/AR visualization system are disclosed. The VR/AR visualization system includes a computing device operatively coupled to a VR/AR device, and the VR/AR device includes a holographic display and at least one sensor. The holographic display is configured to display a holographic image to an operator. The computing device is configured to receive at least one stored 3D image of a subject'"'"'s anatomy and at least one real-time 3D position of at least one surgical instrument. The computing device is further configured to register the at least one real-time 3D position of the at least one surgical instrument to correspond to the at least one 3D image of the subject'"'"'s anatomy, and to generate the holographic image comprising the at least one real-time position of the at least one surgical instrument overlaid on the at least one 3D image of the subject'"'"'s anatomy.

Claims

1. A system comprising:

a catheter configured to be inserted inside a cardiovascular organ of a subject, the catheter comprising one or more sensors generating a first set of sensor data;

a head-mounted display (HMD) configured to be worn by an operator and to display an electroanatomic visualization representing the cardiovascular organ in 3D in a field of view of the operator in response to receiving 3D images, wherein the 3D images include at least a first 3D image and a second 3D image;

an input device configured to generate a second set of sensor data responsive to a motion of the operator; and

one or more computers communicatively coupled to the catheter to receive the first set of sensor data, communicatively coupled to the input device to receive the second set of sensor data, and communicatively coupled to the HMD via a wireless data exchange protocol, wherein the one or more computers comprises one or more processors and computer program instructions that when executed cause the one or more processors to;

generate the first 3D image by processing the first set of sensor data;

provide the first 3D image to the HMD;

receive the second set of sensor data responsive to the motion of the operator to interact with the electroanatomic visualization;

determine a path of the motion of the operator by processing the second set of sensor data;

determine an angle of rotation for the electroanatomic visualization based on the path of the motion; and

provide the second 3D image to the HMD to update the display of the electroanatomic visualization by rotating the cardiovascular organ by the angle of rotation in the field of view of the operator.

2. The system of claim 1, wherein the operator performs at least one of the motions using a hand of the operator, and wherein the hand does not physically contact the input device while the operator performs the at least one of the motions.

3. The system of claim 1, wherein the one or more sensors include at least one of an impedance sensor, a magnetic sensor, and an ultrasound sensor.

4. The system of claim 1, wherein the one or more computers include further computer program instructions that when executed cause the one or more processors to:

provide a third 3D image to the HMD to display an electrocardiogram of the cardiovascular organ in the field of view of the operator;

receive from the input device, a third set of sensor data indicative of subsequent motion of the body part of the operator to position one or more virtual markers on the electrocardiogram;

determine a metric based on positions of the one or more virtual markers by processing the third set of sensor data; and

provide the metric to the HMD to update the display of the electrocardiogram in the field of view of the operator.

5. The system of claim 4, wherein generating the first 3D image of the electroanatomic visualization comprises:

mapping a first plurality of points of the first set of sensor data to a second plurality of points of the third set of sensor data in 3D, wherein the first set of sensor data and the third set of sensor data are generated in different physical units.

6. The system of claim 1, wherein generating the first 3D image of the electroanatomic visualization comprises:

generating a series of connected 2D polygons to represent 3D geometries of the cardiovascular organ.

7. The system of claim 1, wherein the one or more computers include further computer program instructions that when executed cause the one or more processors to:

provide, simultaneously with providing the second 3D image to the HMD, a third 3D image to another HMD worn by another operator to display the electroanatomic visualization rotated by the angle of rotation in another field of view of the another operator, wherein the another field of view is positioned at an offset angle from the field of view.

8. A method comprising:

generating a first 3D image of an electroanatomic visualization representing a cardiovascular organ of a subject in 3D by processing a first set of sensor data generated by a catheter inserted inside the cardiovascular organ;

providing the first 3D image to a head-mounted display (HMD) worn by an operator to display the electroanatomic visualization in a field of view of the operator;

receiving from an input device, a second set of sensor data indicative of motion of a body part of the operator to interact with the electroanatomic visualization;

determining a path of the motion of the body part by processing the second set of sensor data;

determining an angle of rotation for the electroanatomic visualization based on the path of motion; and

providing a second 3D image to the HMD to update the display of the electroanatomic visualization by rotating the cardiovascular organ by the angle of rotation in the field of view of the operator.

9. The method of claim 8, further comprising:

receiving a third set of sensor data indicative of another motion of the operator to select a command of a plurality of commands; and

determining that the selected command corresponds to a rotation command by processing the third set of sensor data.

10. The method of claim 9, wherein the third set of sensor data is received from the HMD, and wherein the another motion is motion of a head of the operator.

11. The method of claim 8, wherein the input device is physically connected to the HMD.

12. The method of claim 8, further comprising:

providing a third 3D image to the HMD to display an electrocardiogram of the cardiovascular organ in the field of view of the operator;

receiving from the input device, a third set of sensor data indicative of subsequent motion of the body part of the operator to position one or more virtual markers on the electrocardiogram;

determining a metric based on positions of the one or more virtual markers by processing the third set of sensor data; and

providing the metric to the HMD to update the display of the electrocardiogram in the field of view of the operator.

13. The method of claim 8, further comprising:

receiving from the input device, a third set of sensor data indicative of subsequent motion of the body part of the operator;

determining an increase in a level of zoom of the electroanatomic visualization responsive to the subsequent motion; and

providing a third 3D image to the HMD to display the electroanatomic visualization including one or more additional components in the field of view of the operator.

14. The method of claim 8, wherein the electroanatomic visualization further includes a representation of a portion of the catheter positioned relative to the cardiovascular organ in 3D.

15. The method of claim 14, further comprising:

receiving a third set of sensor data from the catheter indicative of detected motion of the catheter; and

providing a third 3D image to the HMD to update the display of the electroanatomic visualization by moving the portion of the catheter based on the detected motion in the field of view of the operator.

16. The method of claim 8, wherein the first 3D image is generated further by aggregating the first set of sensor data with a third set of sensor data generated by at least one of an impedance sensor, a magnetic sensor, and an ultrasound sensor.

17. The method of claim 16, wherein generating the first 3D image of the electroanatomic visualization comprises:

mapping a first plurality of points of the first set of sensor data to a second plurality of points of the third set of sensor data in 3D, wherein the first set of sensor data and the third set of sensor data are generated in different physical units.

18. The method of claim 8, wherein generating the first 3D image of the electroanatomic visualization comprises:

generating a series of connected 2D polygons to represent 3D geometries of the cardiovascular organ.

19. The method of claim 8, further comprising:

providing, simultaneously with providing the second 3D image to the HMD, a third 3D image to another HMD worn by another operator to display the electroanatomic visualization rotated by the angle of rotation in another field of view of the another operator, wherein the another field of view is positioned at an offset angle from the field of view.

20. A non-transitory computer readable storage medium having instructions encoded thereon that, when executed by a processor, cause the processor to perform steps including:

generating a first 3D image of an electroanatomic visualization representing a cardiovascular organ of a subject in 3D by processing a first set of sensor data generated by a catheter inserted inside the cardiovascular organ;

providing the first 3D image to a head-mounted display (HMD) worn by an operator to display the electroanatomic visualization in a field of view of the operator;

receiving from an input device, a second set of sensor data indicative of motion of a body part of the operator to interact with the electroanatomic visualization;

determining a path of the motion of the body part by processing the second set of sensor data;

determining an angle of rotation for the electroanatomic visualization based on the path of motion; and

providing a second 3D image to the HMD to update the display of the electroanatomic visualization by rotating the cardiovascular organ by the angle of rotation in the field of view of the operator.

21. The computer readable storage medium of claim 20, further comprising instructions that, when executed by the processor, cause the processor to perform steps including:

receiving a third set of sensor data indicative of another motion of the operator to select a command of a plurality of commands; and

determining that the selected command corresponds to a rotation command by processing the third set of sensor data.

22. The computer readable storage medium of claim 21, wherein the third set of sensor data is received from the HMD, and wherein the another motion is motion of a head of the operator.

23. The computer readable storage medium of claim 20, wherein the input device is physically connected to the HMD.

24. The computer readable storage medium of claim 20, further comprising instructions that, when executed by the processor, cause the processor to perform steps including:

providing a third 3D image to the HMD to display an electrocardiogram of the cardiovascular organ in the field of view of the operator;

receiving from the input device, a third set of sensor data indicative of subsequent motion of the body part of the operator to position one or more virtual markers on the electrocardiogram;

determining a metric based on positions of the one or more virtual markers by processing the third set of sensor data; and

providing the metric to the HMD to update the display of the electrocardiogram in the field of view of the operator.

25. The computer readable storage medium of claim 20, further comprising instructions that, when executed by the processor, cause the processor to perform steps including:

receiving from the input device, a third set of sensor data indicative of subsequent motion of the body part of the operator;

determining an increase in a level of zoom of the electroanatomic visualization responsive to the subsequent motion; and

providing a third 3D image to the HMD to display the electroanatomic visualization including one or more additional components in the field of view of the operator.

26. The computer readable storage medium of claim 20, wherein the electroanatomic visualization further includes a representation of a portion of the catheter positioned relative to the cardiovascular organ in 3D.

27. The computer readable storage medium of claim 26, further comprising instructions that, when executed by the processor, cause the processor to perform steps including:

receiving a third set of sensor data from the catheter indicative of detected motion of the catheter; and

providing a third 3D image to the HMD to update the display of the electroanatomic visualization by moving the portion of the catheter based on the detected motion in the field of view of the operator.

28. The computer readable storage medium of claim 20, wherein generating the first 3D image comprises:

aggregating the first set of sensor data with a third set of sensor data generated by at least one of an impedance sensor, a magnetic sensor, and an ultrasound sensor; and

mapping a first plurality of points of the first set of sensor data to a second plurality of points of the third set of sensor data in 3D, wherein the first set of sensor data and the third set of sensor data are generated in different physical units.

29. The computer readable storage medium of claim 20, wherein generating the first 3D image of the electroanatomic visualization comprises:

generating a series of connected 2D polygons to represent 3D geometries of the cardiovascular organ.

30. The computer readable storage medium of claim 20, further comprising instructions that, when executed by the processor, cause the processor to perform steps including:

providing, simultaneously with providing the second 3D image to the HMD, a third 3D image to another HMD worn by another operator to display the electroanatomic visualization rotated by the angle of rotation in another field of view of the another operator, wherein the another field of view is positioned at an offset angle from the field of view.