AUGMENTED REALITY IMAGE DISPLAY SYSTEMS AND METHODS

• US 20190310819A1
• Filed: 04/10/2019
• Published: 10/10/2019
• Est. Priority Date: 04/10/2018
• Status: Abandoned Application

• Current Assignee: Carto Technologies LLC
• Original Assignee: Carto Technologies LLC
• Inventors: Xu, Jimmy, Sloop, Austin, Hu, Peng, Gildersleeve, Tara
• Application Number:
  US16/379,968
• Publication Number:
  US 20190310819A1
• Time in Patent Office: Days
• CPC Class Codes: G06F 3/013 Eye tracking input arrangements G06F3/015 takes precedence
• CPC Class Codes: G06F 3/017 Gesture based interaction, e.g. based on a set of recognized hand gestures interaction based on gestures traced on a digitiser G06F3/04883
• CPC Class Codes: G06F 3/1454 involving copying of the display data of a local workstation or window to a remote workstation or window so that an actual copy of the data is displayed simultaneously on two or more displays, e.g. teledisplay
• CPC Class Codes: G06F 3/167 Audio in a user interface, e.g. using voice commands for navigating, audio feedback
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10116 X-ray image
• CPC Class Codes: G06T 2207/30021 Catheter; Guide wire
• CPC Class Codes: G06T 2207/30196 Human being; Person
• CPC Class Codes: G06T 2207/30204 Marker
• CPC Class Codes: G06T 7/70 Determining position or orientation of objects or cameras camera calibration G06T7/80
• CPC Class Codes: G06T 7/73 using feature-based methods
• CPC Class Codes: G09G 2340/0492 Change of orientation of the displayed image, e.g. upside-down, mirrored
• CPC Class Codes: G09G 2340/12 Overlay of images, i.e. displayed pixel being the result of switching between the corresponding input pixels
• CPC Class Codes: G09G 2370/16 Use of wireless transmission of display information
• CPC Class Codes: G09G 2370/20 Details of the management of multiple sources of image data
• CPC Class Codes: G09G 2380/08 Biomedical applications
• CPC Class Codes: G16H 30/40 for processing medical images, e.g. editing
• CPC Class Codes: G16H 40/63 for local operation
• CPC Class Codes: G16H 50/70 for mining of medical data, e.g. analysing previous cases of other patients
• CPC Class Codes: G16H 80/00 ICT specially adapted for facilitating communication between medical practitioners or patients, e.g. for collaborative diagnosis, therapy or health monitoring
• CPC Class Codes: H04N 5/272 : Means for inserting a foreground image in a background image, i.e. inlay, outlay

Abstract

This invention is an augmented reality device that displays virtual screens at user specified positions taking inputs from the electronic medical record, PACs system, and imaging device. The virtual screens can be displayed in any orientation. Machine learning tools will improve ease of workflow. Collaboration tools will be available. An API will ease interoperability between radiology imaging devices and augmented reality systems. Devices such as catheters/needles impregnated with radiopaque markers and other radiopaque target markers are also discussed.

Claims

1. A system, comprising:

a visual input sensor, wherein the visual input sensor is operable to capture a live stream video of a patient;

an image interface, wherein the image interface is operable to receive image data of the patient from a plurality of different imaging devices;

at least one visual display device;

at least one processor; and

at least one non-transitory computer-readable storage medium, wherein the at least one non-transitory computer-readable storage medium stores one or more processor-executable instructions that, when executed by the at least one processor;

receive the live stream video of the patient,

receive the image data of the patient,

determine an orientation of the patient from the live video stream,

generate a video output of the live video stream for outputting via the at least one visual display device, and

overlay the image data of the patient in the video output, wherein the overlaid image data is aligned with the orientation of the patient.

2. The system of claim 1, further comprising a touchless user input sensor, wherein the touchless user input sensor is operable to receive a touchless input from an operator to manipulate at least one of the video output and the overlaid image data.

3. The system of claim 2, wherein the touchless input includes at least one of eye tracking, gesture tracking, and speech.

4. The system of claim 1, further comprising a network interface to communicate the video output and the overlaid image data to one or more other visual display devices, thereby enabling collaboration among a plurality of operators.

5. The system of claim 1, wherein the image data includes data representing one or more radiopaque markers arranged on a surgical tool, and wherein the at least one non-transitory computer-readable storage medium further stores one or more processor-executable instructions that, when executed by the at least one processor:

determine an orientation of the radiopaque markers;

determine a position of the surgical tool relative to the patient from the orientation of the radiopaque markers; and

overlay a visual representation of the surgical tool at the determined position of the patient in the video output.

6. A method, comprising:

capturing a live stream video of a patient via at least one visual input sensor;

receiving image data of the patient from a plurality of different imaging devices;

determining an orientation of the patient from the live video stream;

generating a video output of the live video stream;

overlaying the image data of the patient in the video output, wherein the overlaid image data is aligned with the orientation of the patient; and

outputting, via at least one visual display device, the video output and overlaid image data.

7. The method of claim 6, further comprising receiving, via a touchless user input sensor, a touchless user input from an operator to manipulate at least one of the video output and the overlaid image data.

8. The method of claim 7, wherein the touchless input includes at least one of eye tracking, gesture tracking, and speech.

9. The method of claim 6, further comprising communicating, via network interface, the video output and the overlaid image data to one or more other visual display devices, thereby enabling collaboration among a plurality of operators.

10. The method of claim 6, further comprising:

receiving image data representing one or more radiopaque markers arranged on a surgical tool;

determining an orientation of the radiopaque markers;

determining a position of the surgical tool relative to the patient from the orientation of the radiopaque markers; and

overlaying a visual representation of the surgical tool at the determined position of the patient in the video output.

11. A non-transitory computer readable storage medium comprising a set of instructions executable by a computer, the non-transitory computer readable storage medium comprising:

instructions for capturing a live stream video of a patient via at least one visual input sensor;

instructions for receiving image data of the patient from a plurality of different imaging devices;

instructions for determining an orientation of the patient from the live video stream;

instructions for generating a video output of the live video stream;

instructions for overlaying the image data of the patient in the video output, wherein the overlaid image data is aligned with the orientation of the patient; and

instructions for outputting, via at least one visual display device, the video output and overlaid image data.

12. The non-transitory computer readable storage medium of claim 11, further comprising instructions for receiving, via a touchless user input sensor, a touchless user input from an operator to manipulate at least one of the video output and the overlaid image data.

13. The non-transitory computer readable storage medium of claim 12, wherein the touchless input includes at least one of eye tracking, gesture tracking, and speech.

14. The non-transitory computer readable storage medium of claim 11, further comprising instructions for communicating, via network interface, the video output and the overlaid image data to one or more other visual display devices, thereby enabling collaboration among a plurality of operators.

15. The non-transitory computer readable storage medium of claim 11, further comprising:

instructions for receiving image data representing one or more radiopaque markers arranged on a surgical tool;

instructions for determining an orientation of the radiopaque markers;

instructions for determining a position of the surgical tool relative to the patient from the orientation of the radiopaque markers; and

instructions for overlaying a visual representation of the surgical tool at the determined position of the patient in the video output.