Loupe display

• US 9,675,319 B1
• Filed: 06/30/2016
• Issued: 06/13/2017
• Est. Priority Date: 02/17/2016
• Status: Active Grant

• Current Assignee: Inneroptic Technology, Inc.
• Original Assignee: Inneroptic Technology, Inc.
• Inventors: State, Andrei, Heaney, Brian, Kohli, Luv, Keller, Kurtis, Green, Caroline, Razzaque, Sharif
• Primary Examiner(s): Elahi, Towfiq
• Application Number:
  US15/199,630
• Time in Patent Office: 348 Days
• CPC Class Codes: A61B 5/0066 Optical coherence imaging
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 5/066 Superposing sensor position on an image of the patient, e.g. obtained by ultrasound or x-ray imaging
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 6/037 Emission tomography
• CPC Class Codes: A61B 6/466 adapted to display 3D data
• CPC Class Codes: A61B 6/487 involving fluoroscopy
• CPC Class Codes: A61B 8/0841 for locating instruments
• CPC Class Codes: A61B 8/4245 involving determining the position of the probe, e.g. with respect to an external reference frame or to the patient
• CPC Class Codes: A61B 8/466 adapted to display 3D data 3D imaging and stereoscopic displays in devices for short-range imaging with acoustic waves G01S15/8993 and G01S7/52068, respectively; 3D image rendering G06T15/00; manipulating 3D models for computer graphics G06T19/00
• CPC Class Codes: A61B 8/483 involving the acquisition of a 3D volume of data

Abstract

A system and method for providing image guidance for placement of one or more medical devices at a target location. The system can determine one or more intersections between a medical device and an image region based at least in part on first emplacement data and second emplacement data. Using the determined intersections, the system can cause one or more displays to display perspective views of image guidance cues, including an intersection indicator in a virtual 3D space.

Claims

1. A method of presenting virtual content to a user of a head mounted display, the method comprising:

receiving emplacement data associated with a first emplacement sensor with respect to a first coordinate system;

receiving emplacement data associated with a second emplacement sensor with respect to the first coordinate system;

receiving emplacement data associated with a third emplacement sensor with respect to a second coordinate system, the third emplacement sensor associated with a head mounted display;

determining a first emplacement of a first virtual medical device with respect to a third coordinate system based at least in part on the received emplacement data associated with the first emplacement sensor, a predetermined offset, and the received emplacement data associated with the third emplacement sensor;

determining a first emplacement of a second virtual medical device with respect to the third coordinate system based at least in part on the received emplacement data associated with the second emplacement sensor, the predetermined offset, and the received emplacement data associated with the third emplacement sensor;

receiving a real-time medical imaging stream associated with the second virtual medical device;

determining a first emplacement of the real-time medical imaging stream with respect to the third coordinate system based at least in part on the received emplacement data associated with the second emplacement sensor, the predetermined offset, and the received emplacement data associated with the third emplacement sensor;

determining a second emplacement of the first virtual medical device with respect to a fourth coordinate system based at least in part on the received emplacement data associated with the first emplacement sensor, the predetermined offset, and the received emplacement data associated with the third emplacement sensor;

determining a second emplacement of the second virtual medical device with respect to the fourth coordinate system based at least in part on the received emplacement data associated with the second emplacement sensor, the predetermined offset, and the received emplacement data associated with the third emplacement sensor;

determining a second emplacement of the real-time medical imaging stream with respect to the fourth coordinate system based at least in part on the received emplacement data associated with the second emplacement sensor, the predetermined offset, and the received emplacement data associated with the third emplacement sensor;

causing a first opaque stereoscopic display of the head mounted display to display a first 3D rendered view of a virtual 3D volume including;

a 3D rendering of the first virtual medical device based at least in part on the determined first emplacement of the first virtual medical device,

a 3D rendering of the second virtual medical device based at least in part on the determined first emplacement of the second virtual medical device, and

a 3D rendering of the real-time medical imaging stream based at least in part on the determined first emplacement of the real-time medical imaging stream; and

causing a second opaque stereoscopic display of the head mounted display to display a second 3D rendered view of the virtual 3D volume including;

a 3D rendering of the first virtual medical device based at least in part on the determined second emplacement of the first virtual medical device,

a 3D rendering of the second virtual medical device based at least in part on the determined second emplacement of the second virtual medical device, and

a 3D rendering of the real-time medical imaging stream based at least in part on the determined second emplacement of the real-time medical imaging stream,

wherein the first 3D rendered view and the second 3D rendered view change in relation to a change in the emplacement data associated with the third emplacement sensor.

2. The method of claim 1, wherein the first emplacement sensor is associated with a first real medical device and the second emplacement sensor is associated with a second real medical device.

3. The method of claim 2, wherein the predetermined offset is configured to allow a wearer of the head mounted display to simultaneously view at least a portion of the first virtual medical device, and at least a portion of the real-time medical imaging stream on the first and second opaque stereoscopic displays, and at least a portion of a first medical device corresponding to the first virtual medical device and the second real medical device corresponding to the second virtual medical device at a location below the first and second opaque stereoscopic displays.

4. The method of claim 1, wherein the real-time medical imaging stream comprises at least one ultrasound image.

5. The method of claim 1, wherein the first opaque stereoscopic display is at least partially located within, and comprises only a subset of a field of view of a left eye of a wearer of the head mounted display, and wherein the second opaque stereoscopic display is at least partially located within, and comprises only a subset of, the field of view of a right eye of a wearer of the head mounted display.

6. A method of presenting virtual content to a user, the method comprising:

receiving first emplacement data associated with a first emplacement sensor with respect to a first coordinate system;

receiving second emplacement data associated with a second emplacement sensor with respect to the first coordinate system;

receiving third emplacement data associated with a third emplacement sensor with respect to a second coordinate system, the third emplacement sensor associated with a user;

determining an emplacement of a first virtual medical device with respect to a third coordinate system based at least in part on the first emplacement data;

applying a predetermined offset to the emplacement of the first virtual medical device in the third coordinate system to provide an offset emplacement of the first virtual medical device with respect to the third coordinate system;

determining an emplacement of a second virtual medical device with respect to the third coordinate system based at least in part on the second emplacement data;

applying the predetermined offset to the emplacement of the second virtual medical device in the third coordinate system to provide an offset emplacement of the second virtual medical device with respect to the third coordinate system;

receiving a medical imaging stream associated with the second virtual medical device;

determining an emplacement of the medical imaging stream with respect to the third coordinate system based at least in part on the second emplacement data;

applying the predetermined offset to the emplacement of the medical imaging stream in the third coordinate system to provide an offset emplacement of the medical imaging stream with respect to the third coordinate system; and

causing a display to display a 3D rendered view of a scene including;

a 3D rendering of the first virtual medical device based at least in part on the offset emplacement of the first virtual medical device and the third emplacement data,

a 3D rendering of the second virtual medical device based at least in part on the offset emplacement of the second virtual medical device and the third emplacement data, and

a 3D rendering of the medical imaging stream based at least in part on the offset emplacement of the medical imaging stream and the third emplacement data,

wherein the 3D rendered view changes in relation to a change in the determined emplacement of the third emplacement sensor.

7. The method of claim 6, wherein the first emplacement sensor is associated with a first real medical device and the second emplacement sensor is associated with a second real medical device.

8. The method of claim 7, wherein the predetermined offset is configured to allow a viewer of the display to simultaneously view at least a portion of the first virtual medical device and at least a portion of the medical imaging stream on the display, and at least a portion of the first real medical device corresponding to the first virtual medical device above and/or below the display.

9. The method of claim 6, wherein the medical imaging stream comprises at least one ultrasound image.

10. A guidance system, comprising one or more processors, the guidance system configured to:

receive first emplacement data associated with a first emplacement sensor with respect to a first coordinate system;

receive second emplacement data associated with a second emplacement sensor with respect to the first coordinate system;

determine an emplacement of a medical image with respect to a second coordinate system based at least in part on the received first emplacement data;

determine an emplacement of a virtual medical device based at least in part on the second emplacement data;

apply a predetermined offset to the emplacement of the medical image in the second coordinate system to provide offset emplacement of the medical image with respect to the second coordinate system;

apply the predetermined offset to the emplacement of the virtual medical device in the second coordinate system to provide offset emplacement of the virtual medical device with respect to the second coordinate system; and

cause a display to display;

a perspective rendering of the virtual medical device based at least in part on the offset emplacement of the virtual medical device, and

a perspective rendering of the medical image based at least in part on the offset emplacement of the medical image,

wherein the perspective rendering of the medical image varies in relation to a change in the received first emplacement data.

11. The guidance system of claim 10, wherein the virtual medical device is a first medical device, and wherein the guidance system is further configured to:

determine an emplacement of a second virtual medical device with respect to the second coordinate system based at least in part on the received first emplacement data;

apply the predetermined offset to the emplacement of the second virtual medical device in the second coordinate system to provide offset emplacement of the second virtual medical device with respect to the second coordinate system; and

cause the display to display a perspective rendering of the second virtual medical device based at least in part on the offset emplacement of the second virtual medical device.

12. The guidance system of claim 10, wherein the second emplacement sensor is coupled to a physical medical device.

13. The guidance system of claim 12, wherein the guidance system is configured to cause the display to display the perspective rendering of the medical image and the perspective rendering of the virtual medical device using the predetermined offset such that a viewer of the display simultaneously views at least a portion of the perspective rendering of the medical image and at least a portion of the perspective rendering of the virtual medical device on the display, and at least a portion of the physical medical device above and/or below the display.

14. The guidance system of claim 10, wherein the guidance system is further configured to receive third emplacement data associated with a third emplacement sensor associated with a user, wherein at least one of the perspective rendering of the virtual medical device or the perspective rendering of the medical image varies in relation to a change in the third emplacement data.

15. The guidance system of claim 10, wherein the display comprises an opaque, stereoscopic display of a head mounted display.

16. The guidance system of claim 10, wherein the guidance system is configured to cause the display to concurrently display the perspective rendering of the medical image and the perspective rendering of the virtual medical device.

17. The guidance system of claim 14, wherein the perspective rendering of the virtual medical device is based at least in part on the third emplacement data.

18. The guidance system of claim 14, wherein the perspective rendering of the medical image is based at least in part on the third emplacement data.

19. The guidance system of claim 14, wherein the third emplacement sensor is associated with a head mounted display.

20. The guidance system of claim 19, wherein the head mounted display comprises:

a frame configured to mount to a head of the user;

a first opaque display coupled to the frame and located within a portion of a field of view of a left eye of the user when the frame is mounted to the head of the user, the first opaque display comprising only a subset of the field of view of the left eye of the user; and

a second opaque display coupled to the frame and located within a field of view of a right eye of the user when the frame is mounted to the head of the user, the second opaque display comprising only a subset of the field of view of the right eye of the user.