Imaging system and methods displaying a fused multidimensional reconstructed image

• US 9,552,660 B2
• Filed: 08/30/2013
• Issued: 01/24/2017
• Est. Priority Date: 08/30/2012
• Status: Active Grant

• Current Assignee: Alcon Incorporated
• Original Assignee: TrueVision Systems, Inc.
• Inventors: Tripathi, Ashok Burton, Polchin, George Charles, Ng, Yen Ting
• Primary Examiner(s): Akhavannik, Hadi
• Application Number:
  US14/424,985
• Publication Number:
  US 20150221105A1
• Time in Patent Office: 1,243 Days
• Field of Search: None
• US Class Current: 1/1
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/3983 Reference marker arrangements for use with image guided surgery
• CPC Class Codes: A61B 2576/00 Medical imaging apparatus involving image processing or analysis A61B1/00009, A61B6/52 and A61B8/52 take precedence; image processing per se G06T; details of algorithms for analysing medical images G06T7/0012
• CPC Class Codes: A61B 2576/026 for the brain
• CPC Class Codes: A61B 5/0077 Devices for viewing the surface of the body, e.g. camera, magnifying lens
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 5/7425 Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image geometric transforms for image registration G06T3/14; enhancement using image fusion G06T5/50; algorithms for registration of images G06T7/30
• CPC Class Codes: A61B 5/743 Displaying an image simultaneously with additional graphical information, e.g. symbols, charts, function plots
• CPC Class Codes: A61B 5/745 using a holographic display
• CPC Class Codes: A61B 6/03 Computed tomography [CT] echo-tomography A61B8/14
• CPC Class Codes: A61B 6/032 Transmission computed tomography [CT]
• CPC Class Codes: A61B 6/037 Emission tomography
• CPC Class Codes: A61B 90/20 Surgical microscopes characterised by non-optical aspects
• CPC Class Codes: A61B 90/37 Surgical systems with images on a monitor during operation
• CPC Class Codes: G06T 11/008 Specific post-processing after tomographic reconstruction, e.g. voxelisation, metal artifact correction
• CPC Class Codes: G06T 19/00 Manipulating 3D models or images for computer graphics
• CPC Class Codes: G06T 19/20 Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2200/24 involving graphical user interfaces [GUIs]
• CPC Class Codes: G06T 2207/10021 Stereoscopic video; Stereoscopic image sequence
• CPC Class Codes: G06T 2207/10028 Range image; Depth image; 3D point clouds
• CPC Class Codes: G06T 2207/10088 : Magnetic resonance imaging [MRI]
• CPC Class Codes: G06T 2207/20104 : Interactive definition of region of interest [ROI]
• CPC Class Codes: G06T 2207/20221 : Image fusion; Image merging
• CPC Class Codes: G06T 2207/30016 : Brain
• CPC Class Codes: G06T 2207/30196 : Human being; Person
• CPC Class Codes: G06T 2207/30204 : Marker
• CPC Class Codes: G06T 2210/41 : Medical
• CPC Class Codes: G06T 2210/62 : Semi-transparency
• CPC Class Codes: G06T 2211/40 : Computed tomography
• CPC Class Codes: G06T 2219/028 : Multiple view windows (top-side-front-sagittal-orthogonal)
• CPC Class Codes: G06T 2219/2004 : Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 5/70 : Denoising; Smoothing
• CPC Class Codes: G06T 7/0014 : using an image reference approach
• CPC Class Codes: G06T 7/337 : involving reference images or patches
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 30/20 : for handling medical images, e.g. DICOM, HL7 or PACS
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: H04N 13/204 : using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 13/30 : Image reproducers optical systems for producing stereoscopic or other three-dimensional effects G02B30/00

Abstract

In some embodiments, an imaging system displays a multidimensional visualization of a surgical site. In some embodiments, the imaging system receives a selection corresponding to a portion of the displayed multidimensional visualization of the surgical site. At the selected portion of the multidimensional visualization, the imaging system displays a portion of a multidimensional reconstructed image which corresponds to the selected multidimensional visualization such that the displayed portion of the multidimensional reconstructed image is fused with the displayed multidimensional visualization.

Claims

1. An imaging system comprising:

a processor;

an input device operatively connected to the processor;

a display device operatively connected to the processor; and

a memory device storing instructions which when executed by the processor, cause the processor to operate with the input device, the display device, and the memory device to;

(a) display a multidimensional visualization of a surgical site;

(b) receive a selection corresponding to a portion of the displayed multidimensional visualization of the surgical site; and

(c) at the selected portion of the multidimensional visualization, display a portion of a multidimensional reconstructed image which corresponds to the selected multidimensional visualization, the displayed portion of the multidimensional reconstructed image being fused with the displayed multidimensional visualization by;

increasing a transparency of the selected portion of the multidimensional visualization; and

applying a smoothing function to a boundary between the multidimensional reconstructed image and the multidimensional visualization.

2. The imaging system of claim 1, wherein the multidimensional visualization is three dimensional and displayed in real time.

3. The imaging system of claim 1, wherein the multidimensional reconstructed image is:

(a) three dimensional; and

(b) generated based on image data captured by a medical imaging machine.

4. The imaging system of claim 3, wherein the image data includes two dimensional pre-operative image slice data.

5. The imaging system of claim 1, wherein the selected portion of the multidimensional visualization is represented as a window.

6. The imaging system of claim 5, wherein the window has a shape selected from the group consisting of circular shape, square shape, rectangular shape and anatomic shape.

7. The imaging system of claim 5, wherein the instructions, when executed by the processor, cause the processor to move the position of the window in response to the window being clicked and dragged.

8. The imaging system of claim 1, wherein the multidimensional reconstructed image includes a first feature and a second feature.

9. The imaging system of claim 8, wherein:

(a) the first feature corresponds to a tumor; and

(b) the second feature corresponds to bone structure.

10. The imaging system of claim 8, wherein the instructions, when executed by the processor, highlight the first feature.

11. The imaging system of claim 8, wherein the instructions, when executed by the processor, filter out the first feature.

12. An imaging system comprising:

a processor;

an input device operatively connected to the processor;

a display device operatively connected to the processor; and

a memory device storing instructions which when executed by the processor, cause the processor to operate with the input device, the display device, and the memory device to;

(a) display a multidimensional visualization of a surgical site;

(b) receive a selection corresponding to a portion of the displayed multidimensional visualization of the surgical site;

(c) at the selected portion of the multidimensional visualization, display a portion of a multidimensional reconstructed image which corresponds to the selected multidimensional visualization, the displayed portion of the multidimensional reconstructed image being fused with the displayed multidimensional visualization;

(d) receive a request for a second portion of the multidimensional reconstructed image having a specific depth; and

(e) display the requested second portion of the multidimensional reconstructed image having the specific depth.

15. The imaging system of claim 12, wherein the multidimensional visualization is three dimensional and displayed in real time, and

wherein the multidimensional reconstructed image is:

(a) three dimensional; and

(b) generated based on image data captured by a medical imaging machine.

16. The imaging system of claim 15, wherein the image data includes two dimensional pre-operative image slice data.

17. The imaging system of claim 12, wherein the selected portion of the multidimensional visualization is represented as a window and

wherein the instructions, when executed by the processor, cause the processor to move the position of the window in response to the window being clicked and dragged.

18. The imaging system of claim 12, wherein the multidimensional reconstructed image includes a first feature and a second feature, and

wherein the instructions, when executed by the processor, (i) highlight the first feature, or (ii) filter out the first feature.

13. An imaging system comprising:

a processor;

an input device operatively connected to the processor;

a display device operatively connected to the processor; and

a memory device storing instructions which when executed by the processor, cause the processor to operate with the input device, the display device, and the memory device to;

(a) display a multidimensional visualization of a surgical site;

(b) receive a selection corresponding to a portion of the displayed multidimensional visualization of the surgical site;

(c) at the selected portion of the multidimensional visualization, display a portion of a multidimensional reconstructed image which corresponds to the selected multidimensional visualization, the displayed portion of the multidimensional reconstructed image being fused with the displayed multidimensional visualization;

(d) for the displayed multidimensional reconstructed image, receive a request for a depth setting to correspond to a depth that is a specific distance below coordinates of the displayed multidimensional visualization; and

(e) display the portion of the multidimensional reconstructed image having a depth that is at the specific distance below the coordinates of the displayed multidimensional visualization.

19. The imaging system of claim 13, wherein the multidimensional visualization is three dimensional and displayed in real time, and

wherein the multidimensional reconstructed image is:

(a) three dimensional; and

(b) generated based on image data captured by a medical imaging machine.

20. The imaging system of claim 19, wherein the image data includes two dimensional pre-operative image slice data.

21. The imaging system of claim 13, wherein the selected portion of the multidimensional visualization is represented as a window and

wherein the instructions, when executed by the processor, cause the processor to move the position of the window in response to the window being clicked and dragged.

22. The imaging system of claim 13, wherein the multidimensional reconstructed image includes a first feature and a second feature, and

wherein the instructions, when executed by the processor, (i) highlight the first feature, or (ii) filter out the first feature.

14. An imaging system comprising:

a processor;

an input device operatively connected to the processor;

a display device operatively connected to the processor; and

a memory device storing instructions which when executed by the processor, cause the processor to operate with the input device, the display device, and the memory device to;

(a) display a multidimensional visualization of a surgical site;

(b) receive a selection corresponding to a portion of the displayed multidimensional visualization of the surgical site; and

(c) at the selected portion of the multidimensional visualization, display a portion of a multidimensional reconstructed image which corresponds to the selected multidimensional visualization, the displayed portion of the multidimensional reconstructed image being fused with the displayed multidimensional visualization by identifying first features of the multidimensional visualization and aligning the first features to correspond to second features of the multidimensional reconstructed image.

23. The imaging system of claim 14, wherein the multidimensional visualization is three dimensional and displayed in real time, and

wherein the multidimensional reconstructed image is:

(a) three dimensional; and

(b) generated based on image data captured by a medical imaging machine.

24. The imaging system of claim 23, wherein the image data includes two dimensional pre-operative image slice data.

25. The imaging system of claim 14, wherein the selected portion of the multidimensional visualization is represented as a window and

wherein the instructions, when executed by the processor, cause the processor to move the position of the window in response to the window being clicked and dragged.

26. The imaging system of claim 14, wherein the multidimensional reconstructed image includes a first feature and a second feature, and

wherein the instructions, when executed by the processor, (i) highlight the first feature, or (ii) filter out the first feature.