ALIGNMENT OF Q3D MODELS WITH 3D IMAGES

• US 20170181809A1
• Filed: 03/28/2015
• Published: 06/29/2017
• Est. Priority Date: 03/28/2014
• Status: Abandoned Application

• Current Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Original Assignee: Intuitive Surgical Operations, Inc. (Intuitive Surgical Holdings, LLC)
• Inventors: Jones, Daniel H., Panescu, Dorin
• Application Number:
  US15/300,259
• Publication Number:
  US 20170181809A1
• Time in Patent Office: Days
• CPC Class Codes: A61B 1/00048 Constructional features of the display
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/04 combined with photographic or television appliances
• CPC Class Codes: A61B 1/043 for fluorescence imaging
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2065 Tracking using image or pattern recognition
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2090/363 Use of fiducial points
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/367 creating a 3D dataset from 2D images using position information
• CPC Class Codes: A61B 2090/373 using light, e.g. by using optical scanners
• CPC Class Codes: A61B 2090/374 NMR or MRI
• CPC Class Codes: A61B 2090/3762 using computed tomography systems [CT]
• CPC Class Codes: A61B 2090/378 using ultrasound
• CPC Class Codes: A61B 2090/502 Headgear, e.g. helmet, spectacles
• CPC Class Codes: A61B 34/32 operating autonomously
• CPC Class Codes: A61B 34/35 for telesurgery
• CPC Class Codes: A61B 5/055 involving electronic [EMR] or nuclear [NMR] magnetic resonance, e.g. magnetic resonance imaging
• CPC Class Codes: A61B 5/7425 Displaying combinations of multiple images regardless of image source, e.g. displaying a reference anatomical image with a live image geometric transforms for image registration G06T3/14; enhancement using image fusion G06T5/50; algorithms for registration of images G06T7/30
• CPC Class Codes: A61B 6/03 Computed tomography [CT] echo-tomography A61B8/14
• CPC Class Codes: A61B 6/032 : Transmission computed tomography [CT]
• CPC Class Codes: A61B 6/037 : Emission tomography
• CPC Class Codes: A61B 6/466 : adapted to display 3D data
• CPC Class Codes: A61B 8/466 : adapted to display 3D data 3D imaging and stereoscopic displays in devices for short-range imaging with acoustic waves G01S15/8993 and G01S7/52068, respectively; 3D image rendering G06T15/00; manipulating 3D models for computer graphics G06T19/00
• CPC Class Codes: A61B 90/37 : Surgical systems with images on a monitor during operation
• CPC Class Codes: G06T 19/20 : Editing of 3D images, e.g. changing shapes or colours, aligning objects or positioning parts
• CPC Class Codes: G06T 2207/10064 : Fluorescence image
• CPC Class Codes: G06T 2207/10068 : Endoscopic image
• CPC Class Codes: G06T 2207/10081 : Computed x-ray tomography [CT]
• CPC Class Codes: G06T 2207/10088 : Magnetic resonance imaging [MRI]
• CPC Class Codes: G06T 2207/10104 : Positron emission tomography [PET]
• CPC Class Codes: G06T 2207/10132 : Ultrasound image
• CPC Class Codes: G06T 2219/2004 : Aligning objects, relative positioning of parts
• CPC Class Codes: G06T 7/33 : using feature-based methods
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 30/40 : for processing medical images, e.g. editing
• CPC Class Codes: H04N 13/232 : using fly-eye lenses, e.g. arrangements of circular lenses

Abstract

A method is provided to align a quantitative three-dimensional (Q3D) model of a three dimensional (3D) structure with a 3D visual representation of a sub-surface target object internal to the anatomical structure, the method comprising: identifying fiducial points within the external surface of the 3D structure represented in the 3D visual representation; identifying the same fiducial points within the Q3D model; aligning the identified fiducial points in the 3D visual representation with the identified fiducial points in the Q3D model; and producing a visual image representation of the 3D structure that includes a view of an external surface and a view of the internal sub-surface target structure.

Claims

1. A system for producing an image of a surgical scene comprising:

a quantitative three-dimensional (Q3D) endoscope disposed to image a scene within a surgical field of view;

at least one processor configured to;

determine a Q3D model of a tissue structure imaged by the Q3D endoscope;

input a 3D visual representation of the tissue structure;

determine a geometric transformation to align an outer surface of the tissue structure of the 3D visual representation of the tissue structure with the Q3D model of said tissue structure; and

produce, based at least in part upon said geometric transformation, a visual output representing the combined Q3D model of the tissue structure and the 3D visual representation of the tissue structure.

2. The system of claim 1,

wherein the 3D visual representation can be one of the following:

MRI, CT, PET, ultrasound or fluorescent images.

3. The system of claim 1,

identifying multiple fiducial points on an outer surface of the tissue structure represented in the 3D visual representation;

identifying substantially the same multiple fiducial points of the tissue structure represented within the Q3D model;

wherein the at least one processor is further configured to;

apply a geometric transformation to the 3D visual representation of the tissue structure to align the identified fiducial points in the 3D visual representation with the identified fiducial points in the Q3D model representation of the tissue structure.

4. The system of claim 1,

wherein a sub-surface target tissue structure visible as ghost image within the visual output.

5. The system of claim 1,

wherein the at least one processor is further configured to:

identify a view of a sub-surface target structure within the aligned 3D visual representation;

wherein producing a visual output representing the combined Q3D model of the tissue structure and the 3D visual representation of the tissue structure includes producing an output that includes the identified view of the sub-surface target structure.

6. The system of claim 4,

wherein a view includes a slice of the 3D visual representation.

7. The system of claim 1,

wherein the Q3D model display information includes a display of (x, y, z) coordinates in reference to details visible in the image.

8. The system of claim 1,

wherein the Q3D model display information includes a display of information indicating distance between points on an outer surface of the tissue structure from the Q3D endoscope.

9. A method producing an image of a surgical scene comprising:

creating a Q3D model of structure surgical scene that includes a tissue structure;

inputting a 3D visual representation of the tissue structure;

using a geometric transformation to align an outer surface structure of the 3D visual representation of the tissue structure with the Q3D model of the tissue structure; and

producing, based at least in part upon said geometric transformation, a visual output representation of the tissue structure.

10. The method of claim 9,

wherein producing the combined Q3D model and 3D visual representation is based, at least in part, upon:

identifying multiple fiducial points within the surface of the structure represented in the 3D visual representation;

identifying substantially the same multiple fiducial points within the Q3D model;

applying a geometric transformation to the 3D visual representation of the tissue structure to align the identified fiducial points in the 3D visual representation of the tissue structure with the identified fiducial points in the Q3D model representation of the tissue structure.

11. The method of claim 9,

wherein the 3D visual representation can be one of the following:

MRI, CT, PET, ultrasound or fluorescent images.

12. The method of claim 9,

wherein a sub-surface target tissue structure visible as ghost image within the visual output.

13. The system of claim 9 further including:

identifying a view of a sub-surface target structure within the aligned 3D visual representation;

wherein producing a visual output representing the combined Q3D model of the tissue structure and the 3D visual representation of the tissue structure includes producing an output that includes the identified view of the sub-surface target structure.

14. The method of claim 13,

wherein the view includes a slice of the 3D visual representation.

15. The method of claim 13,

wherein the Q3D model display information includes a display of (x, y, z) coordinates in reference to details visible in the image.

16. The method of claim 9,

wherein the Q3D model display information includes a display of information indicating distance between points on an outer surface of the tissue structure from the Q3D endoscope.