Robotic control of an endoscope from blood vessel tree images

• US 9,615,886 B2
• Filed: 09/13/2011
• Issued: 04/11/2017
• Est. Priority Date: 09/15/2010
• Status: Active Grant

• Current Assignee: Koninklijke Philips N.V.
• Original Assignee: Koninklijke Philips N.V.
• Inventors: Popovic, Aleksandra
• Primary Examiner(s): Nguyen, Anhtuan T
• Assistant Examiner(s): CHOU, WILLIAM B
• Application Number:
  US13/822,001
• Publication Number:
  US 20130165948A1
• Time in Patent Office: 2,037 Days
• Field of Search: 600102, 600109, 600114, 600117, 600118, 600160
• CPC Class Codes: A61B 1/00006 of control signals
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/0016 using motor drive units
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 1/3137 for examination of the interior of blood vessels
• CPC Class Codes: A61B 2017/00252 for by-pass connections, i.e. connections from heart chamber to blood vessel or from blood vessel to blood vessel
• CPC Class Codes: A61B 2034/301 for introducing or steering flexible instruments inserted into the body, e.g. catheters or endoscopes
• CPC Class Codes: A61B 2090/3614 using optical fibre
• CPC Class Codes: A61B 2090/364 Correlation of different images or relation of image positions in respect to the body
• CPC Class Codes: A61B 34/10 Computer-aided planning, simulation or modelling of surgical operations
• CPC Class Codes: A61B 34/30 Surgical robots
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/10072 Tomographic images
• CPC Class Codes: G06T 2207/20072 Graph-based image processing
• CPC Class Codes: G06T 2207/30101 Blood vessel; Artery; Vein; Vascular
• CPC Class Codes: G06T 7/344 involving models
• CPC Class Codes: G06T 7/75 involving models

Abstract

A robot guiding system employs a robot unit (10) and a control unit (20). The robot unit (10) includes an endoscope (12) for generating an intra-operative endoscopic image (14) of a blood vessel tree within an anatomical region, and a robot (11) for moving the endoscope (12) within the anatomical region. The control unit (20) includes an endoscope controller (22) for generating an endoscopic path within the anatomical region, wherein the endoscopic path is derived from a matching of a graphical representation of the intra-operative endoscopic image (14) of the blood vessel tree to a graphical representation of a pre-operative three-dimensional image (44) of the blood vessel tree. The control unit (20) further includes a robot controller (21) for commanding the robot (11) to move the endoscope (12) within the anatomical region in accordance with the endoscopic path.

Claims

1. A robot guiding system, comprising:

an endoscope including an endoscope camera configured to generate an intra-operative endoscopic image of an exterior of a subregion of a blood vessel tree within an anatomical region;

a robot configured to move the endoscope within the anatomical region exterior to blood vessels of the blood vessel tree;

an endoscope controller configured to;

generate an endoscopic path exterior to the blood vessels within the anatomical region, wherein the endoscopic path follows vessels of the blood vessel tree and the endoscopic path is derived from matching a graphical representation of the intra-operative endoscopic image of the subregion of the blood vessel tree to a graphical representation of a pre-operative three-dimensional image of the blood vessel tree,

overlay the geometrical representation of the pre-operative three-dimensional image of the exterior of the blood vessel tree onto the intra-operative endoscopic image of the subregion of the blood vessel tree corresponding to the subgraph in accordance with the matching of the graphical representation of the intra-operative endoscopic image of the blood vessel tree to the graphical representation of the pre-operative three-dimensional image of the blood vessel tree, and

generate the endoscopic path within the geometrical representation of the pre-operative three-dimensional image of the exterior of the blood vessel tree overlaid onto the intra-operative endoscopic image of the corresponding portion of the blood vessel tree; and

a robot controller configured to command the robot to move the endoscope within the anatomical region exterior to the blood vessels in accordance with the endoscopic path;

wherein the endoscopic controller is further configured to update the generating of the endoscopic path, the overlay, and the generating of the endoscopic path within the geometrical representation as the robot moves the endoscope within the anatomical region exterior to blood vessels of the blood vessel tree.

2. The robot guiding system of claim 1, wherein the matching of the graphical representation of the intra-operative endoscopic image of the blood vessel tree to the graphical representation of the pre-operative three-dimensional image of the blood vessel tree includes:

generating a main graph derived from the geometrical representation of the pre-operative three-dimensional image of the blood vessel tree;

generating a subgraph derived from a geometrical representation of the intra-operative endoscopic image of the subregion of the blood vessel tree; and

matching the subgraph to the main graph.

3. The robot guiding system of claim 2, wherein the matching of the subgraph to the main graph includes matching nodes of the subgraph to nodes of subregions of the main graph to locate one or more candidate subregions of the main graph corresponding to the subgraph;

and

wherein one of the plurality of the candidate subregions located from the matching is selected as a match of the subgraph to the main graph.

4. The robot guiding system of claim 1, wherein the blood vessel tree is a coronary artery tree.

5. A robot guiding system comprising:

an endoscope including an endoscope camera configured to generate an intra-operative endoscopic image of an exterior of a subregion of a blood vessel tree within an anatomical region;

a robot configured to move the endoscope within the anatomical region exterior to blood vessels of the blood vessel tree;

an endoscope controller configured to generate an endoscopic path exterior to the blood vessels within the anatomical region, wherein the endoscopic path is derived from matching a graphical representation of the intra-operative endoscopic image of the subregion of the blood vessel tree to a graphical representation of a pre-operative three-dimensional image of the blood vessel tree, wherein the matching of the graphical representation of the intra-operative endoscopic image of the blood vessel tree to the graphical representation of the pre-operative three-dimensional image of the blood vessel tree includes;

generating a main graph derived from the geometrical representation of the pre-operative three-dimensional image of the blood vessel tree, wherein the main graph includes a main set of nodes representative of each furcation of the blood vessel tree within the pre-operative three-dimensional image of the blood vessel tree,

generating a subgraph derived from a geometrical representation of the intra-operative endoscopic image of the subregion of the blood vessel tree, wherein the subgraph includes a subset of the main set of nodes, the subset of nodes being representative of each furcation of the blood vessel tree within the intra-operative endoscopic image of the blood vessel tree, and

matching the subgraph to the main graph, wherein the matching of the subgraph to the main graph includes establishing at least one of a vertical ordering and a horizontal ordering of the nodes in the main graph;

a robot controller configured to command the robot to move the endoscope within the anatomical region exterior to the blood vessels in accordance with the endoscopic path.

6. A robot guiding method, comprising:

with an endoscope mounted camera, generating an intra-operative endoscopic image of a subregion of a blood vessel tree visible to the endoscope within an anatomical region;

with a processor, matching a graphical representation of the intra-operative endoscopic image of the subregion of the blood vessel tree to a graphical representation of a pre-operative three-dimensional image of the blood vessel tree, wherein the matching of the graphical representation of the intra-operative endoscopic image of the subregion of the blood vessel tree to the graphical representation of the pre-operative three-dimensional image of the blood vessel tree includes;

with the processor, generating a main graph derived from a geometrical representation of the pre-operative three-dimensional image of the blood vessel tree;

with the processor, generating a subgraph derived from a geometrical representation of the intra-operative endoscopic image of the subregion of the blood vessel tree; and

with the processor, matching the subgraph to subregions of the main graph to locate a main graph subregion corresponding to the subgraph;

with the processor, generating a path within the anatomical region that follows vessels of the blood vessel tree from a first location in the subregion of the blood vessel tree visible to the endoscope camera to a second location in the pre-operative three-dimensional image of the blood vessel tree;

with a robot controller, commanding a robot to move the endoscope within the anatomical region in accordance with the path to a location in the anatomical region corresponding to the second location;

with the processor, controlling a display device to overlay the geometrical representation of the pre-operative three-dimensional image of the blood vessel tree onto the intra-operative endoscopic image of the subregion of the blood vessel tree in accordance with the matching of the graphical representation of the intra-operative endoscopic image of the subregion of the blood vessel tree to the graphical representation of the pre-operative three-dimensional image of the blood vessel tree; and

with the processor, updating the matching and controlling the display device to update the overlay of the geometrical representation of the pre-operative three-dimensional image of the blood vessel tree onto the intra-operative endoscopic image of the subregion of the blood vessel tree as the robot moves the endoscope within the anatomical region in accordance with the path.

7. The robot guiding method of claim 6, wherein the main graph includes a main set of nodes representative of each furcation of the blood vessel tree within the pre-operative three-dimensional image of the blood vessel tree;

and

wherein the subgraph includes a subset of the main set of nodes, the subset of nodes being representative of each furcation of the blood vessel tree within the intra-operative endoscopic image of the subregion of the blood vessel tree visible to the endoscope camera.

8. A robot guiding system, comprising:

an endoscope including a surgical element and an optical camera configured to generate an intra-operative endoscope image data;

a robot controller configured to control a robot to move the endoscope through an anatomical region of a patient;

a display device; and

one or more processors configured to;

generate a preoperative 3D image of a blood vessel tree from image data generated by a 3D diagnostic imaging system,

generate an intra-operative endoscope image of a subregion of the blood vessel tree visible to the endoscope camera,

match the intra-operative endoscope image and a corresponding subregion of the pre-operative 3D blood vessel tree image,

overlay the matched intra-operative endoscope image and the pre-operative 3D blood vessel tree image,

control the display device to display the overlaid matched intra-operative endoscope and pre-operative 3D blood vessel tree images and a path from the a selected location in the intra-operative endoscope image to a selected surgical site in the pre-operative 3D blood vessel tree image, wherein the path follows vessels of the blood vessel tree, and

update the matching, overlaying, and displaying as the endoscope is moved along the path to provide a visual confirmation that the matching was successful.

9. The robot guiding system of claim 8, wherein the processor is further configured to:

process the pre-operative 3D blood vessel image to generate a main graph in which the blood vessel tree is depicted by nodes representing furcations of the blood vessel tree interconnected by lines depicting vessels of the blood vessel tree;

process the intra-operative endoscope image to generate a subgraph in which furcations of blood vessels are represented by nodes interconnected by lines representing vessels;

wherein the matching includes matching the subgraph with the main graph to find the subregion of the blood vessel tree that corresponds to the subregion of the blood vessel tree visible in the intra-operative endoscope image.

10. A robot guiding system comprising:

an endoscope including a surgical element and an optical camera configured to generate an intra-operative endoscope image data;

a robot controller configured to control a robot to move the endoscope through an anatomical region of a patient;

a display device; and

one or more processors configured to;

generate a preoperative 3D image of a blood vessel tree from image data generated by a 3D diagnostic imaging system,

generate an intra-operative endoscope image of a subregion of the blood vessel tree visible to the endoscope camera,

match the intra-operative endoscope image and a corresponding subregion of the pre-operative 3D blood vessel tree image,

overlay the matched intra-operative endoscope image and the pre-operative 3D blood vessel tree image,

control the display device to display the overlaid matched intra-operative endoscope and pre-operative 3D blood vessel tree images and a path from the a selected location in the intra-operative endoscope image to a selected surgical site in the pre-operative 3D blood vessel tree image,

process the pre-operative 3D blood vessel image to generate a main graph in which the blood vessel tree is depicted by nodes representing furcations of the blood vessel tree interconnected by lines depicting vessels of the blood vessel tree; and

process the intra-operative endoscope image to generate a subgraph in which furcations of blood vessels are represented by nodes interconnected by lines representing vessels;

wherein the matching includes matching the subgraph with the main graph including establishing at least one of a vertical ordering and a horizontal ordering of the nodes in the main graph to find the subregion of the blood vessel tree that corresponds to the subregion of the blood vessel tree visible in the intra-operative endoscope image.

11. A robot guiding system comprising:

an endoscope including a surgical element and an optical camera configured to generate an intra-operative endoscope image data;

a robot controller configured to control a robot to move the endoscope through an anatomical region of a patient;

a display device; and

one or more processors configured to;

generate a preoperative 3D image of a blood vessel tree from image data generated by a 3D diagnostic imaging system,

process the pre-operative 3D blood vessel image to generate a main graph in which the blood vessel tree is depicted by nodes representing furcations of the blood vessel tree interconnected by lines depicting vessels of the blood vessel tree;

generate an intra-operative endoscope image of a subregion of the blood vessel tree visible to the endoscope camera,

process the intra-operative endoscope image to generate a subgraph in which furcations of blood vessels are represented by nodes interconnected by lines representing vessels;

match the intra-operative endoscope image and a corresponding subregion of the pre-operative 3D blood vessel tree image, wherein the matching of the subgraph to the main graph includes;

matching the nodes of the subgraph to the nodes of the main graph to generate a plurality of candidate matches; and

wherein the one or more processors are further configured to display the candidate matches for user selection;

overlay the matched intra-operative endoscope image and the pre-operative 3D blood vessel tree image,

control the display device to display the overlaid matched intra-operative endoscope and pre-operative 3D blood vessel tree images and a path from a selected location in the intra-operative endoscope image to a selected surgical site in the pre-operative 3D blood vessel tree image.

12. A robot guiding system comprising:

an endoscope including a surgical element and an optical camera configured to generate an intra-operative endoscope image data;

a robot controller configured to control a robot to move the endoscope through an anatomical region of a patient;

a display device; and

one or more processors configured to;

generate a preoperative 3D image of a blood vessel tree from image data generated by a 3D diagnostic imaging system, wherein the blood vessel tree is a coronary artery tree,

generate an intra-operative endoscope image of a subregion of the blood vessel tree visible to the endoscope camera,

match the intra-operative endoscope image and a corresponding subregion of the pre-operative 3D blood vessel tree image,

overlay the matched intra-operative endoscope image and the pre-operative 3D blood vessel tree image,

control the display device to display the overlaid matched intra-operative endoscope and pre-operative 3D blood vessel tree images and a path from the a selected location in the intra-operative endoscope image to a selected surgical site in the pre-operative 3D blood vessel tree image.