Endoscope observation assistance system, method, apparatus and program

• US 9,220,468 B2
• Filed: 03/30/2011
• Issued: 12/29/2015
• Est. Priority Date: 03/31/2010
• Status: Active Grant

• Current Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Original Assignee: Fujifilm Corporation (Fujifilm Holdings Corporation)
• Inventors: Kitamura, Yoshiro, Nakamura, Keigo
• Primary Examiner(s): ROZANSKI, MICHAEL T
• Application Number:
  US13/636,829
• Publication Number:
  US 20130018255A1
• Time in Patent Office: 1,735 Days
• Field of Search: 600/101, 600/103, 600/104, 600/111, 600/424, 600/476, 382/128
• US Class Current: 1/1
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 2034/2055 Optical tracking systems
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 6/12 Arrangements for detecting or locating foreign bodies
• CPC Class Codes: A61B 90/361 Image-producing devices, e.g. surgical cameras
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2200/04 involving 3D image data
• CPC Class Codes: G06T 2210/41 Medical

Abstract

A virtual endoscope image generation unit receives, as an input, a three-dimensional medical image formed by a three-dimensional medical image formation unit, and generates a virtual endoscope image representing the body cavity viewed from a position in the three-dimensional medical image corresponding to the position of a treatment tool detected in real time by a treatment tool position detection unit. The generated virtual endoscope image is displayed on a WS display.

Claims

1. An endoscope observation assistance system comprising:

an image formation system configured to form a three-dimensional medical image representing a body cavity of a subject to be examined;

a treatment tool position detection system that detects, in real time, a position of a treatment tool inserted into the body cavity and detects a direction in which the treatment tool moves straight forward at the same time as detection of the position of the treatment tool;

one or more image processors configured to generate a virtual endoscope image, the one or more image processors receiving the three-dimensional medical image as an input, and generating a virtual endoscope image representing the body cavity viewed from a corresponding treatment tool position, which is a position in the three-dimensional medical image corresponding to the detected position of the treatment tool; and

a display that displays the virtual endoscope image,

wherein the one or more image processors generate the virtual endoscope image in which image information on a treatment tool visual line from the corresponding treatment tool position is projected, the treatment tool visual line having a direction in the three-dimensional medical image corresponding to the direction in which the treatment tool moves straight forward,

the one or more image processors generate the virtual endoscope image that identifiably represents a position in the virtual endoscope image at which the image information on the treatment tool visual line has been projected, and

an insertion position of the body surface at which the treatment tool is inserted into the body cavity differs from an insertion position of the body surface at which an endoscope is inserted into the body cavity.

2. The endoscope observation assistance system, according to claim 1, wherein the one or more image processors generate the virtual endoscope image by determining an angle of view of the virtual endoscope image in such a manner that the specified position of interest is included in a field of view of the virtual endoscope image.

3. The endoscope observation assistance system, according to claim 1, wherein one or more image processors generate the virtual endoscope image in which the treatment tool visual line is a center visual line, which is a visual line at the center of the field of view of the virtual endoscope image.

4. The endoscope observation assistance system, according to claim 1, wherein the one or more image processors specify, as a position of interest, a position of a structure of interest in the body cavity in the three-dimensional medical image, and

the one or more image processors generate the virtual endoscope image in which a visual line from the corresponding treatment tool position toward the specified position of interest is a center visual line of the virtual endoscope image.

5. The endoscope observation assistance system, according to claim 1, wherein the one or more image processors generate the virtual endoscope image that identifiably represents at least a position of interest in a structure of interest.

6. The endoscope observation assistance system, according to claim 5, wherein the one or more image processors detect, in the three-dimensional medical image, a second structure of interest in the body cavity, and

wherein the one or more image processors generate the virtual endoscope image that identifiably represents the second structure of interest.

7. The endoscope observation assistance system, according to claim 6, wherein one of the structure of interest and the second structure of interest is a region on which an endoscopic surgery is to be performed, and the other one of the structure of interest and the second structure of interest is an anatomical structure that requires caution in the endoscopic surgery.

8. The endoscope observation assistance system, according to claim 1,

wherein the one or more image processors present a warning when the corresponding treatment tool position and a structure of interest are close to each other to such an extent to satisfy a predetermined criterion.

9. The endoscope observation assistance system, according to claim 1, wherein a structure of interest is a region on which an endoscopic surgery is to be performed.

10. The endoscope observation assistance system, according to claim 1, wherein a structure of interest is an anatomical structure that requires caution in an endoscopic surgery.

11. The endoscope observation assistance system, according to claim 1, wherein the one or more image processors determine pixel values of the virtual endoscope image based on a distance from the corresponding treatment tool position to a surface of a structure in the body cavity.

12. The endoscope observation assistance system, according to claim 1, wherein the one or more image processors determine pixel values of the virtual endoscope image by using a color template that is defined so that an external view of each part in the body cavity represented in the virtual endoscope image is substantially the same as an external view of each part in the body cavity represented in a real endoscope image obtained by imaging by an endoscope.

13. The endoscope observation assistance system, according to claim 1, the system further comprising:

a real endoscope image formation processor that forms a real endoscope image representing the body cavity by imaging in real time by an endoscope,

wherein the display further displays the real endoscope image formed substantially at the same timing as detection of the position of the treatment tool used at the time of generation of the virtual endoscope image.

14. The endoscope observation assistance system as defined in claim 1, wherein the one or more image processors specify, as a position of interest, a position of a structure of interest in the body cavity in the three-dimensional medical image, and

the one or more image processors generate a virtual endoscope image, a field of view of which includes the specified position of interest.

15. An endoscope observation assistance system, comprising:

an image formation system configured to form a three-dimensional medical image representing a body cavity of a subject to be examined;

a treatment tool position detection system that detects, in real time, a position of a treatment tool inserted into the body cavity;

one or more image processors configured to generate a virtual endoscope image, the one or more image processors receiving the three-dimensional medical image as an input, and generating the virtual endoscope image representing the body cavity viewed from a corresponding treatment tool position, which is the same position in the three-dimensional medical image as the detected position of the treatment tool;

a display that displays the virtual endoscope image; and

an endoscope position detection system that detects the position of an endoscope inserted into the body cavity,

wherein the one or more image processors generate the virtual endoscope image, a field of view of which includes a corresponding endoscope position, which is the same position in the three-dimensional medical image as the detected position of the endoscope, the virtual endoscope image identifiably representing the corresponding endoscope position,

an insertion position of the body surface at which the treatment tool is inserted into the body cavity differs from an insertion position of the body surface at which the endoscope is inserted into the body cavity, and

the one or more image processors generate the virtual endoscope image in which image information on a treatment tool visual line from the corresponding treatment tool position is projected.

16. An endoscope observation assistance method comprising:

forming a three-dimensional medical image representing a body cavity of a subject to be examined;

detecting, in real time, a position of a treatment tool inserted into the body cavity;

detecting a direction in which the treatment tool moves straight forward at the same time as detection of the position of the treatment tool;

receiving the three-dimensional medical image as an input, and generating a virtual endoscope image representing the body cavity viewed from a corresponding treatment tool position, which is a position in the three-dimensional medical image corresponding to the detected position of the treatment tool; and

displaying the virtual endoscope image,

wherein the virtual endoscope image is generated in which image information on a treatment tool visual line from the corresponding treatment tool position is projected, the treatment tool visual line having a direction in the three-dimensional medical image corresponding to the direction in which the treatment tool moves straight forward,

the virtual endoscope image that is generated identifiably represents a position in the virtual endoscope image at which the image information on the treatment tool visual line has been projected, and

an insertion position of the body surface at which the treatment tool is inserted into the body cavity differs from an insertion position of the body surface at which the endoscope is inserted into the body cavity.

17. An endoscope observation assistance apparatus comprising:

one or more image processors configured to

obtain a three-dimensional medical image representing a body cavity of a subject to be examined,

obtain a position of a treatment tool inserted into the body cavity and a direction in which the treatment tool moves straight forward, the position having been detected in real time by a treatment tool position detection system, the direction in which the treatment tool moves straight forward having been detected by the treatment tool position detection system at the same time as detection of the position of the treatment tool, and receive the three-dimensional medical image as an input, and generate a virtual endoscope image representing the body cavity viewed from a corresponding treatment tool position, which is a position in the three-dimensional medical image corresponding to the obtained position of the treatment tool; and

a display controller that makes a display display the virtual endoscope image,

wherein the one or more image processors generate the virtual endoscope image in which image information on a treatment tool visual line from the corresponding treatment tool position is projected, the treatment tool visual line having a direction in the three-dimensional medical image corresponding to the direction in which the treatment tool moves straight forward,

the one or more image processors generate the virtual endoscope image that identifiably represents a position in the virtual endoscope image at which the image information on the treatment tool visual line has been projected, and

an insertion position of the body surface at which the treatment tool is inserted into the body cavity differs from an insertion position of the body surface at which the endoscope is inserted into the body cavity.

18. A non-transitory computer-readable storage medium storing therein an endoscope observation assistance program for causing a computer to execute the steps of:

obtaining a three-dimensional medical image representing a body cavity of a subject to be examined;

obtaining a position of a treatment tool inserted into the body cavity, the position having been detected in real time by a treatment tool position detection system;

obtaining a direction in which the treatment tool moves straight forward, the direction in which the treatment tool moves straight forward having been detected at the same time as detection of the position of the treatment tool;

receiving the three-dimensional medical image as an input, and generating a virtual endoscope image representing the body cavity viewed from a corresponding treatment tool position, which is a position in the three-dimensional medical image corresponding to the obtained position of the treatment tool; and

making a display display the virtual endoscope image,

wherein the virtual endoscope image is generated in which image information on a treatment tool visual line from the corresponding treatment tool position is projected, the treatment tool visual line having a direction in the three-dimensional medical image corresponding to the direction in which the treatment tool moves straight forward,

the virtual endoscope image that is generated identifiably represents a position in the virtual endoscope image at which the image information on the treatment tool visual line has been projected, and

an insertion position of the body surface at which the treatment tool is inserted into the body cavity differs from an insertion position of the body surface at which the endoscope is inserted into the body cavity.

19. An endoscope observation assistance method comprising:

forming a three-dimensional medical image representing a body cavity of a subject to be examined;

detecting, in real time, a position of a treatment tool inserted into the body cavity;

detecting a position of an endoscope inserted into the body cavity;

receiving the three-dimensional medical image as an input, and generating a virtual endoscope image representing the body cavity viewed from a corresponding treatment tool position, which is the same position in the three-dimensional medical image as the detected position of the treatment tool; and

displaying the virtual endoscope image,

wherein a field of view of the generated virtual endoscope image includes a corresponding endoscope position, which is the same position in the three-dimensional medical image as the detected position of the endoscope, the virtual endoscope image identifiably representing the corresponding endoscope position,

an insertion position of the body surface at which the treatment tool is inserted into the body cavity differs from an insertion position of the body surface at which the endoscope is inserted into the body cavity, and

the virtual endoscope image is generated in which image information on a treatment tool visual line from the corresponding treatment tool position is projected.

20. An endoscope observation assistance apparatus comprising:

one or more image processors configured to

obtain a three-dimensional medical image representing a body cavity of a subject to be examined,

obtain a position of a treatment tool inserted into the body cavity, the position having been detected in real time by a treatment tool position detection system,

obtain a position of an endoscope inserted into the body cavity, and

receive the three-dimensional medical image as an input, and generate a virtual endoscope image representing the body cavity viewed from a corresponding treatment tool position, which is the same position in the three-dimensional medical image as the obtained position of the treatment tool; and

a display controller that makes a display display the virtual endoscope image,

wherein a field of view of the generated virtual endoscope image includes a corresponding endoscope position, which is the same position in the three-dimensional medical image as the detected position of the endoscope, the virtual endoscope image identifiably representing the corresponding endoscope position,

an insertion position of the body surface at which the treatment tool is inserted into the body cavity differs from an insertion position of the body surface at which the endoscope is inserted into the body cavity, and

the one or more image processors generate the virtual endoscope image in which image information on a treatment tool visual line from the corresponding treatment tool position is projected.

21. A non-transitory computer-readable storage medium storing therein an endoscope observation assistance program for causing a computer to execute the steps of:

obtaining a three-dimensional medical image representing a body cavity of a subject to be examined;

obtaining a position of a treatment tool inserted into the body cavity, the position having been detected in real time by a treatment tool position detection system;

obtain a position of an endoscope inserted into the body cavity,

receiving the three-dimensional medical image as an input, and generating a virtual endoscope image representing the body cavity viewed from a corresponding treatment tool position, which is the same position in the three-dimensional medical image as the obtained position of the treatment tool; and

making a display display the virtual endoscope image,

wherein a field of view of the generated virtual endoscope image includes a corresponding endoscope position, which is the same position in the three-dimensional medical image as the detected position of the endoscope, the virtual endoscope image identifiably representing the corresponding endoscope position,

an insertion position of the body surface at which the treatment tool is inserted into the body cavity differs from an insertion position of the body surface at which the endoscope is inserted into the body cavity, and

the virtual endoscope image is generated in which image information on a treatment tool visual line from the corresponding treatment tool position is projected.