Virtualized endoscope system

• US 6,346,940 B1
• Filed: 02/27/1998
• Issued: 02/12/2002
• Est. Priority Date: 02/27/1997
• Status: Expired due to Term

• Current Assignee: Toshiba Medical Systems Corporation (Canon Inc.)
• Original Assignee: Kabushiki Kaisha Toshiba (Toshiba Corporation)
• Inventors: Fukunaga, Tomohisa
• Primary Examiner(s): Vo, Cliff N.
• Assistant Examiner(s): Nguyen, Kimbinh T.
• Application Number:
  US09/031,661
• Time in Patent Office: 1,446 Days
• Field of Search: 345/419, 345/435, 345/473, 345/424, 345/420, 345/427, 345/629, 345/630, 600/111, 600/427, 128/653.1, 382/103
• US Class Current: 345/427
• CPC Class Codes: G06T 15/20 Perspective computation
• CPC Class Codes: G06T 19/003 Navigation within 3D models or images
• CPC Class Codes: G06T 2210/41 Medical
• CPC Class Codes: G06T 2219/028 Multiple view windows (top-side-front-sagittal-orthogonal)

Abstract

An image processing system displaying an endoscopic image of a three-dimensional object from a desired viewpoint position and view direction. A three-dimensional model preparation unit prepares a three-dimensional model of the object and an endoscopic image preparation unit prepares a first image of the three-dimensional model as viewed from the viewpoint position and view direction. A simulated endoscopic examination of a virtual subject can be realized by permitting an operator to change the viewpoint position and view direction to coincide with the movement of a virtual endoscope. The usefulness of the virtual endoscope is enhanced with various features of the invention which facilitate control over the virtual endoscope and make the virtual images useful in the insertion of a real endoscope. Such features include: the superimposition of endoscopic images; an indicator image which indicates the orientation of the virtual endoscope relative to the organ being examined; a compass image indicating the orientation of a virtual subject relative to the virtual endoscope tip; guiding markers which indicate the direction in which the virtual or real endoscope is to be advanced; and focus markers which reveal the location of predetermined conditions such as a nidus within the virtual subject. A virtual operation device, modeled after an actual endoscope operating member, is used to control movement of the virtual endoscope tip within the virtual subject.

Claims

1. An image processing system for displaying a three-dimensional object from a viewpoint position and in a view direction, comprising:

a three-dimensional model preparation unit configured to construct a three-dimensional model of the three-dimensional object;

an input unit configured to change the viewpoint position and the view direction by receiving commands to change the viewpoint position and the view direction from an input device;

an endoscopic image preparation unit configured to prepare for displaying an endoscopic image of the three-dimensional object as viewed from the viewpoint position and in the view direction;

an endoscopic image display which displays the endoscopic image;

an outer shape image preparation unit configured to prepare an outer shape image of the three-dimensional object from the three-dimensional model;

an indicator preparation unit configured to prepare for displaying an indicator image based on the viewpoint position, the view direction and a body position reference information, wherein the indicator image indicates the viewpoint position, the view direction and a reference direction; and

an outer shape image display which displays the indicator image and the outer shape image simultaneously next to the displayed endoscopic image such that the location and orientation of the indicator image relative to the three-dimensional model indicates at least the viewpoint position and view direction relative to the three-dimensional model.

2. The image processing system of claim 1, wherein the indicator image comprises:

a first element indicative of the viewpoint position;

a second element indicative of the view direction; and

a third element indicative of the reference direction.

3. The image processing system of claim 2, wherein the first, second, and third elements are different colors.

4. The image processing system of claim 1, further comprising:

a guiding marker preparation unit configured to prepare for display on the endoscopic image an at least one guiding marker indicative of a direction in which an endoscope tip is to be advanced; and

said endoscopic image display configured to display the endoscopic image and the at least one guiding marker.

5. A method for displaying a three-dimensional object from a viewpoint position and in a view direction, said method comprising the steps of:

constructing a three-dimensional model of the three-dimensional object;

preparing an outer shape image of the three-dimensional object from the three dimensional model and a body position reference information;

displaying the outer shape image in an outer shape image display;

preparing an indicator image indicating at least the viewpoint position and the view direction; and

displaying the indicator image and the outer shape image simultaneously such that the location and orientation of the indicator image relative to the outer shape image indicates at least the viewpoint position and the view direction relative to the three-dimensional model.

6. The method of claim 5 further comprising the step of:

providing the indicator image with a first element indicating the viewpoint position, a second element indicating the view direction, and a third element indicating a reference direction.

7. The method of claim 5, wherein the step of displaying the indicator image further includes the step of:

displaying the first, second, and third elements of the indicator image in different colors.

8. A computer-readable medium having computer-executable instructions for performing the steps in the method recited in any one of claims 5, 6 or 7.

9. The method of claim 5, further comprising:

preparing an at least one guiding marker for display on the endoscopic image, said guiding marker indicating a direction in which an endoscope is to be advanced; and

displaying the endoscopic image and the at least one guiding marker in an endoscopic image display.

10. The method of claim 9, further comprising the step of:

selecting the location of the guiding marker on the endoscopic image.

11. The method of claim 9, further comprising the step of:

storing the location of the guiding marker relative to the three-dimensional model.

12. The method of claim 9, wherein the step of preparing the at least one guiding marker includes the step of:

providing the at least one guiding marker with the appearance of at least one of an X-shape, an up arrow, a down arrow, a right arrow, a left arrow, optional direction arrows, and text.

13. A computer-readable medium having computer-executable instructions for performing the steps in the method recited in any one of claims 9-12.

14. An image processing system for displaying a three-dimensional object from a viewpoint position and in a view direction, comprising:

a three-dimensional model preparation unit configured to construct a three-dimensional model of the three-dimensional object;

an endoscopic image preparation unit configured to prepare for displaying an endoscopic image of the three-dimensional model as viewed from the viewpoint position and in the view direction;

a compass image preparation unit configured to prepare for displaying a compass image, wherein the compass image comprises projections in the axial direction of a subject and in directions perpendicular to the axial direction of a virtual subject; and

an endoscopic image display which displays the endoscopic image and displays the compass image over the endoscopic image.

15. The image processing system of claim 14, wherein the compass, image further comprises:

labels proximal to each of the projections, each label being indicative of the direction in which the corresponding projection points relative to the virtual subject.

16. The image processing system of claim 14, wherein the projections which point in opposite directions have the same color and the projections which do not point in opposite directions have different colors.

17. The image processing system of claim 14, wherein the compass image preparation unit comprises:

shaping means for providing the compass image with the shape of the three-dimensional object.

18. The image processing system of claim 14, wherein said shaping means comprises:

means for providing the compass image with a spherical shape.

19. The image processing system of claim 14, wherein the compass image preparation unit comprises:

pattern forming means for providing the compass image with a network pattern.

20. The image processing system of claim 14, wherein the compass image preparation unit comprises:

coloring means for providing each half of the compass image with a different color.

21. The image processing system of claim 17, wherein said shaping means comprises:

means for providing the compass image with the shape of a human body.

22. The image processing system of claim 17, wherein said shaping means comprises:

means for providing the compass image with the shape of an organ.

23. The image processing system of claim 14, further comprising:

an operation path history control unit configured to prepare for display a sequence of past images corresponding to a sequence of past viewpoint positions, view directions, and reference directions;

wherein said endoscopic image display is further configured to display said past images and to display said compass image over said past images.

24. A method for displaying a three-dimensional object from a viewpoint position and in a view direction, said method comprising the steps of:

constructing a three-dimensional model of the three-dimensional object;

preparing for display an endoscopic image of the three-dimensional model as viewed from the viewpoint position and in the view direction;

displaying the endoscopic image in an endoscopic display;

generating for display a compass image including projections in the axial direction of a subject and in directions perpendicular to the axial direction of the subject; and

displaying the compass image over the endoscopic image in the endoscopic display.

25. The method of claim 24, wherein the step of displaying the compass image includes:

displaying the projections which point in opposite directions int he same color; and

displaying the projections which do not point in opposite directions in different colors.

26. The method of claim 24, wherein the step of generating includes the step of:

providing the compass image with the shape of a sphere.

27. The method of claim 24, wherein the step of generating includes the step of:

providing the surface of the compass image with a network pattern.

28. The method of claim 24, wherein the step of displaying includes the step of:

displaying each half of the compass image in a different color.

29. A computer-readable medium having computer-executable instructions for performing the steps in the method recited in any one of claim 24 or 25-28.