Object observation system and method of controlling object observation system

• US 20050033117A1
• Filed: 06/01/2004
• Published: 02/10/2005
• Est. Priority Date: 06/02/2003
• Status: Abandoned Application

• Current Assignee: Olympus Corporation
• Original Assignee: Olympus Corporation
• Inventors: Ozaki, Takashi, Tashiro, Koichi, Nakamura, Takeaki, Fujita, Masaya, Gotanda, Masakazu, Uchikubo, Akinobu
• Application Number:
  US10/858,440
• Publication Number:
  US 20050033117A1
• Time in Patent Office: Days
• US Class Current: 600/109
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/0004 for electronic operation
• CPC Class Codes: A61B 1/00042 for mechanical operation
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/00194 adapted for three-dimensional imaging
• CPC Class Codes: A61B 1/042 characterised by a proximal camera, e.g. a CCD camera
• CPC Class Codes: A61B 1/045 Control thereof
• CPC Class Codes: A61B 17/0281 Abdominal wall lifters
• CPC Class Codes: A61B 2017/00199 with a console, e.g. a control panel with a display
• CPC Class Codes: A61B 2017/00207 with hand gesture control or hand gesture recognition
• CPC Class Codes: A61B 2017/00216 with eye tracking or head position tracking control
• CPC Class Codes: A61B 2034/105 Modelling of the patient, e.g. for ligaments or bones
• CPC Class Codes: A61B 2034/2048 using an accelerometer or inertia sensor
• CPC Class Codes: A61B 2034/2051 Electromagnetic tracking systems
• CPC Class Codes: A61B 2034/256 having a database of accessory information, e.g. including context sensitive help or scientific articles
• CPC Class Codes: A61B 2090/365 augmented reality, i.e. correlating a live optical image with another image
• CPC Class Codes: A61B 2090/372 Details of monitor hardware
• CPC Class Codes: A61B 2090/373 using light, e.g. by using optical scanners
• CPC Class Codes: A61B 2090/502 Headgear, e.g. helmet, spectacles
• CPC Class Codes: A61B 34/20 Surgical navigation systems; Devices for tracking or guiding surgical instruments, e.g. for frameless stereotaxis
• CPC Class Codes: A61B 34/25 : User interfaces for surgical systems
• CPC Class Codes: A61B 34/74 : Manipulators with manual electric input means
• CPC Class Codes: A61B 5/06 : Devices, other than using radiation, for detecting or locating foreign bodies for removing same A61B17/50 ; determining position of probes within or on the body of the patient
• CPC Class Codes: A61B 5/062 : using magnetic field
• CPC Class Codes: A61B 5/1076 : for measuring dimensions inside body cavities, e.g. using catheters A61B3/1005 takes precedence
• CPC Class Codes: A61B 90/36 : Image-producing devices or illumination devices not otherwise provided for
• CPC Class Codes: A61B 90/361 : Image-producing devices, e.g. surgical cameras
• CPC Class Codes: A61B 90/37 : Surgical systems with images on a monitor during operation
• CPC Class Codes: A61B 90/50 : Supports for surgical instruments, e.g. articulated arms
• CPC Class Codes: G06T 15/08 : Volume rendering
• CPC Class Codes: G06T 19/003 : Navigation within 3D models or images
• CPC Class Codes: G06T 2207/30004 : Biomedical image processing
• CPC Class Codes: G06T 7/30 : Determination of transform parameters for the alignment of images, i.e. image registration
• CPC Class Codes: G16H 20/40 : relating to mechanical, radiation or invasive therapies, e.g. surgery, laser therapy, dialysis or acupuncture
• CPC Class Codes: G16H 40/63 : for local operation

Abstract

An object observation system of the present invention has an observation apparatus for observing a body to be examined, a three-dimensional image recording apparatus for recording three-dimensional images, which are obtained in advance, of the body to be examined, and an image constructing apparatus for constructing a three-dimensional image based on images in synchronization with the observation apparatus, which are recorded in the three-dimensional image recording apparatus.

Claims

1. An object observation system, comprising:

an observation apparatus for observing a body to be examined;

a three-dimensional image recording apparatus for recording three-dimensional images, which are obtained in advance, of the body to be examined; and

an image constructing apparatus for constructing a three-dimensional image based on images in synchronization with the observation apparatus, which are recorded in the three-dimensional image recording apparatus.

2. An object observation system according to claim 1, further comprising a positional relationship detecting portion for detecting a relative positional information of the observation apparatus with respect to the body to be examined.

3. An object observation system according to claim 2,

wherein the observation apparatus is an endoscope, and

the image constructing apparatus constructs the three-dimensional image in accordance with the distal end of the endoscope based on information on the positional relationship detected by the positional relationship detecting portion.

4. An object observation system according to claim 3, further comprising a recording portion for recording the three-dimensional image constructed by the image constructing apparatus.

5. An object observation system according to claim 4,

wherein the recording portion records an image resulting from synthesis of an image of the endoscope and the three-dimensional image.

6. An object observation system according to claim 2,

wherein the positional relationship detecting portion has at least one of an amount-of-insertion detecting section for detecting an amount of insertion of the observation apparatus into a body to be examined, an insertion position detecting portion for detecting a position of insertion of the observation apparatus into a body to be examined, an inclination-angle-of-insertion detecting portion for detecting an inclination angle of insertion of the observation apparatus into a body to be examined, a position-of-point-of-vision specifying portion for specifying a position of a point of vision with respect to a part of concern of the body to be examined, and an observation direction detecting portion for detecting information indicating an observation direction of the observation apparatus.

7. An object observation system according to claim 6, comprising:

a first three-dimensional image data creating portion for creating three-dimensional image data relating to the body to be examined corresponding to an image observed by the observation apparatus based on angle information of the inclination angle of insertion detected by the inclination-angle-of-insertion detecting portion; and

a second three-dimensional image data creating portion for creating three-dimensional image data relating to the body to be examined corresponding to a plane having a predetermined angle with respect to an observation image plane of the observation apparatus.

8. An object observation system according to claim 7,

wherein the observation apparatus is an endoscope, and

the image constructing apparatus constructs the three-dimensional image in accordance with the distal end of the endoscope based on information on the positional relationship detected by the positional relationship detecting portion.

9. An object observation system according to claim 6, comprising:

a first three-dimensional image data creating portion for creating three-dimensional image data relating to the body to be examined corresponding to an observation image plane of the observation apparatus based on angle information of the inclination angle of insertion detected by the inclination-angle-of-insertion detecting portion; and

a second three-dimensional image data creating portion for creating three-dimensional image data relating to the body to be examined corresponding to an observation image plane of the observation apparatus based on the axial angle specified by the axial angle specifying portion.

10. An object observation system according to claim 9,

wherein the observation apparatus is an endoscope, and

the image constructing apparatus constructs the three-dimensional image in accordance with the distal end of the endoscope based on information on the positional relationship detected by the positional relationship detecting portion.

11. An object observation system according to claim 1, further comprising:

a three-dimensional image display portion, which can display a three-dimensional image by the image constructing apparatus and an image observed by the observation apparatus; and

a three-dimensional image display control portion for controlling display of the three-dimensional image display portion.

12. An object observation system according to claim 11 in which the observation apparatus is an endoscope, the system further comprising:

a positional relationship detecting portion for detecting a relative positional relationship of the endoscope with respect to the body to be examined.

13. An object observation system according to claim 12,

wherein the positional relationship detecting portion is provided in a trocar holding the endoscope therethrough.

14. An object observation system according to claim 12,

wherein the positional relationship detecting portion is provided in a member provided to the arm of an operator.

17. An object observation system according to claim 1,

wherein the image constructing apparatus has:

an image extracting portion for extracting a desired image from images recorded in the three-dimensional image recording portion; and

an image processing portion for processing the extracted image.

18. An object observation system according to claim 17, further comprising a positional relationship detecting portion for detecting a relative positional relationship of the observation apparatus with respect to the body to be examined.

19. An object observation system according to claim 18,

wherein the observation apparatus is an endoscope, and

the image constructing apparatus constructs the three-dimensional image in accordance with the distal end of the endoscope based on information on the positional relationship detected by the positional relationship detecting portion.

20. An object observation system according to claim 19, further comprising:

a synthesis processing portion for creating a synthesized image by synthesizing multiple extracted images created by the image extracting portion.

21. An object observation system according to claim 1, further comprising:

a display portion for displaying a three-dimensional image constructed by the image constructing apparatus; and

a three-dimensional image change processing portion for performing image processing for changing an image being displayed on the display portion.

22. An object observation system according to claim 21,

wherein the three-dimensional image change processing portion performs the image processing based on a position, direction or display scale of the observation apparatus.

23. An object observation system according to claim 1, further comprising an observation information input portion for inputting focus-point information of the observation apparatus with respect to the body to be examined.

24. An object observation system according to claim 23,

wherein the three-dimensional image constructing apparatus processes a three-dimensional image stored in the three-dimensional image recording apparatus based on information input from the observation information input portion.

25. An object observation system according to claim 3, further comprising:

another image constructing apparatus, which is connected to the positional relationship detecting portion through a communications line, for constructing a three-dimensional image of the body to be examined based on information of the positional information received from the positional relationship detecting portion.

26. An object observation system according to claim 25, further comprising:

a display portion for displaying an observation image of the observation apparatus received through the communications line; and

a support information output portion for creating support information based on the three-dimensional image constructed by the other image constructing apparatus, adding the support information to the observation image and outputting the result through the communications line.

15. An object observation system, comprising:

an endoscope having an insert portion, which can be used for observing a body to be examined;

at least one treating device for performing a treatment on the body to be examined;

a first detecting portion for detecting information indicating an observation direction of the insert portion of the endoscope;

a second detecting portion for detecting information indicating a treatment direction of the treating device;

a three-dimensional image data storing portion for storing three-dimensional image data relating to the body to be examined;

a three-dimensional image data processing portion for creating first and second three-dimensional image data corresponding to the first and second detecting portions by processing the three-dimensional image data based on information detected by the first and second detecting portions;

first and second three-dimensional image display portions, which can display first and second three-dimensional images based on the first and second three-dimensional image data and an endoscopic observation image by the endoscope;

a switching section, which can selectively switch and output, to the first and second three-dimensional image display portion, the first and second three-dimensional image data from the three-dimensional image data processing portion; and

a control portion for controlling the switching section.

16. An object observation system according to claim 15,

wherein the first and second detecting portions have operation portions for instructing display modes of the first and second three-dimensional image display portions.

27. A control method of an object observation system, comprising the steps of:

observing a body to be examined by an observation apparatus;

recording three-dimensional images, which are obtained in advance, of the body to be examined in a three-dimensional image recording apparatus; and

constructing a three-dimensional image based on images in synchronization with the observation apparatus, which are recorded in the three-dimensional image recording apparatus.

28. A control method of an object observation system according to claim 27, further comprising the step of detecting a relative positional information of the observation apparatus with respect to the body to be examined.

29. A control method of an object observation system according to claim 28,

wherein the observation apparatus is an endoscope, and

the step of constructing a three-dimensional image constructs the three-dimensional image in accordance with the distal end of the endoscope based on information on the positional relationship detected by the positional relationship detecting portion.

30. A control method of an object observation system according to claim 29, further comprising the steps of recording the three-dimensional image constructed in the image constructing step in a recording portion.

31. A control method of an object observation system according to claim 30,

wherein the recording step includes a step of recording an image resulting from synthesis of an image of the endoscope and the three-dimensional image.

32. A control method of an object observation system according to claim 28,

wherein the positional relationship detecting step detects at least one of an amount of insertion of the observation apparatus into the body to be examined, a position of insertion of the observation apparatus into the body to be examined, an inclination angle of insertion of the observation apparatus into the body to be examined, a position of a point of vision with respect to a part of concern of the body to be examined and information indicating an observation direction of the observation apparatus.

33. A control method of an object observation system according to claim 32, comprising:

a first step of creating three-dimensional image data relating to the body to be examined corresponding to an image observed by the observation apparatus based on information of the inclination angle of insertion detected by the step of detecting a positional relationship; and

a second step of creating three-dimensional image data relating to the body to be examined corresponding to a plane having a predetermined angle with respect to an observation image plane of the observation apparatus.

34. A control method of an object observation system according to claim 33,

wherein the observation apparatus is an endoscope, and

the step of constructing a three-dimensional image constructs the three-dimensional image in accordance with the distal end of the endoscope based on information on the positional relationship detected by the positional relationship detecting step.

35. A control method of an object observation system according to claim 32, comprising:

a first step of creating three-dimensional image data relating to the body to be examined corresponding to an observation image plane of the observation apparatus based on angle information of the inclination angle of insertion detected by the positional relationship detecting portion; and

a second step of creating three-dimensional image data relating to the body to be examined corresponding to an observation image plane of the observation apparatus based on the axial angle specified by the axial angle specifying portion.

36. A control method of an object observation system according to claim 35,

wherein the observation apparatus is an endoscope, and

the step of constructing a three-dimensional image constructs the three-dimensional image in accordance with the distal end of the endoscope based on information on the positional relationship detected by the positional relationship detecting step.

37. A control method of an object observation system according to claim 27, further comprising the steps of:

displaying a three-dimensional image by the three-dimensional image constructing step and an image observed by the observation apparatus; and

controlling display of the display step.

38. A control method of an object observation system according to claim 37 in which the observation apparatus is an endoscope, the method further comprising the step of:

detecting a relative positional relationship of the endoscope with respect to the body to be examined.

39. A control method of an object observation system according to claim 38,

wherein the positional relationship detecting step is performed by using a detecting portion included in a trocar holding the endoscope therethrough.

40. A control method of an object observation system according to claim 38,

wherein the positional relationship detecting step is performed by using a member detecting portion provided to the arm of an operator.

43. A control method of an object observation system according to claim 27,

wherein the three-dimensional image constructing step has:

a step of extracting a desired image from images recorded in the three-dimensional image recording portion; and

a step of processing the extracted image.

44. A control method of an object observation system according to claim 43, further comprising a step of detecting a relative positional relationship of the observation apparatus with respect to the body to be examined.

45. A control method of an object observation system according to claim 44,

wherein the observation apparatus is an endoscope, and

the three-dimensional image constructing step constructs the three-dimensional image in accordance with the distal end of the endoscope based on information on the positional relationship detected by the positional relationship detecting step.

46. A control method of an object observation system according to claim 45, further comprising:

a step of creating a synthesized image by synthesizing multiple extracted images created by the extracting step.

47. A control method of an object observation system according to claim 27, further comprising the steps of:

displaying a three-dimensional image constructed by the three-dimensional image constructing step; and

performing image processing for changing an image being displayed by the display step.

48. A control method of an object observation system according to claim 47,

wherein the image processing step performs the image processing based on a position, direction or display scale of the observation apparatus.

49. A control method of an object observation system according to claim 27, further comprising the step of inputting focus-point information of the observation apparatus with respect to the body to be examined.

50. A control method of an object observation system according to claim 49,

wherein the three-dimensional image constructing step processes a three-dimensional image stored in the three-dimensional image recording apparatus based on information input at the focus point information input step.

51. A control method of an object observation system according to claim 29, further comprising:

another image constructing apparatus, which is connected to the positional relationship detecting portion through a communications line, for constructing a three-dimensional image of the body to be examined based on information of the positional information received from the positional relationship detecting portion.

52. A control method of an object observation system according to claim 51, further comprising:

a display portion for displaying an observation image of the observation apparatus received through the communications line; and

a support information output portion for creating support information based on the three-dimensional image constructed by the other image constructing apparatus, adding the support information to the observation image and outputting the result through the communications line.

41. A control method of an object observation system including an endoscope having an insert portion, which can be used for observing a body to be examined and at least one treating device for performing a treatment on the body to be examined, the method comprising:

a first step of detecting information indicating an observation direction of the insert portion of the endoscope;

a second step of detecting information indicating a treatment direction of the treating device;

a step of storing three-dimensional image data relating to the body to be examined;

a step of creating first and second three-dimensional image data corresponding to the first and second detecting steps by processing the three-dimensional image data based on information detected by the first and second detecting steps;

a step of displaying first and second three-dimensional images based on the first and second three-dimensional image data and an endoscopic observation image by the endoscope;

a step of selectively switching the first and second three-dimensional image data for the display step; and

a step of controlling the switching step.

42. A control method of an object observation system according to claim 41,

wherein the first and second steps have operation steps for instructing display modes of the display steps.