Endoscopic image processing device for estimating three-dimensional shape of object based on detection of same point on a plurality of different images

• US 5,432,543 A
• Filed: 03/25/1993
• Issued: 07/11/1995
• Est. Priority Date: 03/27/1992
• Status: Expired due to Term

• Current Assignee: Olympus Optical Corporation Limited (Olympus Corporation)
• Original Assignee: Olympus Optical Corporation Limited (Olympus Corporation)
• Inventors: Oyama, Nagaaki, Hasegawa, Jun, Yamaguchi, Masahiro, Nonami, Tetsuo
• Primary Examiner(s): Chin, Tommy P.
• Assistant Examiner(s): Au, A.
• Application Number:
  US08/037,004
• Time in Patent Office: 838 Days
• Field of Search: 348/45, 348/65, 348/136, 348/74, 348/71, 348/699, 382/6, 382/1, 382/17, 359/107, 128/660.03, 128/660.06
• US Class Current: 348/45
• CPC Class Codes: A61B 1/0005 combining images e.g. side-by-side, superimposed or tiled
• CPC Class Codes: A61B 1/00149 using articulated arms
• CPC Class Codes: A61B 1/05 characterised by the image sensor, e.g. camera, being in the distal end portion
• CPC Class Codes: A61B 5/1077 Measuring of profiles
• CPC Class Codes: G06T 2207/10012 Stereo images
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 7/579 from motion
• CPC Class Codes: H04N 13/10 Processing, recording or transmission of stereoscopic or multi-view image signals
• CPC Class Codes: H04N 13/204 using stereoscopic image cameras stereoscopic photography G03B35/00
• CPC Class Codes: H04N 2013/0081 Depth or disparity estimation from stereoscopic image signals
• CPC Class Codes: H04N 2013/0092 Image segmentation from stereoscopic image signals
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 7/183 for receiving images from a single remote source

Abstract

The position of the same point on each of the images of a single object formed by the imaging means of an endoscope is detected, and shift maps between the respective images are formed using the detected positions. Each of the imaging positions of the imaging means is estimated from the shift maps, and a relative shape of the object is then estimated from the imaging positions. Since a relative shape of the object is estimated using the correlation between the respective images, a relative shape of the object can be estimated even when not only the imaging means but also the object are moved. An absolute shape of the object can also be estimated using a length as a reference value.

Claims

1. An endoscopic image processing device comprising:

an endoscope provided with imaging means;

position detection means for detecting the position of the same point on each of a plurality of images of the same object formed by said imaging means at a plurality of imaging positions;

position estimating means for estimating each position of said object and said imaging means from a shift amount of the position of the same point on each of the images and a rotation matrix and a translation vector calculated using the shift amount;

shape estimating means for estimating a three-dimensional shape of said object using the positional information of said object and said imaging means which is estimated by said position estimating means; and

display means for displaying the three-dimensional shape estimated by said shape estimating means.

2. An endoscopic image processing device according to claim 1, further comprising:

repeated processing control means for repeating at least one of a plurality of processes including a first process of detecting the positions of the same point on said plurality of images, a second process of estimating each position of said imaging means and a third process of estimating a three-dimensional shape of said object.

3. An endoscopic image processing device according to claim 1, wherein said position detection means comprises correction means for correcting the aberration caused by an objective light system of said imaging means.

4. An endoscopic image processing device according to claim 1, wherein said position detection means detects the relative position of a point on one of two images to the same point on the other image by setting a first area on one of the two images and detecting a second area on the other of the two images, said second area having substantially the same size as that of said first area and exhibiting the maximum correlation value between images of said first area and said second area.

6. An endoscopic image processing device according to claim 1, wherein said position estimating means computes the relative positional relation between said imaging means and said object from the results of said computation means.

7. An endoscopic image processing device according to claim 1,

wherein said position estimating means comprises 8-point algorithm computation means for determining the relative movement of said imaging means to said object by solving for relations between first and second points on said object and which have unknowns as movement parameters of said imaging means resolved into said rotation matrix and said translation vector, using the movement parameters when said first point is moved to said second point by relative movement of said imaging means and when a correspondence between the points of the images respectively formed by projection of said first and second points on a coordinate system of central projection is provided, and

wherein said position estimating means computes the relative positional relation by said computation means employing a least squares algorithm for determining the translation vector so as to minimize the relation between said first and second points using the movement parameters of said imaging means, which are resolved in the rotation matrix and the translation vector.

8. An endoscopic image processing device according to claim 7, wherein the relation estimated by said position estimating means is derived on the assumption that the vector connecting the position of said imaging means and said first point, the vector obtained from the vector connecting the position of said imaging means and said second point and said rotation matrix, and said translation vector are present in the same plane.

9. An endoscopic image processing device according to claim 1, wherein said shape estimating means comprises coordinate transformation means for transforming the relations between said object and a plurality of positions of said imaging means into a predetermined coordinate system to compute a three-dimensional shape of said object.

10. An endoscopic image processing device according to claim 9, wherein said shape estimating means computes a three-dimensional shape of said object using a weighting function determined by the relations between the three-dimensional shape of said object obtained by said coordinate transformation means and the imaging positions of said imaging means.

11. An endoscopic image processing device according to claim 9, wherein said shape estimating means comprises repeated estimation means for repeatedly estimating a three-dimensional shape of said object so as to minimize an evaluation function into which a weighting function is introduced, said weighting function being determined from the relationship between a surface shape of said object, a three-dimensional shape of said object and the imaging positions of said imaging means, wherein said surface shape and said three-dimensional shape of said object are estimated by said shape estimating means.

12. An endoscopic image processing device according to claim 11, wherein said evaluation function includes an evaluation value of smoothness of an estimated surface shape of said object.

13. An endoscopic image processing device according to claim 11, wherein said weighting function is determined by an inclination of a surface at a predetermined position of said object and the relative position to said imaging means.

14. An endoscopic image processing device according to claim 11, wherein said repeated estimation means samples points, at a predetermined interval, from the three-dimensional shape of said object estimated by said shape estimation means to compute a three-dimensional shape using said evaluation function for the sampled points, and samples points from the three-dimensional shape obtained at an interval smaller than that of the previous sampling to form a new three-dimensional shape.

15. An endoscopic image processing device according to claim 11, wherein the positions of the same point on the respective images detected by said position detection means from the three-dimensional shape obtained by said repeated estimation means are corrected for again estimating the imaging positions of said imaging means from the corrected detected positions, and a three-dimensional shape of said object is repeatedly estimated again by using the estimated positional information.

16. An endoscopic image processing device according to claim 1, further comprising size determination means for determining a value of a size so that values of a three-dimensional shape of said object are computed with a predetermined scale and displayed.

17. An endoscopic image processing device according to claim 16, said size determination means measures the amount of movement of the tip of said endoscope caused by manual operation of said endoscope.

18. An endoscopic image processing device according to claim 16, wherein said size determination means uses imaging means comprising a plurality of sensors having a known relative positional relation.

19. An endoscopic image processing device according to claim 16, wherein said size determination means has a reference substance having a predetermined size so as to obtain an absolute reference value for said object by imaging said reference substance together with said object and displaying said reference substance on the same place together with said object.

20. An endoscopic image processing device according to claim 1, further comprising a light source device for supplying illuminating light to said endoscope.

23. An endoscopic image processing device according to claim 1, wherein said endoscope has a long thin insertion portion which contains said imaging means at the tip portion thereof.

24. An endoscopic image processing device according to claim 1, wherein said endoscope has a long thin insertion portion in which an image guide for transmitting an optical image is inserted so as to photoelectrically convert an optical image by said imaging means.

25. An endoscopic image processing device according to claim 1, wherein said endoscope has a long thin insertion portion, said imaging means having a plurality of image sensors having the ability of photoelectric conversion being provided at the tip of said insertion portion.

26. An endoscopic image processing device according to claim 1, further comprising repeated processing control means for sequentially repeating at least two of a plurality of processes including a first process for detecting the positions of the same point on said plurality of images, a second process of estimating each position of said imaging means and a third process of estimating a three-dimensional shape of said object.

27. An endoscopic image processing device according to claim 1, wherein said shape estimating means estimates a shape of said object with relative values.

28. An endoscopic image processing device according to claim 1, further comprising reference value detection means for detecting a length or an angle value as a reference so that said shape estimating means estimates a shape of said object with absolute values.

5. An endoscopic image processing device comprising:

an endoscope provided with imaging means;

position detection means for detecting the position of a single point on each of a plurality of images of a single object formed by said imaging means at a plurality of imaging positions;

position estimating means for estimating each position of said imaging means from the position of the same point on each of the images;

shape estimating means for estimating a three-dimensional shape of said object using the positional information of said imaging means which is estimated by said position estimating means; and

display means for displaying the three-dimensional shape estimated by said shape estimating means,

wherein said position estimating means comprises 8-point algorithm computation means for determining the relative movement of said imaging means to said object by solving for relations between first and second points on said object and which have unknowns as movement parameters of said imaging means resolved into a rotation matrix and a translation vector, using the movement parameters when said first point is moved to said second point by relative movement of said imaging means and when a correspondence between the points of the images respectively formed by projection of said first and second points on a coordinate system of central projection is provided.

21. An endoscopic image processing device according to claim 21, further comprising signal processing means for processing image signals generated from said imaging means.

22. An endoscopic image processing device according to claim 21, further comprising a monitor for displaying the image signals.

29. An image processing device comprising:

position detection means for detecting the positions of the same point on respective images formed by imaging means at a plurality of positions of the same object;

position estimating means for estimating each position of said object and said imaging means from a shift amount of the positions of the same point on said respective images and a rotation matrix and a translation vector calculated using the shift amount;

shape estimating means for estimating a three-dimensional shape of said object using the positional information of said object and said imaging means estimated by said position estimating means; and

display means for displaying the three-dimensional shape estimated by said shape estimating means.

30. A method of displaying an endoscopic three-dimensional shape comprising the steps of:

forming a plurality of images at a plurality of positions on the same object by an endoscope provided with imaging means;

detecting relative positional deviations of the images by alignment of the positions of the same point on respective images obtained by said image formation step;

estimating relative movement vectors of said imaging means forming the images on the basis of the information about the relative positional deviations of the images, a rotation matrix and a translation vector using the relative positional deviations of the images, and computing relative positional relations between visual points of said imaging means and said object;

estimating a three-dimensional shape of said object using the relative positional relations between said visual points and said object; and

displaying the three-dimensional shape of said object estimated in said shape estimating step on a display device.

31. A display method according to claim 30, further comprising a step of repeating at least one of a plurality of processes including a first process of detecting the positions of the same point on said plurality of images, second process of estimating each position of said imaging means and a third process of estimating a three-dimensional shape of said object.

32. A display method according to claim 30, wherein said positional deviation detection step comprises the step of correcting the aberration caused by an objective optical system of said imaging means.

33. A display method according to claim 30, wherein in said positional deviation detection step, a first area is set on one of two images, and a second area is set on the other image so as to have substantially the same as that of said first area and exhibiting the maximum correlation value between an image in said first area and an image in said second area to detect the positional relation of the same point between the two images.

35. A display method according to claim 30, wherein in said position estimating step, the relative positional relation between said imaging means and said object is computed from the results of said computation step.

36. A display method according to claim 30,

wherein said position estimating step has a step of computing the relative movement of said imaging means to said object by 8-point algorithm calculation for solving for unknowns as movement parameters of said imaging means, which are resolved into said rotation matrix and said translation vector, and relating a first point to a second point on said object using said parameters when the points on respective images produced by central projection of said first and second points correspond to each other when said first point of said object is moved to said second point by the movement of said imaging means relative to said object, and

wherein in said position estimating step, the positional relation is computed in said computation step using a least squares algorithm determining the translation vector so as to minimize the relation between said first and second points using said movement parameters of said imaging means, which are resolved into the rotation matrix and the translation vector.

37. A display method according to claim 36, wherein said relation used in said position estimating step is derived on the assumption that a vector connecting the position of said imaging means and said first point, a vector determined by a vector connecting the position of said imaging means and said second point and said rotation matrix, and said translation vector are present in the same plane.

38. A display method according to claim 30, wherein said shape estimating step has the coordinate transformation step of coordinate transformation processing for transforming relations of said object and a plurality of points of said imaging means into a predetermined coordinate system to compute a three-dimensional shape of said object.

39. A display method according to claim 38, wherein in said shape estimating step, a three-dimensional step is computed using a weighting function determined by the relations between the three-dimensional shape of said object obtained in said coordinate transformation step and the imaging positions of said imaging means.

40. A display method according to claim 38, wherein said shape estimating step has the repeated estimation step of repeatedly estimating again a three-dimensional shape of said object so as to minimize the evaluation function into which a weighting function determined by the relations between the imaging positions of said imaging means and a surface shape and three-dimensional shape of said object, which are estimated in said shape estimation step, is introduced.

41. A display method according to claim 40, wherein said evaluation function includes an evaluation value of smoothness of the estimated surface shape of said object.

42. A display method according to claim 40, wherein said weighting function is determined by an inclination of the surface at a predetermined position of said object and the relative position between said object and said imaging means.

43. A display method according to claim 40, wherein in said repeated estimation step, points are sampled at a predetermined interval from the three-dimensional shape of said object estimated in said shape estimation step, a three-dimensional shape is computed using the evaluation function for each of the sampled points, and points are sampled from the three-dimensional shape obtained at a interval smaller than that of the previous sampling to obtain a new three-dimensional shape.

44. A display method according to claim 40, wherein the position of the same point detected on each image in said positional deviation detection step is corrected on the basis of the three-dimensional shape obtained in said repeated estimation step, the imaging positions of said imaging means are estimated again from the corrected detected positions, and a three-dimensional shape of said object is repeatedly estimated again using the estimated positional information.

45. A display method according to claim 30, further comprising a size determination step of determining a value of a size, computing a three-dimensional shape of said object with values based on a predetermined scale and displaying the three-dimensional shape computed.

46. A display method according to claim 45, wherein in said size determination step, the movement of the tip of said endoscope caused by a manual operation is measured.

47. A display method according to claim 45, said size determination means uses imaging means having a plurality of sensor having a known relative positional relation.

48. A display method according to claim 45, wherein in said size determination step, a reference substance having a predetermined size is imaged together with said object and displayed on the same screen together with said object to obtain an absolute value of said object.

49. A display method according to claim 30, wherein illuminating light is supplied to said endoscope.

50. A display method according to claim 30, further comprising a signal processing step of processing the signal generated from said imaging means to generate an image signal.

51. A display method according to claim 50, wherein said image signal is display on a monitor.

52. A display method according to claim 30, wherein an optical image transmitted through an image guide for transmitting an optical image produced by said endoscope is subjected to photoelectric conversion in said imaging means.

53. A display method according to claim 30, further comprising a repeated process control step of sequentially repeating at least two of a plurality of processes including a first process of detecting the positions of the same point on said plurality of images, a second process of estimating each position of said imaging means, and a third process of estimating a three-dimensional shape of said object.

54. A display method according to claim 30, wherein in said shape estimation step, a relative shape of said object is estimated.

55. A display method according to claim 30, further comprising a reference value computation step of detecting a length or an angle as a reference so that an absolute value of the shape of said object is estimated in said shape estimation step.

34. A display method comprising the steps of:

forming a plurality of images at a plurality of positions on an object by an endoscope provided with imaging means;

detecting relative positional deviations of the images by alignment of the positions of a single point on respective images obtained by said image formation step;

estimating relative movement vectors of said imaging means forming the images on the basis of the information about the relative positional deviations of the images, and computing relative positional relations between visual points of said imaging means and said object;

estimating a three-dimensional shape of said object using the relative positional relations between said visual points and said object,

wherein said position estimating step has a step of computing the relative movement of said imaging means to said object by 8-point algorithm calculation for solving for unknowns as movement parameters of said imaging means, which are resolved into a rotation matrix and a translation vector, and relating a first point to a second point on said object using said parameters when the points on respective images produced by central projection of said first and second points correspond to each other when said first point of said object is moved to said second point by the movement of said imaging means relative to said object.

56. An endoscopic image processing device comprising:

an endoscope provided with imaging means;

position detection means for detecting the position of a single point on each of a plurality of images of a single object formed by said imaging means at a plurality of imaging positions;

position estimating means for estimating each position of said imaging means from the position of the same point on each of the images while taking into account any relative movement between said object and said imaging means, said position estimating means comprising 8-point algorithm computation means for determining the relative movement of said imaging means to said object by solving for relations between first and second points on said object and which have unknowns as movement parameters of said imaging means resolved into a rotation matrix and translation vectors, using the movement parameters when said first point is moved to said second point by relative movement of said imaging means and when a correspondence between the points of the images respectively formed by projection of said first and second points on a coordinate system of central projection is provided;

shape estimating means for estimating a three-dimensional shape of said object using the positional information of said imaging means which is estimated by said position estimating means;

repeated processing control means for repeating at least one of a plurality of processes including a first process of detecting the positions of the same point on said plurality of images, a second process of estimating each position of said imaging means and a third process of estimating a three-dimensional shape of said object; and

display means for displaying the three-dimensional shape estimated by said shape estimating means.