Image processing apparatus using correlation among images

• US 5,764,809 A
• Filed: 09/26/1996
• Issued: 06/09/1998
• Est. Priority Date: 03/26/1991
• Status: Expired due to Term

• Current Assignee: Olympus Optical Company Ltd. (Olympus Corporation)
• Original Assignee: Olympus Optical Company Ltd. (Olympus Corporation)
• Inventors: Nomami, Tetsuo, Nishimura, Hirokazu, Yamaguchi, Masahiro, Nakamura, Masakazu, Ohyama, Nagaaki
• Primary Examiner(s): Couso, Jose L.
• Assistant Examiner(s): Bella, Matthew C.
• Application Number:
  US08/721,397
• Time in Patent Office: 621 Days
• Field of Search: 382/128, 382/151, 382/209, 382/218, 382/293, 382/300, 382/284, 348/71, 348/45, 348/77, 348/241, 348/246, 348/607, 348/615, 348/616, 395/135
• US Class Current: 382/284
• CPC Class Codes: G06T 11/00 2D [Two Dimensional] image generation
• CPC Class Codes: G06T 2207/30004 Biomedical image processing
• CPC Class Codes: G06T 5/80 Geometric correction
• CPC Class Codes: G06T 7/74 involving reference images or patches
• CPC Class Codes: G06V 10/16 using multiple overlapping images; Image stitching
• CPC Class Codes: G06V 10/245 by locating a pattern; Special marks for positioning
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes

Abstract

An image processing apparatus comprises a CPU, an information input unit, a main storage made up of a RAM, an image input interface, a display interface, and a ROM. These components are mutually connected on a bus. The CPU includes an image processor which operates according to programs stored in the ROM. The image processor comprises a corresponding point detector for detecting corresponding points in multiple images, an image matching unit for matching at least two images according to the output result of the corresponding point detector, and an image synthesizer for synthesizing at least part of at least one image portion matched by said image matching unit with at least part of image information of other images to form a single image. The information input unit is a keyboard for entering the type of an electronic endoscope and other data. The image input interface is connected to image memories and receives image data from the memories.

Claims

1. An image processing method comprising:

an image input step for inputting a plurality of two-dimensional endoscopic images acquired from a subject at a plurality of different viewing points;

a corresponding point detecting step for detecting positions of a set of points in one image among said plurality of images with respect to a corresponding set of points in another image among said plurality of images;

an image matching step for matching at least two of said plurality of images by deforming a region in said one image formed by said set of points in said one image detected at said corresponding point detecting step so that its configuration may coincide with that of a region in said another image formed by said corresponding set of points in said another image; and

an image synthesizing step for replacing at least one portion of at least said one image matched at said image matching step with image information of a corresponding portion of said another image matched at said image matching step, so as to form a single two-dimensional image.

2. An image processing method according to claim 1 wherein at said image matching step, images are matched with respect to a coordinate state of at least one of at least two images according to the detection result of said corresponding point detecting step.

3. An image processing method according to claim 1 wherein at said image matching step, at least one of enlargement, reduction, and deformation is performed to match images with respect to a coordinate state of at least one of at least two images according to the detection result of said corresponding point detecting step.

4. An image processing method according to claim 1, wherein said corresponding point detecting step includes a fault detecting step for detecting a fault in an image input by said image input step and a fault corresponding point detecting step for detecting corresponding points relative to said plurality of images and an image containing said fault detected by said fault detecting step, and

said image synthesizing step synthesizes an image by using at least part of image information of other images and at least an image matched by said image matching step to eliminate said fault.

5. An image processing method according to claim 1 wherein at said image input step, a plurality of time-series images are entered;

at said fault corresponding point detecting step, corresponding points relative to an image containing said fault detected at said fault detecting step are detected in at least a succeeding image;

at said image matching step, an image containing said fault is matched with at least one succeeding image according to an output result of said corresponding point detecting step; and

at said image synthesizing step, at least one portion of said image matched at said image matching step is synthesized with at least part of image information of at least a succeeding image to eliminate said fault to form a single image.

6. An image processing method according to claim 1 wherein at said image input step, a plurality of time-series images are entered.

7. An image processing method according to claim 1 wherein at said image synthesizing step, one of at least two images matched at said image matching step is synthesized with at least part of image information of other images to form a single image with a field of view enlarged.

8. An image processing method according to claim 1 wherein said image input step includes a distortion and aberration correcting step for correcting distortion in input images.

9. An image processing method according to claim 1 wherein at said image input step, a plurality of color images are entered.

10. An image processing method according to claim 9 wherein at said corresponding point detecting step, corresponding points are detected in images by performing color matching.

11. An image processing method according to claim 9 wherein at said image input step, said plurality of input color images are divided into R, G and B images;

and

at said corresponding point detecting step, corresponding points relative to a G image are detected in images.

12. An image processing method according to claim 9 wherein at said image input step, said plurality of input color images are divided into R, G, and B;

and

at said corresponding point detecting step, corresponding points are detected in images for each of divided R, G, and B images.

13. An image processing method according to claim 1, wherein said image input step inputs said plurality of images from a stereo endoscope.

14. An image processing method according to claim 1, wherein a part of an image acquired with a wider imaging field and at a larger imaging distance from a subject of at least two images matched by said image matching step is synthesized with a part of image information of an image acquired with a narrower imaging field and at a smaller imaging distance from a subject to form a single image.

15. An image processing method comprising:

an image input step for inputting a plurality of two-dimensional color images acquired from a subject at a plurality of different viewing points;

a corresponding point detecting step for detecting positions of a set of points in one image among said plurality of images with respect to a corresponding set of points in another image among said plurality of images;

an image matching step for matching at least two of said plurality of images by deforming a region in said one image formed by said set of points in said one image detected at said corresponding point detecting step so that its configuration may coincide with that of a region in said another image formed by said corresponding set of points in said another image; and

an image synthesizing step for replacing at least one portion of at least said one image matched at said image matching step with image information of a corresponding portion of said another image matched at said image matching step so as to form a single two-dimensional image.

16. An image processing method according to claim 15 wherein at said image matching step, images are matched with respect to a coordinate state of at least one of at least two images according to the detection result of said corresponding point detecting step.

17. An image processing method according to claim 15 wherein at said image matching step, at least one of enlargement, reduction, and deformation is performed to match images with respect to a coordinate state of at least one of at least two images according to the detection result of said corresponding point detecting step.

18. An image processing method according to claim 15, wherein said corresponding point detecting step includes a fault detecting step for detecting a fault of an image input by said image input step and a fault corresponding point detecting step for detecting corresponding points of an image containing a fault detected by said fault detecting step relative to said plurality of other images, and

said image synthesizing step uses at least part of image information of other images to synthesize an image and at least one image matched by said image matching step to eliminate said fault.

19. An image processing method according to claim 18 wherein at said image input step, a plurality of time-series images are entered;

at said fault corresponding point detecting step, corresponding points relative to an image containing said fault detected at said fault detecting step are detected in at least a succeeding image;

at said image matching step, an image containing said fault is matched with at least one succeeding image according to an output result of said corresponding point detecting step; and

at said image synthesizing step, at least one portion of said image matched at said image matching step is synthesized with at least part of image information of at least a succeeding image to eliminate said fault to form a single image.

20. An image processing method according to claim 15 wherein at said image input step, a plurality of time-series images are entered.

21. An image processing method according to claim 15 wherein at said image synthesizing step, one of at least two images matched at said image matching step is synthesized with at least part of image information of other images to form a single image with a field of view enlarged.

22. An image processing method according to claim 15 wherein said image input step includes a distortion and aberration correcting step for correcting distortion in input images.

23. An image processing method according to claim 15 wherein at said corresponding point detecting step, corresponding points are detected in images by performing color matching.

24. An image processing method according to claim 15 wherein at said image input step, said plurality of input color images are divided into R, G, and B images;

and

at said corresponding point detecting step, corresponding points relative to a G image are detected in images.

25. An image processing method according to claim 15 wherein at said image input step, said plurality of input color images are divided into R, G, and B images;

and

at said corresponding point detecting step, corresponding points are detected in images for each of divided R, G, and B images.

26. An image processing method according to claim 15, wherein a part of an image acquired with a wider imaging field and at a larger imaging distance from a subject of at least two images matched by said image matching step is synthesized with a part of image information of an image acquired with a narrower image field and at a smaller imaging distance from a subject to form a single image.

27. An image processing method comprising:

an image input step for inputting a plurality of images acquired from a subject at a plurality of different viewing points, and having a different imaging distance from a subject;

a corresponding point detecting step for detecting positions of a set of points in one image among said plurality of images with respect to a corresponding set of points in another image among said plurality of images;

an image matching step for matching at least two of said plurality of images by deforming a region in said one image formed by said set of points in said one image detected at said corresponding point detecting step so that its configuration may coincide with that of a region in said another image formed by said corresponding set of points in said another image; and

an image synthesizing step for replacing part of an image acquired with a wider imaging field and at a larger imaging distance from the subject of at least said two images matched at said image matching step, with image information of a corresponding portion of said another image matched at said image matching step and acquired with a narrower imaging field and at a smaller imaging distance from the subject so as to form a single two-dimensional image.

28. An image processing method comprising:

an image input step for inputting a plurality of two-dimensional images acquired by a stereo endoscope from a subject at a plurality of different viewing points;

a corresponding point detecting step for detecting positions of a set of points in one image among said plurality of images with respect to a corresponding set of points in another image among said plurality of images;

an image matching step for matching at least two of said plurality of images by deforming a region in said one image formed by said set of points in said one image detected at said corresponding point detecting step so that its configuration may coincide with that of a region in said another image formed by said corresponding set of points in said another image; and

an image synthesizing step for matching at least one portion of at least said one image matched at said image matching step with image information of a corresponding portion of said another image matched at said image matching step, so as to form a single two-dimensional image.

29. An image processing method comprising:

an image input step for inputting a plurality of time-series endoscopic images in two dimensions acquired from a subject at a plurality of different viewing points;

an input image storage step for storing at least one of said plurality of time-series endoscopic images entered in said image input step;

a fault detecting step for detecting a fault in an image entered at said image input step;

a fault corresponding point detecting step for detecting positions of sets of points relative to one image containing said fault detected in said image at said fault detecting step with a corresponding set of points in a stored image;

an image matching step for matching said one image containing said fault with said stored image at a preceding step by deforming a region in said one image formed by said set of points in said one image detected at said fault corresponding point detecting step so that its configuration may coincide with that of a region in said stored image formed by said corresponding set of points in said stored image; and

an image synthesizing step for eliminating said fault of at least one portion of at least said one image matched at said image matching step with image information of said stored image from a preceding step to form a single two-dimensional image.

30. An image processing method comprising:

an image input step for inputting a plurality of time-series color images in two dimensions acquired from a subject at a plurality of different viewing points;

an input image storage step for storing at least one of said plurality of time-series color images entered in said image input step;

a fault detecting step for detecting a fault in an image entered at said image input step;

a fault corresponding point detecting step for detecting positions of sets of points relative to one image containing said fault detected in said image at said fault detecting step with a corresponding set of points in a stored image;

an image matching step for matching an image containing said fault with said image at a preceding step by deforming a region in said one image formed by said set of points in said one image detected at said fault corresponding point detecting step so that its configuration may coincide with that of a region in said stored image formed by said corresponding set of points in said stored image; and

an image synthesizing step for eliminating said fault of at least one portion of at least one image matched at said image matching step with image information of said stored image from a preceding step to form a single two-dimensional image.

31. An image processing apparatus comprising:

an image input means for inputting a plurality of two-dimensional endoscopic images acquired from a subject at a plurality of different viewing points;

a corresponding point detecting means for detecting positions of a set of points in one image among said plurality of images with respect to a corresponding set of points in another image among said plurality of images;

an image matching means for matching at least two of said plurality of images by deforming a region in said one image formed by said set of points in said one image detected by said corresponding point detecting means so that its configuration may coincide with that of a region in said another image formed by said corresponding set of points in said another image; and

an image synthesizing means for replacing at least one portion of said image matched by said image matching means with image information of a corresponding portion of said another image matched by said image matching means, so as to form a single two-dimensional image.

32. An image processing apparatus comprising:

an image input means for inputting a plurality of two-dimensional color images acquired from a subject at a plurality of different viewing points;

a corresponding point detecting means for detecting positions of a set of points in one image among said plurality of images with respect to a corresponding set of points in another image among said plurality of images;

an image matching means for matching at least two of said plurality of images by deforming a region in said one image formed by said set of points in said one image detected by said corresponding point detecting means so that its configuration may coincide with that of a region in said another image formed by said corresponding set of points in said another image; and

an image synthesizing means for replacing at least one portion of said image matched by said image matching means with image information of a corresponding portion of said another image matched by said image matching means, so as to form a single two-dimensional image.

33. An image processing apparatus comprising:

an image input means for inputting a plurality of time-series endoscopic images in two-dimensions acquired from a subject at a plurality of different viewing points;

an input image storage means for storing at least one image of said plurality of time-series endoscopic images entered by said image input means;

a fault detecting step for detecting a fault in an image entered at said image input step;

a fault corresponding point detecting means for detecting positions of sets of points relative to one image containing said fault detected by said fault detecting means with a corresponding set of points in at least another image stored in said storage means;

an image matching means for matching an image containing said fault with at least said image stored in said storage means by deforming a region in said one image formed by said set of points in said one image detected by said fault corresponding point detecting means so that its configuration may coincide with that of a region in said another image formed by said corresponding set of points in said another image; and

an image synthesizing means for eliminating said fault of at least one portion of said one image matched by said image matching means with image information of at least a portion of said another image stored in said storage means to form a single two-dimensional image.

34. An image processing apparatus comprising:

an image input means for inputting a plurality of time-series color images in two dimensions acquired from a subject at a plurality of different viewing points;

an input image storage means for storing at least one image of said plurality of time-series color images entered by said image input means;

a fault detecting means for detecting a fault in an image entered by said image input means;

a fault corresponding point detecting means for detecting positions of sets of points relative to one image containing said fault detected by said fault detecting means in a corresponding set of points in at least another image stored in said storage means;

an image matching means for matching an image containing said fault with at least said image stored in said storage means by deforming a region in said one image formed by said set of points in said one image detected by said corresponding point detecting means so that its configuration may coincide with that of a region in said another image formed by said corresponding set of points in said another image; and

an image synthesizing means for eliminating at least said fault of at least one portion of said one image matched by said image matching means with image information of at least said another image stored in said storage means to form a single two-dimensional image.