Endoscope apparatus, information storage device, and image processing method

• US 9,492,059 B2
• Filed: 06/24/2013
• Issued: 11/15/2016
• Est. Priority Date: 12/24/2010
• Status: Active Grant

• Current Assignee: Olympus Corporation
• Original Assignee: Olympus Corporation
• Inventors: Sasaki, Hiroshi
• Primary Examiner(s): ELAHI, SHAN E
• Application Number:
  US13/925,197
• Publication Number:
  US 20130286172A1
• Time in Patent Office: 1,240 Days
• Field of Search: 348/65
• US Class Current: 1/1
• CPC Class Codes: A61B 1/000094 extracting biological structures
• CPC Class Codes: A61B 1/000095 for image enhancement
• CPC Class Codes: A61B 1/00188 with focusing or zooming features
• CPC Class Codes: A61B 1/045 Control thereof
• CPC Class Codes: G02B 23/2476 Non-optical details, e.g. housings, mountings, supports
• CPC Class Codes: G06T 2207/10016 Video; Image sequence
• CPC Class Codes: G06T 2207/10024 Color image
• CPC Class Codes: G06T 2207/10068 Endoscopic image
• CPC Class Codes: G06T 2207/10148 Varying focus
• CPC Class Codes: G06T 2207/20221 Image fusion; Image merging
• CPC Class Codes: G06T 5/50 using two or more images, e.g. averaging or subtraction
• CPC Class Codes: G06T 5/73 Deblurring; Sharpening
• CPC Class Codes: H04N 23/555 for picking-up images in sites, inaccessible due to their dimensions or hazardous conditions, e.g. endoscopes or borescopes
• CPC Class Codes: H04N 23/67 Focus control based on electronic image sensor signals
• CPC Class Codes: H04N 23/69 Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
• CPC Class Codes: H04N 23/951 by using two or more images to influence resolution, frame rate or aspect ratio

Abstract

The endoscope apparatus includes an image acquisition section (image composition processing section) that acquires a captured image in a zoom observation state in time series, the zoom observation state being an observation state in which a magnification of an optical system is higher than that in a normal observation state, a defocus amount information extraction section (texture extraction section) that extracts defocus amount information from the captured image in the zoom observation state, and a defocus amount correction section (texture correction amount calculation section, texture correction section, and blending section) that corrects the captured image based on the extracted defocus amount information.

Claims

1. An endoscope apparatus comprising:

an image acquisition section that acquires a captured image in a zoom observation state in time series based on an image signal from an image sensor, the zoom observation state being an observation state in which a magnification of an optical system is higher than that in a normal observation state;

a defocus amount information extraction section that extracts defocus amount information from the captured image in the zoom observation state; and

a defocus amount correction section that corrects the captured image based on the extracted defocus amount information,

wherein the defocus amount information extraction section includes a specific frequency band image extraction section that extracts, as the defocus amount information, a specific frequency band image from the captured image, the specific frequency band image being an image of a specific frequency band which is determined in accordance with the magnification of the optical system in the zoom observation state,

wherein;

the specific frequency band image extraction section includes;

an extraction condition control section that controls an extraction condition for the specific frequency band image based on magnification information about the optical system of the endoscope apparatus, such that the specific frequency band of the specific frequency band image is changed according to the magnification;

a filtering section which performs a filter process based on the extraction condition to obtain a filtered image;

a subtraction section which generates an image by subtracting the filtered image from the captured image which has not been subjected to the filter process; and

a noise amount estimation section which estimates a noise amount based on the extraction condition;

the specific frequency band image extraction section extracts the specific frequency band image based on the image generated by the subtraction section and the noise amount estimated by the noise amount estimation section; and

the defocus amount correction section includes;

a specific frequency band image amplitude calculation section that calculates an amplitude value of the specific frequency band image extracted by the specific frequency band image extraction section based on a pixel value of the specific frequency band image;

a specific frequency band image correction section that corrects the amplitude value of the specific frequency band image to match a correction target value corresponding to an in-focus amplitude value of the specific frequency band image; and

a blending section that blends the captured image and the corrected specific frequency band image to thereby correct the captured image and obtain a corrected captured image that is in focus.

2. The endoscope apparatus as defined in claim 1, wherein the specific frequency band image extraction section further includes a captured image amplitude calculation section that calculates an amplitude value of the captured image,

wherein the specific frequency band image extraction section extracts the specific frequency band image based further on the amplitude value of the captured image calculated by the captured image amplitude calculation section such that (i) the specific frequency band image extraction section does not extract the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation section is within a first amplitude range, (ii) the specific frequency band image extraction section extracts the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation section is within a second amplitude range, the second amplitude range being a range that corresponds to an amplitude value that is smaller than an amplitude value within the first amplitude range, and (iii) the specific frequency band image extraction section does not extract the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation section is within a third amplitude range, the third amplitude range being a range that corresponds to an amplitude value that is smaller than an amplitude value within the second amplitude range.

3. The endoscope apparatus as defined in claim 2, wherein the specific frequency band image, of which the amplitude value calculated by the captured image amplitude calculation section is within the second amplitude range, is an image that corresponds to a surface of a mucous membrane and a blood vessel in a surface of tissue.

4. The endoscope apparatus as defined in claim 2, wherein an image of which the amplitude value calculated by the captured image amplitude calculation section is within the first amplitude range is an image that corresponds to folds on a surface of tissue, and an image of which the amplitude value calculated by the captured image amplitude calculation section is within the third amplitude range is an image that corresponds to noise.

5. The endoscope apparatus as defined in claim 1, wherein the specific frequency band image extraction section includes a captured image amplitude calculation section that calculates an amplitude value of the captured image,

wherein the specific frequency band image extraction section extracts the specific frequency band image based further on the amplitude value of the captured image calculated by the captured image amplitude calculation section such that (i) the specific frequency band image extraction section does not extract the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation section is within a first amplitude range, and (ii) the specific frequency band image extraction section extracts the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation section is within a second amplitude range, the second amplitude range being a range that corresponds to an amplitude value that is smaller than an amplitude value within the first amplitude range,

wherein the specific frequency band image is an image of which the amplitude value calculated by the captured image amplitude calculation section is within the second amplitude range and the frequency band is within a first frequency range, the first frequency range being a range that corresponds to a frequency that is higher than a frequency within a second frequency range and is determined based on the magnification information.

6. The endoscope apparatus as defined in claim 1, wherein the specific frequency band of the specific frequency band image is within a first frequency range, and a frequency within the first frequency range is higher than a frequency within a second frequency range, the first frequency range being a range that is larger than a first frequency threshold value and smaller than a second frequency threshold value, and wherein the specific frequency band image extraction section increases the first frequency threshold value and the second frequency threshold value as the magnification indicated by the magnification information about the optical system of the endoscope apparatus increases.

7. The endoscope apparatus as defined in claim 1, wherein the specific frequency band image extraction section includes:

a frequency band separation section that generates a plurality of sub-band images, the sub-band image being an image obtained by dividing the captured image into a plurality of specific frequency bands determined based on the magnification information; and

a sub-band selection section that selects one or more sub-band images from the plurality of sub-band images to generate the specific frequency band image.

8. The endoscope apparatus as defined in claim 1, wherein:

the filtering section comprises a first filtering section which performs a first filter process as the filter process; and

the specific frequency band image extraction section further includes;

a second filtering section;

the image generated by the subtraction section is a structural image, and the subtraction section calculates a difference between the filtered image obtained by the first filter process performed by the first filtering section and the captured image to generate the structural image,

the noise amount estimation section estimates a noise amount of the structural image based on the extraction condition, and

the second filtering section separates noise from the structural image based on the noise amount estimated by the noise amount estimation section to acquire the specific frequency band image.

9. The endoscope apparatus as defined in claim 8,

wherein the specific frequency band image extraction section includes a captured image amplitude calculation section that calculates an amplitude value of the captured image,

wherein the specific frequency band image extraction section extracts the specific frequency band image based further on the amplitude value of the captured image calculated by the captured image amplitude calculation section such that (i) the specific frequency band image extraction section does not extract the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation section is within a first amplitude range, and (ii) the specific frequency band image extraction section extracts the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation section is within a second amplitude range, the second amplitude range being a range that corresponds to an amplitude value that is smaller than an amplitude value within the first amplitude range, and

wherein the first filtering section subjects an image within the first amplitude range to a low-pass filter process that allows a frequency band that corresponds to a first frequency range and a second frequency range to pass through, the first frequency range being a range that corresponds to a frequency that is higher than a frequency within a second frequency range, and subjects an image within the second amplitude range to a low-pass filter process that allows a frequency band that corresponds to the second frequency range to pass through, and blocks a frequency band that corresponds to the first frequency range.

10. The endoscope apparatus as defined in claim 8, wherein the first filtering section includes:

an area extraction section that extracts a given area around an attention pixel from the captured image;

a filter coefficient calculation section that calculates a spatial distance and a difference in pixel value between the attention pixel and a peripheral pixel within the given area, and calculates a filter coefficient for a pixel within each given area based on the extraction condition; and

a filtering section that performs the first filter process using the given area and the filter coefficient.

11. The endoscope apparatus as defined in claim 8,

wherein the specific frequency band image extraction section includes a captured image amplitude calculation section that calculates an amplitude value of the captured image,

wherein the specific frequency band image extraction section extracts the specific frequency band image based further on the amplitude value of the captured image calculated by the captured image amplitude calculation section such that (i) the specific frequency band image extraction section does not extract the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation section is within a first amplitude range, and (ii) the specific frequency band image extraction section extracts the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation section is within a second amplitude range, the second amplitude range being a range that corresponds to an amplitude value that is smaller than an amplitude value within the first amplitude range, and

wherein the second filtering section subjects an image within the second amplitude range to a low-pass filter process that allows a frequency band that corresponds to a first frequency range and a second frequency range to pass through, the first frequency range being a range that corresponds to a frequency that is higher than a frequency within a second frequency range, and subjects an image within a third amplitude range that is higher than the second amplitude range to a low-pass filter process that allows a frequency band that corresponds to the second frequency range to pass through, and blocks a frequency band that corresponds to the first frequency range.

12. The endoscope apparatus as defined in claim 8, wherein the second filtering section includes:

an area extraction section that extracts a given area around an attention pixel from the structural image;

a filter coefficient calculation section that calculates a spatial distance and a difference in pixel value between the attention pixel and a peripheral pixel within the given area, and calculates a filter coefficient for a pixel within each given area based on the noise amount; and

a noise separation section that performs a filter process using the extracted area and the filter coefficient to separate noise from the attention pixel.

13. The endoscope apparatus as defined in claim 1, wherein the defocus amount correction section corrects the captured image acquired in time series based on a defocus amount of an in-focus captured image which is a captured image that has been determined to be in focus.

14. The endoscope apparatus as defined in claim 13, wherein the in-focus captured image is a captured image extracted when the optical system of the endoscope apparatus has been switched to the zoom observation state.

15. The endoscope apparatus as defined in claim 13, wherein the in-focus captured image is a captured image that has been extracted after the optical system of the endoscope apparatus has been switched to the zoom observation state, and has an amplitude value of the specific frequency band image that has satisfied a given condition within a given period.

16. The endoscope apparatus as defined in claim 15, wherein the in-focus captured image is a captured image that has been extracted after the optical system of the endoscope apparatus has been switched to the zoom observation state, and has an amplitude value of the specific frequency band image that is larger than a reference amplitude determination threshold value and has a maximum value.

17. The endoscope apparatus as defined in claim 1, wherein the specific frequency band image correction section includes:

a reference amplitude value selection section that selects the in-focus amplitude value that has been determined to be in focus from the amplitude value of the specific frequency band image calculated in time series by the specific frequency band image amplitude calculation section as a reference amplitude value corresponding to the correction target value; and

a correction amount multiplication section that calculates a correction amount for the specific frequency band image based on the reference amplitude value, and multiplies the specific frequency band image by the correction amount.

18. The endoscope apparatus as defined in claim 17, wherein:

the specific frequency band image extracted by the specific frequency band image extraction section is one or more sub-band images having one or more frequency bands, and

the correction amount multiplication section includes a sub-band image correction amount multiplication section that calculates the correction amount corresponding to each sub-band image based on one or more amplitude values that are calculated by the specific frequency band image amplitude calculation section and correspond to each sub-band image and the magnification information about the optical system of the endoscope apparatus, and multiplies each sub-band image by the calculated correction amount.

19. The endoscope apparatus as defined in claim 1, further comprising:

a focal distance control section that controls a focal distance; and

a target focal distance calculation section that calculates a target focal distance based on the defocus amount information calculated by the defocus amount information extraction section for the captured images based on a plurality of focal distances moved in time series by the focal distance control section, the focal distance control section controlling the focal distance based on the target focal distance.

20. The endoscope apparatus as defined in claim 19, wherein the defocus amount correction section determines the captured image acquired at a focal distance that has been determined to be in focus by the focal distance control section to be an in-focus captured image, and corrects the captured image based on the defocus amount information about the in-focus captured image.

21. The endoscope apparatus as defined in claim 1, wherein the zoom observation state is a state in which the magnification of the optical system of the endoscope apparatus is equal to or higher than a given magnification.

22. The endoscope apparatus as defined in claim 1,

wherein the specific frequency band image extraction section further includes a captured image amplitude calculation section that calculates an amplitude value of the captured image, and

wherein the specific frequency band image extraction section extracts the specific frequency band image based further on the amplitude value of the captured image calculated by the captured image amplitude calculation section such that (i) the specific frequency band image extraction section does not extract the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation section is within a first amplitude range, and (ii) the specific frequency band image extraction section extracts the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation section is within a second amplitude range, the second amplitude range being a range that corresponds to an amplitude value that is smaller than an amplitude value within the first amplitude range.

23. An information storage device storing a program that instructs a computer to perform functions comprising:

acquiring a captured image in a zoom observation state in time series, the zoom observation state being an observation state in which a magnification of an optical system is higher than that in a normal observation state;

performing a defocus amount information extraction process that extracts defocus amount information from the captured image in the zoom observation state; and

performing a defocus amount correction process that corrects the captured image based on the extracted defocus amount information,

wherein;

the defocus amount information extraction process comprises performing a specific frequency band image extraction process of extracting, as the defocus amount information, a specific frequency band image from the captured image, the specific frequency band image being an image of a specific frequency band which is determined in accordance with the magnification of the optical system in the zoom observation state,

the specific frequency band image extraction process includes;

an extraction condition control process that controls an extraction condition for the specific frequency band image based on magnification information about the optical system of the endoscope apparatus, such that the specific frequency band of the specific frequency band image is changed according to the magnification;

a filter process which is performed based on the extraction condition to obtain a filtered image;

a subtraction process that generates an image by subtracting the filtered image from the captured image which has not been subjected to the filter process; and

a noise amount estimation process that estimates a noise amount based on the extraction condition;

the specific frequency band image extraction process extracts the specific frequency band image based on the image generated by the subtraction process and the noise amount estimated by the noise amount estimation process, and

the defocus amount correction process includes;

a specific frequency band image amplitude calculation process that calculates an amplitude value of the specific frequency band image extracted by the specific frequency band image extraction process based on a pixel value of the specific frequency band image;

a specific frequency band image correction process that corrects the amplitude value of the specific frequency band image to match a correction target value corresponding to an in-focus amplitude value of the specific frequency band image; and

a blending process that blends the captured image and the corrected specific frequency band image to thereby correct the captured image and obtain a corrected captured image that is in focus.

24. The information storage device as defined in claim 23, wherein the specific frequency band image extraction process further includes a captured image amplitude calculation process that calculates an amplitude value of the captured image, and

wherein the specific frequency band image extraction process extracts the specific frequency band image based further on the amplitude value of the captured image calculated by the captured image amplitude calculation process such that (i) the specific frequency band image extraction process does not extract the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation process is within a first amplitude range, and (ii) the specific frequency band image extraction process extracts the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation process is within a second amplitude range, the second amplitude range being a range that corresponds to an amplitude value that is smaller than an amplitude value within the first amplitude range.

25. An image processing method for an image processing apparatus, the method comprising, with the image processing apparatus:

acquiring a captured image in a zoom observation state in time series, the zoom observation state being an observation state in which a magnification of an optical system is higher than that in a normal observation state;

performing a defocus amount information extraction process that extracts defocus amount information from the captured image in the zoom observation state; and

performing a defocus amount correction process that corrects the captured image based on the extracted defocus amount information,

wherein;

the defocus amount information extraction process comprises performing a specific frequency band image extraction process of extracting, as the defocus amount information, a specific frequency band image from the captured image, the specific frequency band image being an image of a specific frequency band which is determined in accordance with the magnification of the optical system in the zoom observation state,

the specific frequency band image extraction process includes;

an extraction condition control process that controls an extraction condition for the specific frequency band image based on magnification information about the optical system of the endoscope apparatus, such that the specific frequency band of the specific frequency band image is changed according to the magnification;

a filter process which is performed based on the extraction condition to obtain a filtered image;

a subtraction process that generates an image by subtracting the filtered image from the captured image which has not been subjected to the filter process; and

a noise amount estimation process that estimates a noise amount based on the extraction condition;

the specific frequency band image extraction process extracts the specific frequency band image based on the image generated by the subtraction process and the noise amount estimated by the noise amount estimation process, and

the defocus amount correction process includes;

a specific frequency band image amplitude calculation process that calculates an amplitude value of the specific frequency band image extracted by the specific frequency band image extraction process based on a pixel value of the specific frequency band image;

a specific frequency band image correction process that corrects the amplitude value of the specific frequency band image to match a correction target value corresponding to an in-focus amplitude value of the specific frequency band image; and

a blending process that blends the captured image and the corrected specific frequency band image to thereby correct the captured image and obtain a corrected captured image that is in focus.

26. The image processing method as defined in claim 25, wherein the specific frequency band image extraction process further includes a captured image amplitude calculation process that calculates an amplitude value of the captured image, and

wherein the specific frequency band image extraction process extracts the specific frequency band image based further on the amplitude value of the captured image calculated by the captured image amplitude calculation process such that (i) the specific frequency band image extraction process does not extract the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation process is within a first amplitude range, and (ii) the specific frequency band image extraction process extracts the specific frequency band image when the amplitude value calculated by the captured image amplitude calculation process is within a second amplitude range, the second amplitude range being a range that corresponds to an amplitude value that is smaller than an amplitude value within the first amplitude range.